LoRA: the 10,000× discount on customization
A 4-lesson mini course: why fine-tuning giants broke, the low-rank insight, the B·A mechanism, and the ecosystem it unlocked.
The fine-tuning wall
Full fine-tuning, honestly priced
Adapting GPT-3 the classic way means gradients for all 175B parameters — plus Adam's optimizer states, which multiply the memory bill several-fold. Then storage: every task yields a complete ~350GB checkpoint. Ten customers with ten styles? Ten full copies of one of the largest artifacts ever trained.
The shortcuts and their taxes
The field had workarounds, each with a tax. Adapter layers (small modules inserted between layers) train cheaply but run at inference forever — the paper measures the latency hit, worst exactly where products live (batch size 1, short sequences). Prefix tuning learns magic tokens but is finicky to optimize and spends your context window. Cheap training OR clean inference — pick one.
The question worth asking
Underneath it all sits an assumption: that adaptation NEEDS billions of adjustable numbers. Does it? How much information is actually in the difference between 'GPT-3' and 'GPT-3, but good at your task'? That measurement is the next lesson — and the answer is the whole paper.
Key takeaway
Full fine-tuning scales with model size (compute, memory, and 350GB per task); prior shortcuts taxed inference or context. The open question: how big is the change, really?
Hands-on — experiment before the quiz
Click any block to inspect it
Matrix A
r × k, Gaussian init
The 'compressor': a skinny matrix projecting the 12,288-dimensional input down to r dimensions — the few directions of change this task needs.
Example
With r = 4: a 4 × 12,288 matrix — reading the input through 4 learned lenses.
Go deeper
Initialized with random Gaussian values so the low-rank subspace starts unbiased.
Tip: follow the animated edges top-to-bottom — that is the exact journey a sentence takes through the model.
Checkpoint
What was the key drawback of adapter layers that LoRA set out to beat?