This paper made customizing giant AI models 10,000× cheaper with one insight: the CHANGE a model needs is tiny in structure — so learn two skinny matrices instead of rewriting billions of weights.
Five animated scenes — everything you need to understand this paper, no reading required. Then scroll on to play with the ideas yourself.
LoRA — the picture-book version
No paper-reading needed — the pictures tell the whole story
Customizing a giant AI used to mean re-tuning ALL 175 billion knobs.
And saving a complete 350GB copy for every single use case. Only the biggest labs could afford it.
The same ideas at three altitudes — start where you're comfortable, climb when you're curious.
Say you want GPT-3 to write in your company's style. The classic answer — fine-tuning — means adjusting all 175 billion of its numbers and saving a complete 350GB copy for every single use case. Ten customizations, ten full copies, each needing a GPU farm to train. Personalizing giant models was becoming a rich-labs-only sport.
Here's the surprise the paper builds on: when you fine-tune a giant model, the CHANGE to its weights is highly redundant — like a full-page edit that's really just three repeated corrections. In math terms, the update matrix has 'low rank': it can be rebuilt almost perfectly from a few building blocks. The lab on this page shows this with a real decomposition — slide the rank up and watch a full update reappear from just 3 layers.
LoRA freezes the original model completely and learns just two skinny matrices per layer whose product IS the update. Result: 10,000× fewer trainable numbers, 3× less GPU memory, zero extra slowness once merged in — and quality matching full fine-tuning. Today, when someone fine-tunes a model on a single gaming GPU, or an app hot-swaps between custom personalities, that's LoRA.
Every great paper is a story: what was broken, the idea that fixed it, and what it unlocked.
Rebinding the entire encyclopedia — all 175 billion words — every time you want to add a page of company notes.
Leave the encyclopedia untouched and clip in a thin sheet of corrections — the reading experience is identical, and you can carry a different sheet for every purpose.
Custom AI went from bespoke tailoring (rebuild the suit) to accessories (swap the tie) — same suit, endless outfits.
A real SVD, sliced layer by layer: slide the rank and see how few directions of change rebuild a full fine-tuning update — the paper's entire bet, visible.
A real SVD, sliced rank by rank
Full fine-tune ΔW
LoRA rank 2 (B·A)
Difference (lost)
The matrix's real singular values — its "directions of change", strongest first
Teal bars = directions your rank-2 adapter keeps. After the first three, the values crash to a noise floor — the update was low-rank all along. That is the paper's "low intrinsic rank" hypothesis, visible.
The decomposition, reconstructions, error percentages, and singular values are all real (SVD via power iteration, computed live). The ΔW matrix itself is synthetic, built with the low intrinsic rank the paper measured in genuine fine-tuning updates.
Rank 2 uses only 96 numbers (17% of the full update) and already captures 78% of the change. Look at the "difference" panel — mostly faint noise.
Analogy: A photo that's secretly just 3 layered gradients: store the 3 layers instead of every pixel and you keep almost the whole picture at a fraction of the cost.
Follow a sentence through the model. Every block is clickable — purpose, example, and the deeper detail.
Click any block to inspect it
r × k, Gaussian init
The 'compressor': a skinny matrix projecting the 12,288-dimensional input down to r dimensions — the few directions of change this task needs.
Example
With r = 4: a 4 × 12,288 matrix — reading the input through 4 learned lenses.
Go deeper
Initialized with random Gaussian values so the low-rank subspace starts unbiased.
Tip: follow the animated edges top-to-bottom — that is the exact journey a sentence takes through the model.
Formula → plain meaning → real-world analogy. Hover the symbols to decode them one by one.
Hover a symbol to decode it.
What it means
The frozen model's output plus a learned correction, where the correction is forced through an r-dimensional bottleneck. B starts at zero, so training begins exactly at the pretrained model and grows the change from nothing.
Real-world analogy
A master chef's recipe (frozen) plus your pencil marks in the margin (B·A) — the dish is recipe-plus-marks, and erasing the marks restores the original perfectly.
Hover a symbol to decode it.
What it means
A rank-r matrix is a sum of r rank-1 'layers' (σ·u·vᵀ each) — r independent directions of change. LoRA bets a handful suffice, and the paper measures it: on GPT-3, r = 1–4 already matches full fine-tuning on many tasks.
Real-world analogy
An orchestra re-tuning for a new hall doesn't need 100 individual instructions — 'strings up a touch, brass softer' covers it. Few directions, big effect.
Hover a symbol to decode it.
What it means
Two skinny matrices versus one full one. For GPT-3's 12,288-wide projections at r = 8: ~197K versus ~151M per matrix — and 10,000× fewer trainable parameters overall versus full fine-tuning (which also drags Adam's optimizer states for every weight).
Real-world analogy
Describing a page edit as 'swap these two paragraphs' instead of retyping the page: the description scales with the CHANGE, not the document.
Hover a symbol to decode it.
What it means
After training, fold the adapter into the base with one addition. The deployed model is a plain weight matrix — identical inference cost to the original. Adapters compose by addition and remove by subtraction.
Real-world analogy
Transparent sheets over a map: draw your route on the sheet, and when you need speed, photocopy map+sheet into one page. Swap sheets for different journeys; the map never changes.
LoRA's skinny B·A updates target attention's query and value projections — the exact Q and V roles you can explore here.
Every word attends to every other word — see it happen
👆 touch a word to see its attention
Note: these attention patterns are an illustration modeled on head behaviors observed in trained Transformers (previous-word heads, coreference heads, …) — not the output of a live model. The equations and mechanism shown are exactly those of the paper.
Hover or tap any word. The lines show where that word is "looking" — this is attention, the paper's big idea.
Analogy: A classroom where every student can listen to every other student at the same time — and each decides who is worth listening to.
Eight pages from 2017 that you now interact with every single day.
LoRA (and QLoRA, its 4-bit descendant) is how individuals fine-tune 7B–70B open models on a single gaming card — the open-model ecosystem's workhorse.
SaaS platforms serve one frozen base model with thousands of hot-swappable customer adapters — personalization without per-customer GPU farms.
Stable Diffusion communities trade LoRA files as portable art styles and characters — megabyte adapters that restyle a multi-gigabyte model.
Medical, legal, and code-specific model variants ship as adapters on shared bases — auditable, cheap to update, trivial to roll back.
LoRA anchored parameter-efficient fine-tuning as a research area: AdaLoRA (adaptive ranks), DoRA (direction/magnitude), and dozens of variants iterate on this recipe.
Tiny trainable footprints make on-device personalization plausible — a phone-sized adapter atop a shared foundation, keeping user data local.
Reading real code is the final test of understanding. This is the paper's core mechanism, minimally.
The entire method in one module: freeze W₀, train two skinny matrices, add their product. Note the zero-init on B — training starts exactly at the pretrained model — and the merge that makes deployment free.
1import torch2import torch.nn as nn345class LoRALinear(nn.Module):6 """A frozen linear layer with a trainable low-rank bypass."""78 def __init__(self, base: nn.Linear, r: int = 8, alpha: int = 16):9 super().__init__()10 self.base = base11 self.base.weight.requires_grad_(False) # W0 is FROZEN1213 d, k = base.weight.shape14 self.A = nn.Parameter(torch.randn(r, k) * 0.01) # Gaussian init15 self.B = nn.Parameter(torch.zeros(d, r)) # ZERO init:16 self.scale = alpha / r # so BA = 0 at step one --17 # training starts at W0 exactly.1819 def forward(self, x: torch.Tensor) -> torch.Tensor:20 # frozen path + low-rank correction21 return self.base(x) + self.scale * (x @ self.A.T @ self.B.T)2223 @torch.no_grad()24 def merge(self):25 """Fold the adapter in: zero-latency deployment."""26 self.base.weight += self.scale * (self.B @ self.A)272829# --- The arithmetic that changed the economics ---30d = k = 12_288 # one GPT-3 attention projection31full = d * k # 150,994,944 trainable params32lora = 2 * d * 8 # 196,608 at rank 833print(f"trainable: {full:,} -> {lora:,} ({full // lora}x fewer)")3435# Applied to W_q and W_v across all 96 layers, versus full fine-tuning36# with Adam states for 175B params: ~10,000x fewer trainable parameters,37# ~3x less GPU memory -- and the merged model runs at base-model speed.
Four questions, each targeting a concept people commonly get wrong. Mistakes come with explanations.
Question 1 of 4
What was the key drawback of adapter layers that LoRA set out to beat?