This paper showed that a large enough language model needs no retraining to learn new tasks — just show it a few examples in the prompt. That discovery is why you can simply talk to AI today.
Five animated scenes — everything you need to understand this paper, no reading required. Then scroll on to play with the ideas yourself.
Language Models are Few-Shot Learners (GPT-3) — the picture-book version
No paper-reading needed — the pictures tell the whole story
Before GPT-3, every single job needed its own specially-trained AI.
A translator model, a summarizer model, a question model… each with its own school and its own thousands of examples.
The same ideas at three altitudes — start where you're comfortable, climb when you're curious.
Before GPT-3, every AI task needed its own specially-trained model: one for translation, one for answering questions, one for summarizing. GPT-3 changed the deal: one giant model, frozen, never retrained — and you 'program' it just by typing. Show it two examples of English→French in your message, and it translates the third line. It learned the task from your prompt, in the moment.
GPT-3 has 175 billion parameters — the little numbers the network tunes while learning. That was 10× larger than any comparable model before it, and over 100× larger than GPT-2. It read roughly 300 billion words' worth of internet text, books, and Wikipedia during training. The bet was simple and audacious: don't invent a cleverer architecture — take the Transformer and make it enormous.
The skill GPT-3 revealed — 'in-context learning', where the model picks up a task from examples in the prompt — is the direct ancestor of chatting with AI. ChatGPT, Claude, and Gemini all work because scale made models that understand instructions written in plain language. This paper is the moment AI stopped needing programmers for every new task and started needing only a good prompt.
Every great paper is a story: what was broken, the idea that fixed it, and what it unlocked.
Like needing to send an employee back to school for a semester every time you want them to do a slightly different job.
A hugely well-read new hire: show them two filled-out forms and they correctly fill out the third — no training course needed.
The moment computers stopped needing code for every new job and started taking instructions in plain English.
GPT-3's output is one next-token raffle, repeated. These are the real sampling controls — temperature, top-k, top-p — that shaped every GPT completion ever generated.
How GPT-style models choose every single word
Next-token probabilities
Note: the candidate tokens and their starting scores are curated examples, but the temperature, top-k, top-p, softmax, and sampling math applied to them is exactly the algorithm production LLMs use to pick every word.
Temperature 0.80 keeps a balanced distribution — mostly confident, with some variety.
Analogy: The model builds a weighted raffle for the next word — temperature decides how fair the raffle is, Top-K and Top-P decide who's even allowed to enter.
Follow a sentence through the model. Every block is clickable — purpose, example, and the deeper detail.
Click any block to inspect it
masked attention + FFN, ×96
The heart: 96 stacked Transformer decoder blocks, each with masked multi-head self-attention (96 heads) and a feed-forward network, wrapped in residual connections. Each token can attend to everything before it — including your examples.
Example
While predicting the French translation, attention heads look back at your worked examples — this is where in-context learning physically happens.
Go deeper
Masked attention means position t sees only positions < t. The blocks alternate dense and locally-banded sparse attention. It's the 2017 Transformer's decoder, scaled: 512→12,288 dims, 6→96 layers, 8→96 heads.
Tip: follow the animated edges top-to-bottom — that is the exact journey a sentence takes through the model.
Formula → plain meaning → real-world analogy. Hover the symbols to decode them one by one.
Hover a symbol to decode it.
What it means
The probability of a whole text is built one token at a time: each token's probability depends on everything before it. Training just maximizes this on 300B tokens — 'predict the next word' is the entire objective.
Real-world analogy
Reading a mystery novel and constantly guessing the next word. Get good enough at that game, and you've implicitly learned grammar, facts, logic, and style — because they all help you guess.
Hover a symbol to decode it.
What it means
Few-shot 'learning' is just conditional probability: the examples sit in the context, and the model's next-token distribution shifts to be consistent with the pattern they establish. No weights move — the 'learning' is inference.
Real-world analogy
A pub-quiz veteran who's seen a million question styles: read them two sample Q&As and they instantly adopt the format. They didn't study — they recognized.
Hover a symbol to decode it.
What it means
Validation loss falls as a smooth power law of training compute — across the paper's 8 model sizes spanning three orders of magnitude. Capability isn't a lottery; it's purchasable and predictable.
Real-world analogy
Like fuel efficiency curves for engines: you don't need to build the big engine to know roughly how it will perform — the small ones trace the curve for you.
Hover a symbol to decode it.
What it means
The same attention as the 2017 Transformer, plus a mask M that sets all future positions to −∞ before softmax — so every token can only consult its past. That honesty at training time is what makes generation possible.
Real-world analogy
Taking an exam where each question's answer may use all previous pages but the next pages are stapled shut — so you genuinely learn to continue, not to peek.
In-context learning happens when attention heads look back at the examples in your prompt. This is that mechanism — GPT-3 runs 96 heads of it, 96 layers deep.
Every word attends to every other word — see it happen
👆 touch a word to see its attention
Note: these attention patterns are an illustration modeled on head behaviors observed in trained Transformers (previous-word heads, coreference heads, …) — not the output of a live model. The equations and mechanism shown are exactly those of the paper.
Hover or tap any word. The lines show where that word is "looking" — this is attention, the paper's big idea.
Analogy: A classroom where every student can listen to every other student at the same time — and each decides who is worth listening to.
Eight pages from 2017 that you now interact with every single day.
ChatGPT is GPT-3's direct descendant: instruction tuning and RLHF layered on top of exactly this model family and its in-context abilities.
Codex — the model behind the original GitHub Copilot — was GPT-3 fine-tuned on code. Few-shot prompting is why it could adapt to your codebase's style.
Few-shot prompting, chain-of-thought, system prompts — the entire craft of prompt engineering exists because this paper showed prompts can replace training.
The API-first model business — where startups build on a frozen general model instead of training their own — began with GPT-3's launch.
GPT-4, Claude, Gemini, Llama — every frontier lab's strategy of predictable capability-from-scale traces to this paper's 8-model sweep.
The recipe — huge autoregressive Transformer + in-context examples — now drives code, images, audio, robotics policies, and biology sequence models.
Reading real code is the final test of understanding. This is the paper's core mechanism, minimally.
This is GPT-3's entire inference algorithm: build a few-shot prompt, then repeatedly sample the next token and append it. The model never trains on your task — swap `model.next_token_probs` for a real 175B forward pass and this is production code.
1# The few-shot prompt IS the programming.2prompt = """Translate English to French.34sea otter => loutre de mer5peppermint => menthe poivree6cheese =>"""789def generate(model, prompt: str, max_tokens: int = 20,10 temperature: float = 0.7, top_p: float = 0.9) -> str:11 """The autoregressive loop behind every GPT completion."""12 tokens = model.tokenize(prompt) # BPE, ~50k vocabulary1314 for _ in range(max_tokens):15 # One full forward pass: 96 layers of masked self-attention.16 # The model attends BACK over the examples in the prompt --17 # that attention is where in-context learning happens.18 probs = model.next_token_probs(tokens)1920 # Temperature: reshape the distribution (see the Playground below)21 probs = probs ** (1.0 / temperature)22 probs = probs / probs.sum()2324 # Nucleus (top-p) sampling: keep the smallest set of tokens25 # covering top_p of the probability mass, then sample.26 next_token = nucleus_sample(probs, top_p)2728 if next_token == model.END_OF_TEXT:29 break30 tokens.append(next_token) # append & repeat3132 return model.detokenize(tokens)333435# No gradient updates. No fine-tuning. The 'training' for the36# translation task is the two examples sitting in the prompt.37completion = generate(model, prompt)38print(completion.splitlines()[-1]) # cheese => fromage
Four questions, each targeting a concept people commonly get wrong. Mistakes come with explanations.
Question 1 of 4
What did fine-tuning require for EVERY new task?