GPT-3: how scale taught AI to take instructions
A 4-lesson mini course: the fine-tuning treadmill, in-context learning, the scaling bet, and an honest look at what 175B parameters bought.
The fine-tuning treadmill
The old deal
By 2019, NLP had a winning recipe: pretrain a Transformer on raw text (like BERT or GPT-2), then fine-tune it on each task with thousands of labeled examples. It worked — but it meant a translation model, a sentiment model, a QA model… a new dataset, training run, and checkpoint for every single job.
Three cracks in the recipe
The paper names the problems: labeled datasets are expensive and don't exist for most tasks; fine-tuned models can exploit quirks of their training data and look better than they truly are; and the recipe is nothing like humans, who pick up a new task from a description and a couple of examples.
The radical alternative
GPT-2 had hinted that a language model could do tasks zero-shot, badly. GPT-3 asks: what if the model were 100× bigger? Could examples IN THE PROMPT replace fine-tuning entirely? Try the playground below and notice the model is frozen — everything you change is in the input, never the weights. That's the paradigm this paper proposed.
Key takeaway
Pretrain-then-fine-tune required a dataset and training run per task. GPT-3's bet: make the model big enough and the prompt becomes the training.
Hands-on — experiment before the quiz
Next-Token Playground
How GPT-style models choose every single word
Next-token probabilities
Note: the candidate tokens and their starting scores are curated examples, but the temperature, top-k, top-p, softmax, and sampling math applied to them is exactly the algorithm production LLMs use to pick every word.
Temperature 0.80 keeps a balanced distribution — mostly confident, with some variety.
Analogy: The model builds a weighted raffle for the next word — temperature decides how fair the raffle is, Top-K and Top-P decide who's even allowed to enter.
Checkpoint
What did fine-tuning require for EVERY new task?