RLHF: teaching AI what we actually want
A 4-lesson mini course: why imitation caps quality, learning a judge from preferences, optimizing on a leash, and the Goodhart trap.
The imitation ceiling
How summarizers were trained
The standard recipe: collect posts with human-written summaries, train the model to reproduce those summaries word by word (maximum likelihood), and evaluate with ROUGE — a score counting word overlap with the reference. Reasonable-sounding, and the whole field ran on it.
Three quiet failures
First, the ceiling: an imitator can at best tie its references — and reference summaries (the paper picked Reddit TL;DR partly because CNN/DailyMail's are notoriously weak) are often mediocre. Second, the metric: word overlap rewards phrasing, not faithfulness — a summary can ace ROUGE while missing the point. Third, and deepest: 'write like the reference' and 'be a good summary' are different objectives that only partially overlap.
The reframe
What we actually want is simple to state: humans should prefer the model's output. This paper's move is to optimize THAT — directly. The catch is that human judgment is slow and expensive, and gradient descent needs millions of evaluations. Lesson 2 solves exactly this. (First, try the token playground in this course to recall what these models look like underneath.)
Key takeaway
Supervised summarization optimizes imitation and ROUGE optimizes word overlap — both are proxies. The real target, human preference, needed its own training signal.
Hands-on — experiment before the quiz
Next-Token Playground
How GPT-style models choose every single word
Next-token probabilities
Note: the candidate tokens and their starting scores are curated examples, but the temperature, top-k, top-p, softmax, and sampling math applied to them is exactly the algorithm production LLMs use to pick every word.
Temperature 0.80 keeps a balanced distribution — mostly confident, with some variety.
Analogy: The model builds a weighted raffle for the next word — temperature decides how fair the raffle is, Top-K and Top-P decide who's even allowed to enter.
Checkpoint
Why can't a supervised summarizer exceed its training references?