This paper replaced 'imitate the reference answer' with 'optimize what humans actually prefer' — the RLHF recipe that later turned raw GPT models into ChatGPT and Claude.
Five animated scenes — everything you need to understand this paper, no reading required. Then scroll on to play with the ideas yourself.
Learning to Summarize from Human Feedback — the picture-book version
No paper-reading needed — the pictures tell the whole story
AI could write fluently — but nobody had told it what GOOD means.
Copying example summaries has a ceiling, and 'good' has no formula. What we really want: outputs humans prefer.
The same ideas at three altitudes — start where you're comfortable, climb when you're curious.
How do you teach an AI to summarize well? The old way: show it thousands of human-written summaries and train it to copy them. But copying has a ceiling — the model can never be better than its examples, and 'similar words to the reference' isn't the same as 'actually a good summary'. What we really want is for humans to READ the outputs and say which ones are good. This paper builds exactly that.
Paying humans to rate every single output during training would be impossibly slow. The trick: collect tens of thousands of human choices ('summary A is better than B'), then train a second AI — a reward model — to predict those choices. Now you have a tireless robot judge that approximates human taste, and the summarizer can practice against it millions of times.
One danger: if the summarizer optimizes the robot judge too hard, it finds the judge's blind spots — outputs that score high but are actually garbage. It's like a student learning to game the grading rubric instead of learning the subject. The fix is a leash (a 'KL penalty') that punishes the model for drifting too far from normal writing. The lab on this page lets you loosen that leash and watch the cheating begin.
Every great paper is a story: what was broken, the idea that fixed it, and what it unlocked.
Grading essays by how many words they share with a model answer — fluent copying wins, insight goes unrewarded.
Train a tireless robot critic from human taste tests, let the writer practice against it a million times — while keeping the writer on a leash so it can't just flatter the critic.
The missing translation layer between 'what AI can do' and 'what people actually want' — the reason talking to AI works.
Drag the KL penalty and watch the policy shift between playing it safe, genuinely improving, and reward hacking — using the true closed-form optimum of KL-regularized RL.
Why aligned models need a KL penalty — feel the trade-off
Post to summarize
My neighbor's cat sneaks into my house every day through the window, eats my cat's food, and then naps on my couch. My own cat just sits there and watches it happen. Should I say something to my neighbor about it?
Candidate summaries — policy probability
○ = supervised baseline's probability
A.Neighbor's cat sneaks in daily, eats my cat's food and naps on my couch — my cat doesn't mind. Should I talk to the neighbor?
r = 2.6
46.7%
B.A neighbor's cat keeps coming into my house and eating food. Asking if I should tell the neighbor.
r = 1.7
42.3%
C.I have a cat and a neighbor who also has a cat.
r = -0.5
8.5%
D.Cats! There are cats everywhere in my house, all day long.
r = -1.2
0.4%
E.Excellent heartwarming summary!!! 10/10 amazing best post reward reward reward
r = 6.5
2.1%
Note: the summaries and their scores are curated; the policy distribution, KL divergence, and the emergence of reward hacking come from the real KL-regularized-RL optimum π ∝ π_SFT·exp(r/β) — the exact trade-off the paper tunes with its KL coefficient.
Sweet spot: at β = 1.00 the policy shifts toward genuinely better summaries — “A” leads — while the KL penalty (0.70 nats) keeps it from wandering into the reward model's blind spots.
Analogy: Training a dog with treats (the reward model) while keeping it on a leash (the KL penalty). Drop the leash entirely and the dog stops herding sheep and starts mugging you for treats.
Follow a sentence through the model. Every block is clickable — purpose, example, and the deeper detail.
Click any block to inspect it
“A or B?” × tens of thousands
Labelers read a post and two candidate summaries and pick the better one. Comparisons — not scores — because 'better than' is far more consistent across people than '7/10'.
Example
Summary A captures the actual question asked; B is fluent but misses it → A.
Go deeper
The paper invested heavily in labeler quality: onboarding, feedback loops, and measured researcher–labeler agreement — arguing data quality mattered as much as quantity.
Tip: follow the animated edges top-to-bottom — that is the exact journey a sentence takes through the model.
Formula → plain meaning → real-world analogy. Hover the symbols to decode them one by one.
Hover a symbol to decode it.
What it means
Train the judge from comparisons: for each human choice, push the winning summary's score above the loser's. Only the DIFFERENCE matters — humans said 'better than', not '7 out of 10', because relative judgments are what people give consistently.
Real-world analogy
A wine critic trained purely on taste-offs: never asked for absolute ratings, only 'which of these two is better' — thousands of times, until their palate mirrors the tasters'.
Hover a symbol to decode it.
What it means
The policy's true reward: the judge's score MINUS a penalty for writing text the supervised baseline finds improbable. β sets the leash length — the central knob of all RLHF.
Real-world analogy
A performance bonus (reward) minus a fine for every step taken away from the trail (KL). Wander far only if the bonus truly pays for it.
Hover a symbol to decode it.
What it means
KL-regularized RL has an exact solution: reweight the baseline's distribution by exponentiated reward. High-reward outputs get boosted; how much depends on β. This single formula is what the RLHF Lab on this page computes live — and what DPO later turned into a direct training loss.
Real-world analogy
The baseline model proposes; the reward whispers 'more of this one, less of that one'; β decides how loudly the whisper counts.
Hover a symbol to decode it.
What it means
The paper's cautionary measurement: push optimization past the sweet spot and the proxy score keeps climbing while real human-judged quality falls. The judge's blind spots become the policy's habits.
Real-world analogy
Goodhart's law, verbatim: when a measure becomes a target, it ceases to be a good measure. The rubric-gaming student, at scale.
Underneath RLHF sits an ordinary next-token predictor. These sampling controls are the raw material that human feedback then shapes.
How GPT-style models choose every single word
Next-token probabilities
Note: the candidate tokens and their starting scores are curated examples, but the temperature, top-k, top-p, softmax, and sampling math applied to them is exactly the algorithm production LLMs use to pick every word.
Temperature 0.80 keeps a balanced distribution — mostly confident, with some variety.
Analogy: The model builds a weighted raffle for the next word — temperature decides how fair the raffle is, Top-K and Top-P decide who's even allowed to enter.
Eight pages from 2017 that you now interact with every single day.
InstructGPT applied this paper's exact three-stage pipeline to instruction following; ChatGPT productized it. Assistant training at every lab descends from this recipe.
Reward hacking, Goodharting, scalable oversight — this paper's measured over-optimization curves seeded core research agendas in AI alignment.
Direct Preference Optimization derives its loss from the same closed-form optimum this paper's KL setup implies — collapsing the pipeline into one supervised step.
Human comparison collection — labeler pipelines, agreement metrics, quality control — became an industry because this paper showed data quality drives alignment quality.
RLHF now aligns image generators to aesthetic preferences, code models to correctness-and-style, and robots to human comfort — anywhere 'good' is easier to judge than to specify.
RLAIF and constitutional methods swap the human judge for an AI judge guided by principles — inheriting this paper's machinery while attacking its labeling bottleneck.
Reading real code is the final test of understanding. This is the paper's core mechanism, minimally.
The whole pipeline as runnable-shaped pseudocode: imitate, learn a judge from comparisons, then optimize against the judge — with the KL leash that keeps it honest. Swap 'summarize' for 'follow instructions' and this is how assistants are trained.
1import torch2import torch.nn.functional as F34# ---- Stage 1: Supervised fine-tuning (imitate human TL;DRs) ----5def sft_step(policy, post, human_summary):6 logits = policy(post, human_summary)7 return F.cross_entropy(logits, human_summary.tokens) # plain imitation8910# ---- Stage 2: Reward model from human comparisons ----11def reward_model_step(rm, post, winner, loser):12 # Bradley-Terry: the human-preferred summary must score higher13 gap = rm(post, winner) - rm(post, loser)14 return -F.logsigmoid(gap) # loss = -log sigma(r_w - r_l)151617# ---- Stage 3: RL against the judge, on a KL leash ----18def rlhf_reward(rm, policy, sft_policy, post, summary, beta=0.05):19 proxy = rm(post, summary) # what the judge thinks2021 # The leash: pay for every token the SFT baseline finds weird22 kl = (policy.logprob(summary, post)23 - sft_policy.logprob(summary, post))2425 return proxy - beta * kl # the objective PPO maximizes262728# The closed-form optimum of stage 3 (what the lab visualizes):29# pi*(y|x) ~ pi_sft(y|x) * exp( r(x,y) / beta )30#31# beta -> 0: highest-reward output takes everything -- including32# reward-model MISTAKES. That's reward hacking.33# beta -> inf: policy never moves; human feedback wasted.34# The art of RLHF is the leash length in between.
Four questions, each targeting a concept people commonly get wrong. Mistakes come with explanations.
Question 1 of 4
Why can't a supervised summarizer exceed its training references?