This paper taught AI to read in both directions at once — by playing fill-in-the-blank on the internet — and instantly broke the record on eleven language tasks.
Five animated scenes — everything you need to understand this paper, no reading required. Then scroll on to play with the ideas yourself.
BERT — the picture-book version
No paper-reading needed — the pictures tell the whole story
BERT learns language the way you did: fill in the blank!
“She played the ____ on stage last night.” Easy for you. Now let's see why it was hard for computers.
The same ideas at three altitudes — start where you're comfortable, climb when you're curious.
GPT-style models read like a flashlight moving left to right — when guessing a word, they only see what came before it. BERT reads like the room lights are on: for the blank in 'She played the ___ on stage last night', it sees 'on stage last night' too, and instantly knows the blank is an instrument. Both-sides context is often the whole clue.
How do you teach both-directions reading? Hide words and make the model guess them. BERT's training covers 15% of the words in billions of sentences with a [MASK] sticker and grades the guesses. No human labeling needed — the text itself is the answer key. Try the Masked Word Lab below to play the exact same game.
After this fill-in-the-blank school, BERT deeply 'gets' language. Then comes the magic: bolt one tiny layer on top and briefly tune it, and the same brain answers questions, rates reviews, or detects paraphrases — eleven records broken with one recipe. Within a year it was reading every Google search you typed.
Every great paper is a story: what was broken, the idea that fixed it, and what it unlocked.
Solving a crossword while only allowed to read each clue's first half.
Fill-in-the-blank worksheets generated for free from the entire internet — with the answer key built in.
One universally educated brain, given a one-day orientation for each new job — instead of raising a specialist from scratch every time.
Fill-in-the-blank with X-ray vision: move the mask, then toggle between left-only reading (GPT) and both-sides reading (BERT) — and watch the confidence jump.
BERT's training game: fill in the blank — with X-ray vision
Blue-bordered words are maskable — click one to move the blank.
The model's guesses for [MASK]
Note: candidate words and their scores are curated illustrations of real BERT behavior (the softmax over them is computed live); the left-vs-both comparison is the paper's actual case for bidirectional pre-training.
Seeing BOTH sides (BERT-style), the words after the blank give it away — “chef” jumps to 61%, versus 27% confidence with left context alone. That confidence gap is the entire argument for bidirectional pre-training.
Analogy: A crossword clue you can read from both ends: knowing the words after the blank is often worth more than everything before it.
Follow a sentence through the model. Every block is clickable — purpose, example, and the deeper detail.
Click any block to inspect it
15% → 80/10/10
The training game: 15% of tokens are chosen as targets. 80% become [MASK], 10% become a random word, 10% are left alone — and the model must reconstruct the originals.
Example
"The chef cooked a [MASK] in the kitchen" → predict "meal".
Go deeper
The 10/10 tricks exist because [MASK] never appears at fine-tuning time — the model must stay suspicious of every token, keeping representations useful for real inputs.
Tip: follow the animated edges top-to-bottom — that is the exact journey a sentence takes through the model.
Formula → plain meaning → real-world analogy. Hover the symbols to decode them one by one.
Hover a symbol to decode it.
What it means
Only the hidden tokens are graded: maximize the probability of each original token given the corrupted sentence. Because the corrupted sentence contains BOTH sides of every blank, the model is forced to learn bidirectional understanding.
Real-world analogy
A worksheet where a few words are covered by stickers — you're only graded on the stickers, but to fill them you must genuinely read the whole sentence.
Hover a symbol to decode it.
What it means
The original Transformer attention with the causal mask simply removed: every token's query scores against every key in the sentence, before AND after it. One deleted mask is the entire architectural difference from GPT.
Real-world analogy
GPT reads through a moving slot that hides the future; BERT reads the page whole. Same eyes, no slot.
Hover a symbol to decode it.
What it means
A binary classifier on the [CLS] vector: did sentence B really follow sentence A (true 50% of the time), or was it randomly sampled? This teaches relationships BETWEEN sentences, which single-sentence masking can't.
Real-world analogy
A shuffled-story quiz: 'does this sentence belong right after that one?' — you can only pass by understanding how ideas connect.
Hover a symbol to decode it.
What it means
Of the 15% of selected tokens: most become [MASK], some become a wrong word, some stay correct — but all must be predicted. Since [MASK] never appears in real downstream text, the model must maintain rich representations for EVERY token, not just stickers.
Real-world analogy
A teacher who mostly covers words, sometimes swaps in a wrong word, and sometimes changes nothing — so students learn to check every word instead of only the covered ones.
The Attention Lab's every-word-to-every-word view IS BERT's encoder. GPT sees this picture with all rightward connections cut.
Every word attends to every other word — see it happen
👆 touch a word to see its attention
Note: these attention patterns are an illustration modeled on head behaviors observed in trained Transformers (previous-word heads, coreference heads, …) — not the output of a live model. The equations and mechanism shown are exactly those of the paper.
Hover or tap any word. The lines show where that word is "looking" — this is attention, the paper's big idea.
Analogy: A classroom where every student can listen to every other student at the same time — and each decides who is worth listening to.
Eight pages from 2017 that you now interact with every single day.
Deployed in 2019 to understand queries bidirectionally — famously fixing searches where a preposition like 'to' or 'for' flips the meaning. It initially affected ~10% of English queries.
Sentiment analysis, intent detection, content moderation, spam filtering — encoder models descended from BERT remain the workhorses for classify-and-understand tasks.
Multilingual BERT and its successors (XLM-R) brought pre-trained understanding to 100+ languages from a single model.
Dense retrieval — embedding queries and documents so meaning, not keywords, matches — grew directly out of BERT-style encoders. Modern RAG systems' retrievers are its grandchildren.
The masked-prediction recipe transferred to proteins (ESM), code, and chemistry — mask part of a sequence, predict it, learn deep structure without labels.
RoBERTa (better training), ALBERT (parameter sharing), ELECTRA (replaced-token detection), DeBERTa (disentangled attention) — an entire lineage of encoders iterating on this paper's recipe.
Reading real code is the final test of understanding. This is the paper's core mechanism, minimally.
The pre-training data pipeline in miniature: corrupt a sentence with the 80/10/10 recipe, then train the encoder to reconstruct the hidden words. This loop, run over 3.3 billion words, is all of BERT's schooling.
1import random23MASK, VOCAB = "[MASK]", ["meal", "song", "cake", "plan", "dish"] # toy vocab456def corrupt(tokens: list[str], mask_rate: float = 0.15):7 """BERT's 80/10/10 masking recipe. Returns (corrupted, targets)."""8 corrupted, targets = [], {}9 for i, tok in enumerate(tokens):10 if random.random() < mask_rate:11 targets[i] = tok # we must predict the ORIGINAL12 roll = random.random()13 if roll < 0.8:14 corrupted.append(MASK) # 80%: hide it15 elif roll < 0.9:16 corrupted.append(random.choice(VOCAB)) # 10%: wrong word17 else:18 corrupted.append(tok) # 10%: leave it (stay suspicious!)19 else:20 corrupted.append(tok)21 return corrupted, targets222324def training_step(encoder, tokens):25 corrupted, targets = corrupt(tokens)2627 # The encoder sees the FULL corrupted sentence — both sides of28 # every blank. No causal mask. This is the bidirectional part.29 hidden = encoder(corrupted) # (seq_len, d_model)3031 loss = 0.032 for pos, original in targets.items():33 # Predict the original token from its two-sided context34 probs = softmax(hidden[pos] @ encoder.vocab_matrix.T)35 loss += -log(probs[vocab_id(original)])36 return loss / max(1, len(targets))373839sentence = "the chef cooked a delicious meal in the kitchen".split()40# corrupt() might yield: the chef cooked a delicious [MASK] in the kitchen41# ...and the model learns that both "delicious" AND "kitchen" point to "meal".
Four questions, each targeting a concept people commonly get wrong. Mistakes come with explanations.
Question 1 of 4
Why couldn't a deep bidirectional model just train on next-word prediction?