BERT: reading in both directions
A 4-lesson mini course: why one-way reading fails, the masking game, the encoder machinery, and the fine-tuning revolution.
The one-way problem
Two ways to read
By 2018 there were two pre-training camps. GPT read strictly left-to-right — powerful for generating text, but when representing a word it was blind to everything after it. ELMo ran one forward LSTM and one backward LSTM and glued their outputs together — technically two directions, but each reading was still one-eyed; they never truly combined.
Why understanding needs both sides
Try it in the Masked Word Lab below: for 'She played the ___ on stage last night', left context alone leaves you torn between game, piano, and role. The words AFTER the blank — 'on stage' — are the giveaway. Question answering, entity tagging, and inference are full of cases where the decisive clue sits to the right.
The blocker
So why not just train a deep bidirectional model on next-word prediction? Because it cheats: with both directions flowing through multiple attention layers, each word can indirectly 'see itself' and the objective collapses. BERT's masking trick is exactly the workaround — and the next lesson plays it.
Key takeaway
One-way reading misses right-side clues; shallow gluing (ELMo) doesn't fix it; naive deep bidirectional LM training self-leaks. A new objective was needed.
Hands-on — experiment before the quiz
Masked Word Lab
BERT's training game: fill in the blank — with X-ray vision
Blue-bordered words are maskable — click one to move the blank.
The model's guesses for [MASK]
Note: candidate words and their scores are curated illustrations of real BERT behavior (the softmax over them is computed live); the left-vs-both comparison is the paper's actual case for bidirectional pre-training.
Seeing BOTH sides (BERT-style), the words after the blank give it away — “chef” jumps to 61%, versus 27% confidence with left context alone. That confidence gap is the entire argument for bidirectional pre-training.
Analogy: A crossword clue you can read from both ends: knowing the words after the blank is often worth more than everything before it.
Checkpoint
Why couldn't a deep bidirectional model just train on next-word prediction?