The Transformer, from zero
A 4-lesson mini course that takes you from 'why old models failed' to building attention in code.
Why RNNs failed
One word at a time
Before 2017, language AI read text the way you read a ticker tape: strictly left to right, one word per step. A Recurrent Neural Network (RNN) keeps a single 'memory' vector that gets updated after each word. By the time it reaches word 50, that memory must somehow still contain word 1 — and mostly, it doesn't.
Two fatal problems
First: long-range forgetting. Gradients shrink as they flow backward through many steps ('vanishing gradients'), so the model can't learn connections between distant words. Second: no parallelism. Step 50 needs step 49's result, so a GPU with thousands of cores sits mostly idle. Training was slow, and slow training capped how big models could get.
The patch that hinted at the cure
In 2015, researchers bolted an 'attention' mechanism onto RNN translators: while producing each output word, the model could glance back at all input words. It helped enormously — and buried in that fix was a radical question: if attention does the heavy lifting, do we need the RNN at all?
Key takeaway
RNNs were slow (sequential) and forgetful (vanishing gradients). Attention started as a patch for RNNs — this paper made it the whole engine.
Hands-on — experiment before the quiz
Gradient Descent Lab
How neural networks actually learn
Press play to drop the ball! It always rolls downhill toward lower loss — that is how a neural network learns. Can you get it to the flag?
Analogy: A ball rolling down a foggy mountain: it can only feel the slope right under its feet, and the learning rate is the size of each hop.
Checkpoint
Why couldn't RNNs take full advantage of GPUs?