This paper introduced the Transformer — the architecture that replaced recurrent networks with pure attention and became the foundation of GPT, BERT, and modern AI.
Five animated scenes — everything you need to understand this paper, no reading required. Then scroll on to play with the ideas yourself.
Attention Is All You Need — the picture-book version
No paper-reading needed — the pictures tell the whole story
Before 2017, computers read like this: one word at a time.
Slow — and each new word pushed the old ones further out of memory.
The same ideas at three altitudes — start where you're comfortable, climb when you're curious.
Imagine a classroom where students must pass a message down a line, one person at a time. By the end, the message is garbled — that's how older AI models (RNNs) read sentences: word by word, slowly forgetting the beginning. The Transformer changes the classroom: now every student can listen to every other student at the same time. Each word in a sentence can directly 'look at' every other word and decide which ones matter for understanding it.
When you read the sentence 'The animal didn't cross the street because it was too tired', your brain instantly knows 'it' means the animal, not the street. You paid attention to the right word. The Transformer gives AI that same skill — a mathematical way for each word to focus on the words that explain it.
Because every word is processed at the same time instead of one-by-one, Transformers train dramatically faster on modern hardware (GPUs love parallel work). Faster training meant researchers could build much bigger models — and those bigger models became GPT, ChatGPT, and almost every AI system you use today.
Every great paper is a story: what was broken, the idea that fixed it, and what it unlocked.
Like passing a message down a long line of people — slow, and the start of the message gets garbled.
Like a meeting where everyone hears everyone at once, and each person decides who to listen to.
The 'internal combustion engine' moment of AI — one design that everything since is built on.
This is the paper's core idea, running live. Hover words, switch heads, and watch the model decide what matters.
Every word attends to every other word — see it happen
👆 touch a word to see its attention
Note: these attention patterns are an illustration modeled on head behaviors observed in trained Transformers (previous-word heads, coreference heads, …) — not the output of a live model. The equations and mechanism shown are exactly those of the paper.
Hover or tap any word. The lines show where that word is "looking" — this is attention, the paper's big idea.
Analogy: A classroom where every student can listen to every other student at the same time — and each decides who is worth listening to.
Follow a sentence through the model. Every block is clickable — purpose, example, and the deeper detail.
Click any block to inspect it
every word looks at every word
The heart of the paper. Each word asks a question (Query), every word advertises what it offers (Key), and information flows from the best matches (Values). Eight heads run in parallel, each learning a different kind of relationship.
Example
For 'it' in "The animal didn't cross the street because it was too tired", one attention head links 'it' strongly to 'animal'.
Go deeper
Attention(Q,K,V) = softmax(QKᵀ/√d_k)V. With 8 heads of dimension 64, one head may track subject–verb agreement, another coreference, another adjacent-word syntax.
Tip: follow the animated edges top-to-bottom — that is the exact journey a sentence takes through the model.
Formula → plain meaning → real-world analogy. Hover the symbols to decode them one by one.
Hover a symbol to decode it.
What it means
For each word, compare its Query against every word's Key to get relevance scores, scale them down, convert to percentages with softmax, then blend the Values using those percentages.
Real-world analogy
You're at a library with a question (Q). Every book has a title on its spine (K) and content inside (V). You compare your question to every title, decide how relevant each book is, then read a mix of the books weighted by relevance.
Hover a symbol to decode it.
What it means
Instead of one attention computation, run 8 smaller ones in parallel — each with its own learned Q/K/V projections — then concatenate the results and mix them with one final matrix.
Real-world analogy
Eight specialists read the same sentence: a grammar expert, a pronoun-tracker, a rhyme detector… Each reports what they noticed, and a manager (Wᴼ) combines their reports into one summary.
Hover a symbol to decode it.
What it means
Each position in the sentence gets a unique fingerprint built from sine and cosine waves at different frequencies, which is added to the word's embedding.
Real-world analogy
Like a clock with many hands moving at different speeds — hour, minute, second. Any moment in time is uniquely identified by the combination of all hand positions. Here, every word position gets its own unique 'clock reading'.
Hover a symbol to decode it.
What it means
Each word's vector is independently expanded to 2048 dimensions, passed through ReLU (which zeroes out negatives), and compressed back to 512. Same weights for every position.
Real-world analogy
After the group discussion (attention), each student goes home to process their notes alone. Everyone uses the same 'thinking method', but on their own notes.
Transformers became GPT by predicting the next token. Play with temperature and top-k to feel how generation really works.
How GPT-style models choose every single word
Next-token probabilities
Note: the candidate tokens and their starting scores are curated examples, but the temperature, top-k, top-p, softmax, and sampling math applied to them is exactly the algorithm production LLMs use to pick every word.
Temperature 0.80 keeps a balanced distribution — mostly confident, with some variety.
Analogy: The model builds a weighted raffle for the next word — temperature decides how fair the raffle is, Top-K and Top-P decide who's even allowed to enter.
Eight pages from 2017 that you now interact with every single day.
Every modern large language model is a Transformer. GPT models are decoder-only Transformers scaled to billions of parameters — architecturally, they're this paper minus the encoder.
BERT — an encoder-only Transformer — has powered Google Search query understanding since 2019, affecting billions of searches daily.
The paper's original task. Google Translate, DeepL, and virtually all production translation systems moved to Transformer backbones within two years of publication.
Vision Transformers (ViT, 2020) treat image patches as 'words', beating convolutional networks at scale. They power modern image classification, detection, and generation.
DeepMind's protein-structure breakthrough — which earned a share of the 2024 Nobel Prize in Chemistry — is built on attention over amino-acid sequences.
Code-generation models like Copilot and Claude Code are Transformers trained on source code — attention tracks variables and scope across a file the way it tracks pronouns in a sentence.
Reading real code is the final test of understanding. This is the paper's core mechanism, minimally.
This is a minimal, readable implementation of the paper's core idea — scaled dot-product self-attention with multiple heads. Real implementations add dropout, masking, and fused kernels, but this is the essence.
1import torch2import torch.nn as nn3import torch.nn.functional as F456class MultiHeadSelfAttention(nn.Module):7 """The core mechanism from 'Attention Is All You Need'."""89 def __init__(self, d_model: int = 512, n_heads: int = 8):10 super().__init__()11 assert d_model % n_heads == 012 self.d_k = d_model // n_heads # 64 per head, as in the paper13 self.n_heads = n_heads1415 # Learned projections for Query, Key, Value (+ output mix)16 self.w_q = nn.Linear(d_model, d_model)17 self.w_k = nn.Linear(d_model, d_model)18 self.w_v = nn.Linear(d_model, d_model)19 self.w_o = nn.Linear(d_model, d_model)2021 def forward(self, x: torch.Tensor) -> torch.Tensor:22 # x: (batch, seq_len, d_model)23 B, T, C = x.shape2425 # 1. Project into Q, K, V and split into heads26 # (batch, heads, seq_len, d_k)27 q = self.w_q(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)28 k = self.w_k(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)29 v = self.w_v(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)3031 # 2. Attention scores: how relevant is every word to every word?32 # scaled by sqrt(d_k) so softmax stays in a healthy range33 scores = (q @ k.transpose(-2, -1)) / (self.d_k ** 0.5)3435 # 3. Softmax -> attention weights that sum to 1 per word36 weights = F.softmax(scores, dim=-1)3738 # 4. Weighted sum of values: each word gathers information39 # from the words it attends to40 context = weights @ v # (B, heads, T, d_k)4142 # 5. Merge heads back together and mix43 context = context.transpose(1, 2).reshape(B, T, C)44 return self.w_o(context)454647# --- Try it ---48attn = MultiHeadSelfAttention(d_model=512, n_heads=8)49sentence = torch.randn(1, 10, 512) # batch of 1, 10 tokens50out = attn(sentence)51print(out.shape) # torch.Size([1, 10, 512])
Four questions, each targeting a concept people commonly get wrong. Mistakes come with explanations.
Question 1 of 4
Why couldn't RNNs take full advantage of GPUs?