Vision Transformer: teaching attention to see
A 4-lesson mini course: what convolution assumes, turning pictures into words, seeing globally from layer one, and the data-hungry trade-off.
What convolution assumes
A built-in assumption
Every convolutional filter — from AlexNet's edge detectors to ResNet's deep stacks — makes one hard-coded assumption: nearby pixels are related, and that relationship looks the same everywhere in the image (translation equivariance). That assumption is usually true for photos, and it's why CNNs learned to see so well with relatively modest data.
The cost of an assumption
A built-in assumption is also a limitation: a CNN can't easily learn a DIFFERENT rule even if the data suggests one, and connecting two far-apart pixels still requires stacking enough layers for the receptive field to grow that large. Play with the CNN Lab in this lesson and notice: one filter only ever looks at its small window.
The question this paper asks
What if, instead of hard-coding 'nearby pixels matter', we let the model learn whatever spatial relationships the data actually shows it — using the same attention mechanism already proven on text? The cost: the model starts with no assumptions at all, and must learn structure entirely from examples.
Key takeaway
CNNs hard-code locality and translation equivariance — a helpful shortcut, but a fixed one. ViT removes the shortcut entirely and learns spatial structure from data instead.
Hands-on — experiment before the quiz
CNN Convolution Lab
How computers see: a tiny magnifying glass scanning a picture
Input
Feature map (10×10)
Compares each pixel's left side to its right side. Big difference = a vertical edge lives here. Press play to watch the filter scan the image — this exact multiply-and-add is what every CNN layer does, millions of times.
Analogy: Sliding a small stencil over a photo and asking at every spot: how much does this patch look like my pattern?
Checkpoint
What built-in assumption do convolutional filters make?