FoundationalComputer VisionTransformersClassificationFoundational

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

This paper cut photos into little squares, called them 'words', and fed them to the same attention machine that reads sentences — proving Transformers could see as well as they read.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn +8 moreICLR 2021 · 202060k+ citations9 min interactive
The story

The whole paper, in pictures

Five animated scenes — everything you need to understand this paper, no reading required. Then scroll on to play with the ideas yourself.

An Image is Worth 16x16 Words — the picture-book version

No paper-reading needed — the pictures tell the whole story

convolution: one small windowbeak ↔ tail?needs MANY layers to connectevery AI vision model since 2012 has seen photos this way

For 8 years, every vision AI saw the world through a tiny sliding window.

Convolution connects far-apart parts of a photo only slowly, layer after layer — like reading with a magnifying glass held very close.

1/5
Paper overview

Pick your depth

The same ideas at three altitudes — start where you're comfortable, climb when you're curious.

Two ways to look at a picture

A CNN (like AlexNet or ResNet) looks at a photo the way you'd read with a magnifying glass held very close: it slides a tiny window over small patches, one neighborhood at a time, and only slowly — layer after layer — connects far-apart parts of the image. This paper asks: what if we looked at the WHOLE photo at once instead, the way attention lets a sentence's first and last word talk directly?

A photo, cut into 'words'

The trick is almost silly in its simplicity: slice the image into a grid of small squares — 16×16 pixels each — and treat every square like a word in a sentence. A 224×224 photo becomes 196 'words'. Feed that sequence of patches into the exact same Transformer that powers ChatGPT, and self-attention lets any patch look at any other patch, near or far, from the very first layer.

The catch — and the payoff

There's an honest twist: unlike a CNN, this method has no built-in assumption that nearby pixels matter more than far ones — it must LEARN that from scratch. So on a modest dataset it actually loses to CNNs. But given hundreds of millions of training photos, it learns those patterns itself and then matches or beats the best CNNs, while costing less compute to train. Today, this patches-and-attention idea sits behind image search, self-driving perception, and any AI that looks at a picture and understands it in words (like this website's tutor).

Key innovation

Problem → Solution → Impact

Every great paper is a story: what was broken, the idea that fixed it, and what it unlocked.

Problem

Convolution was the only serious way to see

  • CNNs process images through small local windows, connecting distant pixels only after many stacked layers.
  • Attention had transformed language by 2020, but almost nobody had tried removing convolution from vision entirely.
  • It was unclear whether a general-purpose sequence model could match CNNs' hard-won, vision-specific design.

Reading a photo through a moving magnifying glass instead of glancing at the whole thing at once.

Solution

Cut the image into 'words', and use pure attention

  • Slice the photo into a grid of 16×16 patches — 196 of them for a standard 224×224 image.
  • Linearly embed each patch, add a position embedding and a [class] token, and feed the sequence to a standard Transformer encoder.
  • No convolutions anywhere — every patch can attend to every other patch from the very first layer.

Chopping a photo into puzzle pieces, numbering them, and handing the whole pile to the same reading machine that understands sentences.

Impact

Vision became a language problem

  • Matched or beat state-of-the-art CNNs (88.55% ImageNet top-1) once pretrained on hundreds of millions of images, at lower pretraining compute.
  • Became the default visual backbone for multimodal AI — CLIP, and GPT-4V/Gemini-class vision-language models.
  • Spawned an entire lineage: DeiT, Swin, hybrid CNN-ViT designs, and segmentation/detection Transformers.

The moment cameras and keyboards started speaking the same language inside one brain.

Interactive simulator

The same attention, now reading image patches

This is the exact mechanism inside ViT — self-attention connecting distant 'words'. Here they're sentence words; in ViT they're 16×16 image patches. The math doesn't change at all.

Self-Attention Lab

Every word attends to every other word — see it happen

👆 touch a word to see its attention

Note: these attention patterns are an illustration modeled on head behaviors observed in trained Transformers (previous-word heads, coreference heads, …) — not the output of a live model. The equations and mechanism shown are exactly those of the paper.

Hover or tap any word. The lines show where that word is "looking" — this is attention, the paper's big idea.

Analogy: A classroom where every student can listen to every other student at the same time — and each decides who is worth listening to.

Architecture explorer

The full machine, block by block

Follow a sentence through the model. Every block is clickable — purpose, example, and the deeper detail.

Click any block to inspect it

Transformer Encoder

alternating MSA + MLP, ×12 to ×32

The exact same bidirectional encoder architecture as BERT: multi-head self-attention lets every patch attend to every other patch, followed by a per-patch MLP, both wrapped in residual connections and layer norm.

Example

The patch covering the bird's eye can attend directly to the patch covering its tail — one hop, any distance.

Go deeper

ViT-Base: 12 layers, 12 heads, D=768 (~86M params). ViT-Large: 24 layers, 16 heads, D=1024 (~307M). ViT-Huge: 32 layers, 16 heads, D=1280 (~632M). Explore this exact mechanism in the Attention Lab below.

Tip: follow the animated edges top-to-bottom — that is the exact journey a sentence takes through the model.

Math, demystified

Every equation earns its keep

Formula → plain meaning → real-world analogy. Hover the symbols to decode them one by one.

Patch Embedding

Hover a symbol to decode it.

What it means

Flatten each of the N image patches, project them all with the same learned matrix E (exactly like a word-embedding table), stick a learnable [class] token on the front, and add position information. The result is one sequence — indistinguishable, structurally, from a sentence of token embeddings.

Real-world analogy

Turning a jigsaw puzzle into a numbered list of pieces, then handing that list to a reader who processes lists for a living.

The Encoder Block

Hover a symbol to decode it.

What it means

Every layer does exactly two things, each wrapped in a residual shortcut: let patches exchange information via attention, then let each patch think it over independently in an MLP. This is character-for-character the same recipe as the 2017 Transformer and BERT — nothing about it is vision-specific.

Real-world analogy

A committee round (attention: patches compare notes) followed by solo reflection (MLP: each patch updates its own opinion) — repeated a dozen or more times.

How Many 'Words' in a Picture

Hover a symbol to decode it.

What it means

The image's height and width, divided by the patch area, tells you exactly how long the 'sentence' is. Bigger images or smaller patches mean more patches — and since attention compares every patch to every other, cost grows with the SQUARE of this number.

Real-world analogy

A large-print book has fewer, bigger 'chunks' per page than fine print — but here, chunk size (patch size) is a design choice, and the picture is the 'page'.

Classification from the [class] Token

Hover a symbol to decode it.

What it means

After all L encoder layers, only ONE of the 197 output vectors is used for the final answer: position 0, the [class] token — which by then has attended to every patch, repeatedly, and absorbed the whole image into one vector.

Real-world analogy

A committee chair who sat through every discussion round and is the only one asked to summarize the meeting's conclusion.

Interactive playground

The old way ViT replaced

Convolution's small sliding window — the approach every vision model used before ViT. Compare its local view to attention's global one above.

CNN Convolution Lab

How computers see: a tiny magnifying glass scanning a picture

Input

Vertical

Feature map (10×10)

Compares each pixel's left side to its right side. Big difference = a vertical edge lives here. Press play to watch the filter scan the image — this exact multiply-and-add is what every CNN layer does, millions of times.

Analogy: Sliding a small stencil over a photo and asking at every spot: how much does this patch look like my pattern?

Real-world impact

Where this research lives today

Eight pages from 2017 that you now interact with every single day.

Modern image classification

ViT and its descendants (DeiT, Swin, ConvNeXt-inspired hybrids) are now standard vision backbones, often outperforming pure CNNs at scale.

Multimodal AI — CLIP and beyond

CLIP's image encoder is a ViT, trained so images and captions land in the same embedding space — the visual half of nearly every modern vision-language model.

GPT-4V, Gemini and vision-capable assistants

Because patches embed just like text tokens, ViT-style encoders let a single Transformer read an image and a question together — the basis of 'show it a picture and ask about it' AI.

Detection and segmentation

DETR and its successors, plus Swin-based segmentation models, replace hand-designed detection heads with attention over image patches.

Medical and scientific imaging

ViT backbones read X-rays, pathology slides, and satellite imagery, often fine-tuned from the same large-scale pretrained weights this paper introduced.

Video understanding

ViViT and video-Transformer variants extend patches into space-TIME cubes, applying the same 'chop it up, attend over the pieces' idea to moving images.

Code example

The core idea in ~40 lines of code

Reading real code is the final test of understanding. This is the paper's core mechanism, minimally.

A minimal Vision Transformerpython

The whole idea in code: chop the image into patches, embed them like tokens, prepend a [class] token, add position embeddings, and hand the sequence to a standard Transformer encoder. Everything after patchify() is generic sequence modeling — no convolutions required.

1import torch
2import torch.nn as nn
3 
4 
5class ViT(nn.Module):
6    """Dosovitskiy et al., 2020 — patches in, attention does the rest."""
7 
8    def __init__(self, img_size=224, patch_size=16, in_ch=3,
9                 dim=768, depth=12, heads=12, mlp_dim=3072,
10                 num_classes=1000):
11        super().__init__()
12        n_patches = (img_size // patch_size) ** 2   # 196 for 224/16
13 
14        # Patchify + linear projection in ONE conv call:
15        # a stride-P, kernel-P convolution IS patch extraction + embedding.
16        self.patch_embed = nn.Conv2d(in_ch, dim, kernel_size=patch_size,
17                                      stride=patch_size)
18 
19        self.cls_token = nn.Parameter(torch.zeros(1, 1, dim))
20        self.pos_embed = nn.Parameter(torch.zeros(1, n_patches + 1, dim))
21 
22        # Standard pre-norm Transformer encoder -- identical machinery
23        # to a text Transformer, just fed patches instead of words.
24        layer = nn.TransformerEncoderLayer(
25            d_model=dim, nhead=heads, dim_feedforward=mlp_dim,
26            activation="gelu", norm_first=True, batch_first=True,
27        )
28        self.encoder = nn.TransformerEncoder(layer, num_layers=depth)
29        self.head = nn.Linear(dim, num_classes)
30 
31    def forward(self, x: torch.Tensor) -> torch.Tensor:
32        B = x.shape[0]
33        x = self.patch_embed(x)              # (B, dim, 14, 14)
34        x = x.flatten(2).transpose(1, 2)     # (B, 196, dim) -- 196 "words"
35 
36        cls = self.cls_token.expand(B, -1, -1)
37        x = torch.cat([cls, x], dim=1)       # (B, 197, dim)
38        x = x + self.pos_embed                # learned position info
39 
40        x = self.encoder(x)                   # every patch attends to every patch
41        return self.head(x[:, 0])             # classify from the [class] token
42 
43 
44vit = ViT()
45out = vit(torch.randn(1, 3, 224, 224))
46print(out.shape)      # torch.Size([1, 1000])
Check yourself

Did it stick?

Four questions, each targeting a concept people commonly get wrong. Mistakes come with explanations.

Question 1 of 4

What built-in assumption do convolutional filters make?

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale · PaperLab