ResNet: why depth broke, and the one line that fixed it
A 4-lesson mini course: the degradation mystery, the residual idea, the gradient highway, and the 152-layer machine it enabled.
The mystery: deeper was worse
The obvious plan stopped working
By 2015, the recipe for better vision models was 'add layers': 8-layer AlexNet gave way to 19-layer VGG and 22-layer GoogLeNet. But pushing further backfired: on CIFAR-10, a 56-layer plain network had higher error than a 20-layer one — and crucially, higher TRAINING error. It wasn't overfitting. The deep network was failing to learn at all.
Why this made no sense
Here's the paradox the paper opens with: a 56-layer network can trivially imitate a 20-layer one — copy the 20 layers, and make the other 36 do nothing (pass their input through unchanged). So a deeper network should NEVER be worse. The solution exists; gradient descent just couldn't find it. That makes degradation an optimization problem, not a capacity problem.
See it yourself
The lab below reproduces this at toy scale, for real: two networks of identical depth train on the same data — one plain, one residual. Watch the plain one flatline while its twin learns. The only difference between them is one addition per layer.
Key takeaway
Deeper plain networks had HIGHER training error — a failure of optimization, not overfitting. 'Do nothing' should have been easy for extra layers, but solvers couldn't learn it.
Hands-on — experiment before the quiz
ResNet Lab: the skip-connection race
Two real deep networks train live — one plain, one residual
Plain · 12 layers
loss — · acc —
ResNet · 12 layers + skips
loss — · acc —
The race — loss over time
100% real: both networks genuinely train in your browser with exact backpropagation (gradients verified against finite differences to ~1e-10). The residual network computes literally y = F(x) + x, the equation from the paper. Same initialization luck, same data, same learning rate — the skips are the only difference.
Two identical 12-layer networks, same data, same starting luck. The ONLY difference: the teal one adds its input back to every layer's output — y = F(x) + x, the paper's one-line idea. Press play and watch them race.
Analogy: Passing a message through 12 rooms of people. Plain network: each room must re-tell everything from scratch. ResNet: each room gets the original note and just adds a correction to it.
Checkpoint
Why did the 56-layer plain network's HIGHER TRAINING error matter so much?