Convolutional Networks in Depth
Why convolution works for images, the shape arithmetic you need every day (padding, stride, channels, receptive field), pooling and 1×1 convolutions, and the architectures from LeNet to ResNet and ConvNeXt.
Why convolution
A dense layer on a 224 × 224 × 3 image would need 150,528 weights per hidden unit, and would have to learn "an edge" separately at every position. Images have structure that a convolution exploits:
- Locality: nearby pixels are related, so each output looks only at a small window (e.g. 3 × 3).
- Weight sharing: the same small filter slides over every position, so a pattern learned in one place is detected everywhere. A 3 × 3 filter on RGB has 27 weights + 1 bias, whatever the image size.
- Translation equivariance: shift the input, and the feature map shifts the same way. Pooling then adds some invariance to small shifts.
Stacking convolutions builds a hierarchy: early layers detect edges and colours, middle layers textures and parts, and late layers whole objects. The core course's architectures topic shows a filter sliding over an image; this topic is about the arithmetic and design decisions.
Watch it: Padding and stride: how a convolution sets the output size
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
The shape arithmetic
A Conv2d(C_in, C_out, kernel_size=k, stride=s, padding=p) layer has C_out filters, each of size k × k × C_in. Each filter produces one output channel. For an input of height (or width) n:
- Padding p = (k − 1)/2 with stride 1 ("same" padding) keeps the size: a 3 × 3 conv with p = 1.
- Stride 2 halves the size: the standard way to downsample in modern CNNs.
- Channels usually double when the size halves (64 → 128 → 256 → 512), keeping the compute per layer similar.
The receptive field is the region of the input that can influence one output unit. Each 3 × 3 conv adds 2 pixels (times the product of earlier strides), so two stacked 3 × 3 convs see 5 × 5 and three see 7 × 7, with fewer parameters and more nonlinearity than one big filter: the VGG insight.
Try it: conv arithmetic calculator
Build a small stack of layers and see the output shape, parameters and receptive field at each step. An invalid configuration (output below 1) is flagged.
Interactive: this part runs in your browser.
Pooling, 1×1 and depthwise convolutions
Max pooling (2 × 2, stride 2) keeps the largest value in each window: it halves the size and adds a little shift invariance. Global average pooling averages each channel over all positions, turning (B, C, H, W) into (B, C) before the final linear classifier; it replaced the huge dense layers of early networks and lets the network accept any image size.
A 1 × 1 convolution mixes channels at each position without looking at neighbours: a cheap way to change the number of channels (the "bottleneck" in ResNet). A depthwise separable convolution splits a normal convolution into a per-channel spatial filter followed by a 1 × 1 channel mix, cutting computation by roughly k² (about 9×): the basis of MobileNet and EfficientNet for phones and edge devices.
Architectures that shaped the field
| Year | Network | Key idea |
|---|---|---|
| 1998 | LeNet-5 | Conv → pool → conv → pool → dense: read handwritten cheques. |
| 2012 | AlexNet | Deep CNN on GPUs with ReLU and dropout; won ImageNet by a large margin and started the deep-learning boom. |
| 2014 | VGG | Only 3 × 3 convs, stacked deep: simple and uniform. |
| 2015 | ResNet | Residual connections and BatchNorm: 50 to 152 layers train reliably. Still a standard baseline and backbone. |
| 2019 | EfficientNet | Scales depth, width and resolution together; depthwise convs. |
| 2020 | Vision Transformer (ViT) | Split the image into patches and use a transformer (see the Computer Vision track). |
| 2022 | ConvNeXt | A CNN modernised with transformer-era design choices; competitive with ViTs. |
In practice you rarely design a CNN from scratch: you take a pretrained backbone (ResNet-50, EfficientNet, ConvNeXt) and fine-tune it (see Transfer learning).
Code example: convolution and pooling from scratch
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code example: convolution features for scanned digits
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: a small CNN in PyTorch
A compact, modern-style CNN for 1 × 28 × 28 images, with the shape after each stage in comments.
import torch, torch.nn as nn
def block(cin, cout):
return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1, bias=False), nn.BatchNorm2d(cout), nn.ReLU(),
nn.Conv2d(cout, cout, 3, padding=1, bias=False), nn.BatchNorm2d(cout), nn.ReLU())
model = nn.Sequential(
block(1, 32), nn.MaxPool2d(2), # (B, 32, 14, 14)
block(32, 64), nn.MaxPool2d(2), # (B, 64, 7, 7)
block(64, 128), # (B, 128, 7, 7)
nn.AdaptiveAvgPool2d(1), nn.Flatten(), # (B, 128) global average pooling
nn.Dropout(0.2), nn.Linear(128, 10)) # (B, 10) logits
x = torch.randn(16, 1, 28, 28)
print(model(x).shape, sum(p.numel() for p in model.parameters()))Quiz
1. A Conv2d(3, 16, kernel_size=5, stride=1, padding=0) on a 32 × 32 image gives…
2. How many parameters does Conv2d(64, 128, 3) have (with bias)?
3. Why stack two 3 × 3 convs instead of one 5 × 5?
4. What does global average pooling do?
5. What is a 1 × 1 convolution mainly used for?
6. Which innovation made 100+ layer CNNs trainable?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- CS231n: Deep Learning for Computer Vision (Stanford) ↗ Notes on backprop, CNNs and training
- Deep Residual Learning (He et al., 2015) ↗ The ResNet paper
- Dive into Deep Learning (free book with code) ↗ Interactive textbook in PyTorch, by Zhang, Lipton, Li & Smola