Backpropagation & Autograd from Scratch
Computational graphs, the chain rule as message passing, gradients of the operations networks use, gradient checking, and how PyTorch's autograd works.
Every computation is a graph
Any formula a network computes can be drawn as a computational graph: inputs and parameters at the bottom, each operation (add, multiply, ReLU, matrix product) a node, the loss at the top. The forward pass computes values from the inputs up to the loss. The backward pass computes, for every node, how much the loss would change if that node's value changed slightly: its gradient .
The key fact is the chain rule. Each operation only needs to know its own local gradient (how its output changes with its inputs). The backward pass starts with at the top and passes gradients downwards: each node multiplies the gradient arriving from above by its local gradient and sends the result to its inputs.
Doing this from the top down is called reverse-mode differentiation. It is efficient for networks because there is one output (the loss) and millions of parameters: one backward pass, which costs about the same as one or two forward passes, gives the gradient for all of them.
Watch it: Autograd: values flow forward, gradients flow back
A short narrated animation. Press โถ Play, or step through it with โฎ โญ. Tick ๐ narrate to hear each step.
Interactive: this part runs in your browser.
A worked example
Take with .
| Node | Forward value | Local gradient | Gradient of L (backward) |
|---|---|---|---|
| d = a ยท b | โ6 | โd/โa = b = โ3, โd/โb = a = 2 | โL/โa = 1 ยท (โ3) = โ3, โL/โb = 1 ยท 2 = 2 |
| e = d + c | 4 | โe/โd = 1, โe/โc = 1 | โL/โd = 1, โL/โc = 1 |
| L = relu(e) | 4 | 1 if e > 0 else 0 | โL/โe = 1 (e is positive) |
Read it bottom-up for the forward pass and top-down for the backward pass. Three patterns appear everywhere: an add node passes its gradient unchanged to both inputs; a multiply node sends each input the gradient times the other input; a ReLU passes the gradient through where the input was positive and blocks it where it was negative. If c were โ10, e would be โ4, the ReLU would block everything, and every gradient would be 0: a "dead" unit for this example.
Try it: forward and backward on a graph
Change the inputs. Values (blue) flow up; gradients (orange) flow down. The last row checks each gradient numerically, the way you test an autograd implementation.
Interactive: this part runs in your browser.
Gradients of the operations networks use
A real network uses a handful of operations on whole tensors. Their backward rules, with arriving from above:
| Forward | Backward |
|---|---|
| (dense layer; X: (B, in), W: (in, out)) | , |
| (bias broadcast over the batch) | : sum over the axes that were broadcast |
| softmax + cross-entropy, probabilities P, one-hot targets T, mean over B |
A useful check: a gradient always has the same shape as the thing it is the gradient of. is (in, B)(B, out) = (in, out), the shape of W. If a value is used in two places (for example a residual connection), its gradients from both uses are added; this is why PyTorch accumulates into .grad.
Gradient checking
When you write a backward pass yourself, check it numerically. Nudge one parameter by a small in each direction and compare the change in the loss with your analytic gradient:
Use float64 for the check. A relative error below about means the gradient is right; around or worse means a bug. Check a few random entries of each parameter rather than all of them, and turn off randomness (dropout) while checking. Frameworks provide this as a test utility (torch.autograd.gradcheck).
Code example: build a tiny autograd engine
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
How PyTorch's autograd works
PyTorch builds the graph dynamically as your Python code runs (define-by-run): every operation on a tensor with requires_grad=True records a node with its backward function (y.grad_fn). loss.backward() walks the graph in reverse topological order, exactly like the exercise, and adds the result into .grad of every leaf tensor (parameters). Then the graph is freed (retain_graph=True keeps it).
with torch.no_grad():records nothing: use it for evaluation and inference (less memory, faster).x.detach()returns a tensor cut off from the graph: gradients will not flow back through it (used for targets, logging, stopping gradients deliberately).- Because gradients accumulate, call
optimizer.zero_grad()each step, or accumulate over several batches deliberately (gradient accumulation). - Saved activations for the backward pass are the main memory cost of training; gradient checkpointing recomputes them instead of storing them.
Code: the same example in PyTorch
Autograd reproduces the worked example, and gradcheck tests a custom function.
import torch
a = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(-3.0, requires_grad=True)
c = torch.tensor(10.0, requires_grad=True)
L = torch.relu(a * b + c)
L.backward()
print(L.item(), a.grad.item(), b.grad.item(), c.grad.item()) # 4.0 -3.0 2.0 1.0
# numerical check of a function (needs float64)
f = lambda x: (x.sin() * x).sum()
x = torch.randn(5, dtype=torch.float64, requires_grad=True)
print(torch.autograd.gradcheck(f, (x,))) # TrueQuiz
1. Why is reverse-mode differentiation used to train networks?
2. In the backward pass, a multiply node z = x ยท y sends x the gradientโฆ
3. For a dense layer Y = XW with X: (32, 10) and W: (10, 4), what is the shape of โL/โW?
4. A bias b of shape (4,) is broadcast over a batch of 32. Its gradient isโฆ
5. Your analytic and numerical gradients differ with relative error 0.03. This most likely meansโฆ
6. Why must you call optimizer.zero_grad() each step in PyTorch?
๐ Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Neural Networks: Zero to Hero (Karpathy) โ Video course building backprop, MLPs and GPT from scratch
- CS231n: Deep Learning for Computer Vision (Stanford) โ Notes on backprop, CNNs and training
- Understanding Deep Learning (Prince) โ Modern, clear and free textbook