Debugging Training & Reading Loss Curves
A step-by-step recipe for training networks, diagnosing loss curves, vanishing and exploding gradients, residual connections, the classic bugs, and reproducible experiments.
Neural networks fail silently
Ordinary code crashes when it is wrong. A network with a bug usually still trains, just worse: a shuffled label, a wrong normalisation or a missing eval() costs a few percent and raises no error. So experienced practitioners follow a careful recipe (popularised by Andrej Karpathy's "A Recipe for Training Neural Networks") and verify each step before adding complexity:
- Look at the data: plot examples with their labels, check class balance, duplicates and leakage.
- Set up the full pipeline with a simple model and a baseline (majority class, logistic regression). Fix the random seed.
- Check the initial loss (≈ ln C for C balanced classes) and that the loss decreases.
- Overfit one batch: a model that cannot reach almost zero loss on 1 to 10 examples has a bug.
- Then train properly, visualise predictions and errors, and only then regularise and tune.
Watch it: Residual connections keep the gradient alive
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Reading loss curves
Always plot training and validation loss per epoch (or per few hundred steps) on the same chart. The common patterns:
| What you see | Likely cause | What to try |
|---|---|---|
| Training loss flat from the start | Learning rate far too low, a bug (frozen weights, no optimizer.step()), or dead units | Overfit one batch; check gradients are non-zero |
| Loss jumps around, spikes or becomes NaN | Learning rate too high, exploding gradients, bad data (NaN, huge values), log(0) | Lower LR, warm-up, clipping, check inputs |
| Both decrease slowly, still falling at the end | Underfitting / not trained long enough | Train longer, higher LR, bigger model |
| Training keeps falling, validation turns up | Overfitting | Early stopping, more data/augmentation, regularisation |
| Validation loss far below training loss | Dropout/augmentation only in training (often fine), or leakage / duplicates across splits | Check splits for leakage |
| Training loss drops in steps at each epoch start | Data not shuffled (sorted by class or source) | Shuffle every epoch |
Try it: diagnose the curve
Six real-looking training runs. For each, choose the most likely diagnosis.
Interactive: this part runs in your browser.
Vanishing and exploding gradients
The backward pass multiplies a gradient by each layer's local gradient, layer after layer. If those factors are mostly smaller than 1, the gradient vanishes on the way down and early layers stop learning; if larger than 1, it explodes. Saturating activations (sigmoid, tanh), bad initialisation and long chains (deep networks, RNNs over many time steps) make it worse.
To diagnose it, log the gradient norm of each layer (and the ratio of update size to weight size, which should be around 1e-3). The cures are the topics of this track: good initialisation, ReLU-family activations, normalisation layers, gradient clipping, gated cells (LSTM) and, most importantly, residual connections.
Residual connections
A residual block adds its input back to its output: . The layers only need to learn the change F to make, and an identity mapping is easy (F = 0). In the backward pass the derivative of y with respect to x is : the "+1" path carries the gradient straight through every block, so it cannot vanish however deep the network is.
Residual connections made 100+ layer networks trainable (ResNet, 2015) and are in every modern architecture, including every transformer block. When you design or debug a deep network, check that every block has its skip connection and that the shapes on both sides of the "+" match (a 1×1 convolution or linear projection fixes mismatches).
The classic bugs
Most broken training runs come from a short list. Check these before tuning anything:
- Forgetting
optimizer.zero_grad()(gradients accumulate) oroptimizer.step(). - Applying softmax before
CrossEntropyLoss(softmax twice), or sigmoid beforeBCEWithLogitsLoss. - Labels misaligned with inputs after shuffling one but not the other.
- Broadcasting a (B, 1) prediction against a (B,) target (see Tensors).
- No
model.eval()/torch.no_grad()during validation. - Augmentation or normalisation statistics computed on validation/test data; the same patient or document in training and test (leakage).
- Input not normalised the same way at inference as in training (e.g. pixels 0 to 255 instead of 0 to 1).
- Loss averaged over the wrong axis, or a mask applied after averaging, so padding counts.
Code example: fix a broken training script
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Reproducible experiments
You will run dozens of experiments. Make each one reproducible and comparable: fix seeds (random, NumPy, PyTorch), record the full config (model, data version, hyperparameters, code commit), log metrics and curves to an experiment tracker (Weights & Biases, MLflow, TensorBoard), and save checkpoints with that config. Change one thing at a time, and run important comparisons with 2 to 3 seeds: differences smaller than the seed-to-seed spread are noise. GPUs are not perfectly deterministic by default; torch.use_deterministic_algorithms(True) helps at some speed cost.
Code: overfit one batch and watch gradient norms
Two debugging habits in PyTorch.
xb, yb = next(iter(train_loader)) # ONE batch
for step in range(300):
loss = loss_fn(model(xb), yb)
opt.zero_grad(); loss.backward(); opt.step()
if step % 50 == 0: print(step, loss.item()) # must approach 0, or there is a bug
def grad_report(model):
for name, p in model.named_parameters():
if p.grad is not None:
ratio = (lr * p.grad).norm() / (p.norm() + 1e-12) # update/weight, ~1e-3 is healthy
print(f"{name:30s} grad {p.grad.norm():.2e} update/weight {ratio:.1e}")Quiz
1. Why "overfit one batch" before full training?
2. Training loss keeps falling but validation loss starts rising after epoch 8. You should…
3. The training loss drops sharply at the start of every epoch, then flattens. What is wrong?
4. Why do residual connections stop gradients from vanishing?
5. Validation loss is far below training loss and validation accuracy is near 100%. First suspicion?
6. Two settings differ by 0.3% accuracy; the seed-to-seed spread is 0.8%. You conclude…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- A Recipe for Training Neural Networks (Karpathy) ↗ The classic practical debugging guide
- Deep Learning Tuning Playbook (Google) ↗ How to tune hyperparameters systematically
- CS231n: Deep Learning for Computer Vision (Stanford) ↗ Notes on backprop, CNNs and training