Deep Learning
Topic 8 · Deep Learning · ⏱ about 6 h

Debugging Training & Reading Loss Curves

A step-by-step recipe for training networks, diagnosing loss curves, vanishing and exploding gradients, residual connections, the classic bugs, and reproducible experiments.

Show:
In this topic
Beginner

Neural networks fail silently

Ordinary code crashes when it is wrong. A network with a bug usually still trains, just worse: a shuffled label, a wrong normalisation or a missing eval() costs a few percent and raises no error. So experienced practitioners follow a careful recipe (popularised by Andrej Karpathy's "A Recipe for Training Neural Networks") and verify each step before adding complexity:

  1. Look at the data: plot examples with their labels, check class balance, duplicates and leakage.
  2. Set up the full pipeline with a simple model and a baseline (majority class, logistic regression). Fix the random seed.
  3. Check the initial loss (≈ ln C for C balanced classes) and that the loss decreases.
  4. Overfit one batch: a model that cannot reach almost zero loss on 1 to 10 examples has a bug.
  5. Then train properly, visualise predictions and errors, and only then regularise and tune.
Beginner

Watch it: Residual connections keep the gradient alive

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

Reading loss curves

Always plot training and validation loss per epoch (or per few hundred steps) on the same chart. The common patterns:

What you seeLikely causeWhat to try
Training loss flat from the startLearning rate far too low, a bug (frozen weights, no optimizer.step()), or dead unitsOverfit one batch; check gradients are non-zero
Loss jumps around, spikes or becomes NaNLearning rate too high, exploding gradients, bad data (NaN, huge values), log(0)Lower LR, warm-up, clipping, check inputs
Both decrease slowly, still falling at the endUnderfitting / not trained long enoughTrain longer, higher LR, bigger model
Training keeps falling, validation turns upOverfittingEarly stopping, more data/augmentation, regularisation
Validation loss far below training lossDropout/augmentation only in training (often fine), or leakage / duplicates across splitsCheck splits for leakage
Training loss drops in steps at each epoch startData not shuffled (sorted by class or source)Shuffle every epoch
Try it

Try it: diagnose the curve

Six real-looking training runs. For each, choose the most likely diagnosis.

Interactive: this part runs in your browser.

Beginner

Vanishing and exploding gradients

The backward pass multiplies a gradient by each layer's local gradient, layer after layer. If those factors are mostly smaller than 1, the gradient vanishes on the way down and early layers stop learning; if larger than 1, it explodes. Saturating activations (sigmoid, tanh), bad initialisation and long chains (deep networks, RNNs over many time steps) make it worse.

To diagnose it, log the gradient norm of each layer (and the ratio of update size to weight size, which should be around 1e-3). The cures are the topics of this track: good initialisation, ReLU-family activations, normalisation layers, gradient clipping, gated cells (LSTM) and, most importantly, residual connections.

Beginner

Residual connections

A residual block adds its input back to its output: y=x+F(x)y=x+F(x). The layers only need to learn the change F to make, and an identity mapping is easy (F = 0). In the backward pass the derivative of y with respect to x is 1+F′(x)1+F'(x): the "+1" path carries the gradient straight through every block, so it cannot vanish however deep the network is.

Residual connections made 100+ layer networks trainable (ResNet, 2015) and are in every modern architecture, including every transformer block. When you design or debug a deep network, check that every block has its skip connection and that the shapes on both sides of the "+" match (a 1×1 convolution or linear projection fixes mismatches).

Beginner

The classic bugs

Most broken training runs come from a short list. Check these before tuning anything:

  • Forgetting optimizer.zero_grad() (gradients accumulate) or optimizer.step().
  • Applying softmax before CrossEntropyLoss (softmax twice), or sigmoid before BCEWithLogitsLoss.
  • Labels misaligned with inputs after shuffling one but not the other.
  • Broadcasting a (B, 1) prediction against a (B,) target (see Tensors).
  • No model.eval() / torch.no_grad() during validation.
  • Augmentation or normalisation statistics computed on validation/test data; the same patient or document in training and test (leakage).
  • Input not normalised the same way at inference as in training (e.g. pixels 0 to 255 instead of 0 to 1).
  • Loss averaged over the wrong axis, or a mask applied after averaging, so padding counts.
Try it

Code example: fix a broken training script

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Beginner

Reproducible experiments

You will run dozens of experiments. Make each one reproducible and comparable: fix seeds (random, NumPy, PyTorch), record the full config (model, data version, hyperparameters, code commit), log metrics and curves to an experiment tracker (Weights & Biases, MLflow, TensorBoard), and save checkpoints with that config. Change one thing at a time, and run important comparisons with 2 to 3 seeds: differences smaller than the seed-to-seed spread are noise. GPUs are not perfectly deterministic by default; torch.use_deterministic_algorithms(True) helps at some speed cost.

Code

Code: overfit one batch and watch gradient norms

Two debugging habits in PyTorch.

Python · PyTorch
xb, yb = next(iter(train_loader))                    # ONE batch
for step in range(300):
    loss = loss_fn(model(xb), yb)
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 50 == 0: print(step, loss.item())         # must approach 0, or there is a bug

def grad_report(model):
    for name, p in model.named_parameters():
        if p.grad is not None:
            ratio = (lr * p.grad).norm() / (p.norm() + 1e-12)   # update/weight, ~1e-3 is healthy
            print(f"{name:30s} grad {p.grad.norm():.2e}  update/weight {ratio:.1e}")
This code needs PyTorch and your own model and data loader, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. Why "overfit one batch" before full training?

2. Training loss keeps falling but validation loss starts rising after epoch 8. You should…

3. The training loss drops sharply at the start of every epoch, then flattens. What is wrong?

4. Why do residual connections stop gradients from vanishing?

5. Validation loss is far below training loss and validation accuracy is near 100%. First suspicion?

6. Two settings differ by 0.3% accuracy; the seed-to-seed spread is 0.8%. You conclude…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready