Machine Learning
Topic 28 · Deep learning · ⏱ about 6 h

Training Neural Networks

Backpropagation, activations, optimisers and the tricks that make deep nets work.

Show:
In this topic
Beginner

Forward pass, loss, backpropagation

Training a neural network means adjusting its weights to reduce the loss, using gradient descent. The hard part is computing the gradient: a network may have millions of weights, and we need to know how the loss would change if we nudged each one. The algorithm that does this efficiently is called backpropagation. Each training step consists of four stages.

input xlayer 1layer 2loss L① forward pass: compute and store activations →← ② backward pass: δ = ∂L/∂z flows back, multiplied by local slopes③ each weight gradient = (its input) × (error signal at its output)
Training step: a forward pass computes the loss; backpropagation sends error signals back through the layers to get every gradient.

First comes the forward pass. A mini-batch of examples is fed into the network, and each layer computes its outputs from the previous layer's outputs, until the final layer produces predictions. Along the way, the intermediate values are stored, because they will be needed later.

Second, the loss is computed by comparing the predictions with the true targets, using mean squared error for regression or cross-entropy for classification. This single number summarises how badly the network did on this batch.

Third comes the backward pass, which is backpropagation itself. Using the chain rule, it works out how much each weight contributed to the loss. It starts at the output, where the effect of the predictions on the loss is easy to compute, and works backwards one layer at a time. At each layer, it uses the gradient already computed for the layer above, together with the values stored during the forward pass, to compute the gradient for the current layer. Because each layer reuses the work done for the layer above, the gradients for all the weights are obtained at roughly the cost of one extra forward pass.

Fourth, every weight is updated with a gradient-descent step, usually using an optimiser such as Adam or SGD with momentum. Then the next mini-batch is processed, and the cycle repeats thousands or millions of times.

You will rarely have to implement backpropagation yourself. Deep-learning frameworks such as PyTorch and TensorFlow provide automatic differentiation ("autograd"): you write only the forward pass, and the framework records the operations you used and computes all the gradients for you when you call loss.backward(). Understanding what happens inside is still valuable, though, especially for diagnosing why training fails, which is the subject of the next sections.

Interactive: this part runs in your browser.

Breakdown. One neuron with its forward values and backward gradients; the chain rule multiplies local slopes.
Try it

Try it: train a neural network in your browser

Press Train and watch the decision boundary form as backpropagation and Adam update the weights. Try the spiral with 1 unit versus 16, sigmoid versus ReLU, and a learning rate that is far too high.

Interactive: this part runs in your browser.

Try it

Try it: activation functions

Compare each function (solid) and its derivative (dashed). During backprop, gradients get multiplied by these derivatives layer after layer.

Interactive: this part runs in your browser.

Beginner

Making deep networks train well

Deep networks can be surprisingly hard to train, and over the years researchers have developed a toolkit of techniques that make training reliable. Each one addresses a specific problem.

sigmoid, deep: gradient shrinks ×≤0.25 per layer87654321ReLU + residuals: gradient survives87654321layer (8 = output … 1 = first)
Vanishing gradients: multiplying by small slopes at every layer shrinks the signal exponentially. ReLU, good initialisation and residual connections keep it alive.

The first problem is vanishing and exploding gradients. During backpropagation, the gradient is multiplied by a factor at every layer. If those factors are typically less than 1, the gradient shrinks exponentially as it travels back through many layers, and the early layers barely learn at all (vanishing gradients). If the factors are typically greater than 1, the gradient grows exponentially and training becomes unstable (exploding gradients). The fixes are: use ReLU-family activations, whose slope of 1 for positive inputs doesn't shrink the signal; initialise the weights carefully, at a scale chosen so that signals keep roughly the same size from layer to layer (He initialisation for ReLU networks, Xavier or Glorot initialisation for tanh networks); add normalisation layers (batch normalisation or layer normalisation), which keep each layer's activations at a healthy scale; add residual (skip) connections, which give the gradient a direct path around each block; and use gradient clipping, which caps the size of the gradient to prevent occasional explosions.

The second problem is overfitting, because large networks can easily memorise their training data. The standard remedies are dropout, weight decay (preferably via the AdamW optimiser), data augmentation and early stopping, all described in the regularisation topic.

The third problem is slow or unstable optimisation. Adaptive optimisers such as Adam handle poorly scaled problems well. A learning-rate schedule, typically a short warm-up from a small learning rate followed by a gradual decay, often makes a large difference. And the batch size should be chosen sensibly, adjusting the learning rate along with it.

Finally, a practical debugging recipe that experienced practitioners use. Before training on the full dataset, take a tiny batch of, say, ten examples and check that the network can overfit it completely, driving the training loss close to zero. A correctly implemented network with enough capacity should always manage this. If it can't, there is a bug, perhaps in the data pipeline, the loss function or the way the labels are fed in, and no amount of tuning will help until it is fixed. Once the tiny-batch test passes, scale up to the full data and add regularisation. Throughout, plot both the training loss and the validation loss, since the relationship between them tells you whether you are underfitting or overfitting.

Interactive: this part runs in your browser.

Interactive. Dropout switches off a random set of neurons for every mini-batch.
Intermediate · Maths

Backpropagation derived for a 2-layer network

📐 Maths you'll need: m-calculus · m-gradients · m-matrices
💡 The idea in plain words

Backpropagation answers "how much did each weight contribute to the error?" It starts at the output, where the error signal is simply (prediction − target), and passes that signal backwards through each layer using the chain rule: multiply by the layer's weights (transposed) and by the activation's slope. Each weight's gradient is then (its input) × (the error signal arriving at its output).

🔤 Symbols in this section
zℓz_\ellpre-activation of layer ℓ (before φ)
hhactivation (after φ)
δℓ=∂L/∂zℓ\delta_\ell = \partial L/\partial z_\ellerror signal at layer ℓ
⊙\odotelement-wise multiplication
ϕ′\phi'derivative of the activation

This section derives backpropagation for a network with one hidden layer, with every step shown. The same pattern extends to any number of layers.

The network takes an input row vector xx and computes, in order,

z1=xW1+b1,h=ϕ(z1),z2=hW2+b2,p=softmax⁡(z2). z_1 = xW_1 + b_1, \qquad h = \phi(z_1), \qquad z_2 = hW_2 + b_2, \qquad p = \operatorname{softmax}(z_2).

Here z1z_1 and z2z_2 are the pre-activations of the hidden and output layers, hh is the hidden layer's output, and pp holds the predicted class probabilities. With a one-hot target yy (a 1 for the correct class and 0 elsewhere), the loss is the cross-entropy L=−∑kyklog⁡pkL = -\sum_k y_k\log p_k. We want the gradient of LL with respect to all four sets of parameters. The key bookkeeping device is the error signal at each layer, defined as the gradient of the loss with respect to that layer's pre-activation, δ=∂L/∂z\delta = \partial L/\partial z.

Step 1: the output error. For a softmax output paired with cross-entropy, the error signal at the output is simply prediction minus target:
δ2=∂L∂z2=p−y. \delta_2 = \frac{\partial L}{\partial z_2} = p - y.
(The proof is shown in the expandable box below.)
Step 2: gradients of the output layer. Since z2=hW2+b2z_2 = hW_2 + b_2, each weight in W2W_2 connects one hidden unit to one output unit, and its gradient is that hidden unit's value times that output's error. In matrix form,
∂L∂W2=h⊤δ2,∂L∂b2=δ2. \frac{\partial L}{\partial W_2} = h^\top\delta_2, \qquad \frac{\partial L}{\partial b_2} = \delta_2.
Step 3: send the error back to the hidden layer. Each hidden unit influenced the loss through all of its outgoing weights, so its share of the error is the output errors weighted by those weights, δ2W2⊤\delta_2W_2^\top. The error must then pass back through the activation function, which multiplies it by the activation's slope at that point (by the chain rule):
δ1=∂L∂z1=(δ2W2⊤)⊙ϕ′(z1). \delta_1 = \frac{\partial L}{\partial z_1} = \big(\delta_2 W_2^\top\big)\odot\phi'(z_1).
The symbol ⊙\odot means element-by-element multiplication.
Step 4: gradients of the hidden layer. Exactly as in Step 2, with the input xx in place of hh:
∂L∂W1=x⊤δ1,∂L∂b1=δ1. \frac{\partial L}{\partial W_1} = x^\top\delta_1, \qquad \frac{\partial L}{\partial b_1} = \delta_1.

The general pattern for any depth is now clear: each layer's error signal is the next layer's error signal sent back through the transposed weights and multiplied by the local activation slope, δℓ=(δℓ+1Wℓ+1⊤)⊙ϕ′(zℓ)\delta_\ell = (\delta_{\ell+1}W_{\ell+1}^\top)\odot\phi'(z_\ell), and each weight gradient is the layer's input (transposed) times its error signal, ∇WℓL=Hℓ−1⊤δℓ\nabla_{W_\ell}L = H_{\ell-1}^\top\delta_\ell. For a mini-batch, exactly the same formulas hold with matrices in place of vectors, and the gradients are summed or averaged over the rows.

Why softmax with cross-entropy gives p − y

The derivative of the softmax is ∂pk/∂zj=pk(1[j=k]−pj)\partial p_k/\partial z_j = p_k(\mathbb 1[j=k] - p_j), where 1[j=k]\mathbb 1[j=k] is 1 if j=kj = k and 0 otherwise. Using the chain rule, ∂L∂zj=−∑kykpk pk(1[j=k]−pj)=−∑kyk(1[j=k]−pj)=−yj+pj∑kyk=pj−yj\frac{\partial L}{\partial z_j} = -\sum_k \frac{y_k}{p_k}\,p_k(\mathbb 1[j=k]-p_j) = -\sum_k y_k(\mathbb 1[j=k]-p_j) = -y_j + p_j\sum_k y_k = p_j - y_j, because the one-hot labels add up to 1. ∎

Why this is efficient. Frameworks implement this idea in general form as reverse-mode automatic differentiation. During the forward pass they record every operation in a computation graph, and during the backward pass they apply the chain rule through that graph from the output back to the inputs. The total cost is only about two to three times the cost of the forward pass, regardless of how many parameters there are. Computing each of a billion partial derivatives separately would be hopeless; backpropagation gets them all at once, and that is what makes training very large models possible.

Key takeaway. δ_out = p − y; δ_ℓ = (δ_ℓ₊₁Wᵀ) ⊙ φ′(z_ℓ); ∇W = (layer input)ᵀ δ. Autograd does exactly this for you.
Advanced

Vanishing gradients, initialisation, normalisation and dropout

📐 Maths you'll need: m-prob · m-calculus
💡 The idea in plain words

In a deep network the backward signal is multiplied by something at every layer. Multiply by 0.25 twenty times and it vanishes; by 2 twenty times and it explodes. Every trick here keeps that signal a healthy size: careful initial weights, normalising activations, shortcut (residual) connections, and capping the gradient.

🔤 Symbols in this section
nin, noutn_{in},\ n_{out}inputs and outputs of a layer
Var⁡(w)\operatorname{Var}(w)variance of initial weights
μB, σB2\mu_B,\ \sigma_B^2batch mean and variance
γ, β\gamma,\ \betalearned scale and shift
mi∼Bernoulli(1−p)m_i\sim\text{Bernoulli}(1-p)dropout keep/drop mask
ccgradient-clipping threshold

This section explains the mathematics behind the training techniques listed earlier: why gradients vanish, how initialisation, normalisation and residual connections counteract that, and how dropout and clipping work.

Why gradients vanish or explode. By the chain rule, the gradient reaching the first layer of an LL-layer network is a product of one factor per layer:

∂L∂h1=∂L∂hL∏ℓ=2Ldiag⁡(ϕ′(zℓ)) Wℓ⊤. \frac{\partial L}{\partial h_1} = \frac{\partial L}{\partial h_L}\prod_{\ell=2}^{L}\operatorname{diag}\big(\phi'(z_\ell)\big)\,W_\ell^\top .

Each factor combines the slope of the activation function with a weight matrix. If the typical "gain" of each factor is less than 1, the product shrinks exponentially with depth; if it is greater than 1, the product grows exponentially. The sigmoid's slope is at most 14\tfrac14, so in a 10-layer sigmoid network the activation slopes alone can shrink the gradient by a factor of roughly 4−104^{-10}, about one millionth. That is why deep sigmoid networks were notoriously hard to train, and why ReLU, whose slope is exactly 1 for positive inputs, was such a breakthrough.

Initialisation keeps the signal at a constant size. Consider one neuron's pre-activation, z=∑i=1ninwixiz = \sum_{i=1}^{n_{in}} w_i x_i, with independent, zero-mean weights and inputs. Using the variance rules from the probability topic, Var⁡(z)=ninVar⁡(w)Var⁡(x)\operatorname{Var}(z) = n_{in}\operatorname{Var}(w)\operatorname{Var}(x). For the signal to stay the same size from layer to layer, we want Var⁡(z)=Var⁡(x)\operatorname{Var}(z) = \operatorname{Var}(x), which requires Var⁡(w)=1/nin\operatorname{Var}(w) = 1/n_{in}. Taking the backward pass into account too, and the effect of each activation function, gives the two standard schemes:

Xavier/Glorot (tanh): Var⁡(w)=2nin+nout,He/Kaiming (ReLU): Var⁡(w)=2nin. \text{Xavier/Glorot (tanh): } \operatorname{Var}(w) = \frac{2}{n_{in} + n_{out}}, \qquad \text{He/Kaiming (ReLU): } \operatorname{Var}(w) = \frac{2}{n_{in}}.

The extra factor of 2 for ReLU compensates for the fact that ReLU sets roughly half of its inputs to zero, halving the variance it passes on.

Batch normalisation enforces a healthy scale directly during training. For each feature, it subtracts the mean and divides by the standard deviation computed over the current mini-batch, and then applies a learned scale γ\gamma and shift β\beta so that the network can still choose whatever scale suits it best:

x^=x−μBσB2+ϵ,y=γx^+β. \hat x = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \qquad y = \gamma\hat x + \beta.

At test time, when there may be only one example, running averages of μ\mu and σ2\sigma^2 collected during training are used instead. Layer normalisation applies the same idea across the features of each individual example rather than across the batch, so it doesn't depend on batch size at all, which is why transformers use it.

Residual connections change each block from hℓ+1=F(hℓ)h_{\ell+1} = F(h_\ell) to hℓ+1=hℓ+F(hℓ)h_{\ell+1} = h_\ell + F(h_\ell): the block learns a correction to its input rather than a complete replacement. Differentiating gives ∂hℓ+1∂hℓ=I+∂F∂hℓ\frac{\partial h_{\ell+1}}{\partial h_\ell} = I + \frac{\partial F}{\partial h_\ell}. The identity matrix II means that, even if FF's own gradient is small, the gradient can always flow straight through the "+" unchanged. This is what made it possible to train networks with hundreds of layers, starting with ResNet.

Inverted dropout. During training, each unit is kept with probability 1−p1-p and set to zero with probability pp, and the survivors are divided by 1−p1-p: h~=m⊙h1−p\tilde h = \frac{m\odot h}{1-p}, where each mim_i is 1 with probability 1−p1-p. The division makes the expected value of h~\tilde h equal to hh, so at test time, when dropout is switched off, nothing needs rescaling. Because a different random subset of units is dropped at each step, dropout can be viewed as training a huge collection of overlapping sub-networks (2N2^N of them for NN units) and averaging them.

Gradient clipping guards against occasional explosions: if the gradient's length ∥g∥\lVert g\rVert exceeds a threshold cc, it is rescaled to have length exactly cc, keeping its direction: g←c g/∥g∥g \leftarrow c\,g/\lVert g\rVert. It is standard practice when training recurrent networks and transformers.

Key takeaway. Keep signal variance constant (He/Xavier init), normalise (batch/layer norm), add residual paths, use inverted dropout and clip gradients.
Code

Code: backprop from scratch, then a full PyTorch training loop

The NumPy network uses exactly the equations above, He initialisation, and a numerical gradient check.

Python · NumPy from scratch (2-layer net on two moons)
import numpy as np
from sklearn.datasets import make_moons

X, y = make_moons(n_samples=1000, noise=0.2, random_state=0)
Y = np.eye(2)[y]                                     # one-hot
rng = np.random.default_rng(0)
d_in, d_h, d_out = 2, 32, 2
W1 = rng.normal(0, np.sqrt(2 / d_in), (d_in, d_h)); b1 = np.zeros(d_h)   # He init
W2 = rng.normal(0, np.sqrt(2 / d_h), (d_h, d_out)); b2 = np.zeros(d_out)

def forward(X):
    z1 = X @ W1 + b1
    h = np.maximum(0, z1)
    z2 = h @ W2 + b2
    z2 = z2 - z2.max(1, keepdims=True)               # stable softmax
    p = np.exp(z2) / np.exp(z2).sum(1, keepdims=True)
    return z1, h, p

def loss_fn(p, Y):
    return -np.mean(np.sum(Y * np.log(p + 1e-12), axis=1))

lr = 0.5
for epoch in range(2001):
    z1, h, p = forward(X)
    n = len(X)
    d2 = (p - Y) / n                                 # delta at output
    gW2, gb2 = h.T @ d2, d2.sum(0)
    d1 = (d2 @ W2.T) * (z1 > 0)                      # back through ReLU
    gW1, gb1 = X.T @ d1, d1.sum(0)
    W1 -= lr * gW1; b1 -= lr * gb1; W2 -= lr * gW2; b2 -= lr * gb2
    if epoch % 500 == 0:
        print(f"epoch {epoch}: loss {loss_fn(p, Y):.4f}  acc {np.mean(p.argmax(1) == y):.3f}")

# Gradient check on one weight: analytic vs finite difference
i, j, eps = 0, 3, 1e-5
_, h, p = forward(X); d2 = (p - Y) / len(X)
analytic = (X.T @ ((d2 @ W2.T) * (X @ W1 + b1 > 0)))[i, j]
W1[i, j] += eps; lp = loss_fn(forward(X)[2], Y)
W1[i, j] -= 2 * eps; lm = loss_fn(forward(X)[2], Y)
W1[i, j] += eps
print("grad check:", analytic, (lp - lm) / (2 * eps))
Open in playground ↗
Python · PyTorch training loop (with validation, scheduler, early stopping)
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader, random_split

torch.manual_seed(0)
Xt, yt = torch.tensor(X, dtype=torch.float32), torch.tensor(y)
train_ds, val_ds = random_split(TensorDataset(Xt, yt), [800, 200])
train_dl = DataLoader(train_ds, batch_size=64, shuffle=True)
val_dl = DataLoader(val_ds, batch_size=256)

model = nn.Sequential(nn.Linear(2, 64), nn.ReLU(), nn.Dropout(0.1),
                      nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 2))
opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=1e-2)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=200)
loss_fn = nn.CrossEntropyLoss()

best, patience, bad = float("inf"), 20, 0
for epoch in range(200):
    model.train()
    for xb, yb in train_dl:
        opt.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()                                   # backprop via autograd
        nn.utils.clip_grad_norm_(model.parameters(), 1.0) # gradient clipping
        opt.step()
    sched.step()
    model.eval()                                          # disables dropout
    with torch.no_grad():
        val_loss = sum(loss_fn(model(xb), yb).item() * len(yb) for xb, yb in val_dl) / len(val_ds)
    if val_loss < best - 1e-4:
        best, bad = val_loss, 0
        torch.save(model.state_dict(), "best.pt")
    else:
        bad += 1
        if bad >= patience:
            print("early stop at epoch", epoch); break
model.load_state_dict(torch.load("best.pt"))
print("best val loss:", round(best, 4))
This code needs PyTorch, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. Backpropagation computes gradients using…

2. Why did ReLU help train deep networks?

3. Your network can't overfit a tiny batch of 10 examples. This suggests…

4. In backprop, δℓ = (δℓ₊₁ Wℓ₊₁ᵀ) ⊙ …

5. He initialisation for ReLU layers uses Var(w) =

6. Why do residual connections help very deep networks?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready