Training Neural Networks
Backpropagation, activations, optimisers and the tricks that make deep nets work.
Forward pass, loss, backpropagation
Training a neural network means adjusting its weights to reduce the loss, using gradient descent. The hard part is computing the gradient: a network may have millions of weights, and we need to know how the loss would change if we nudged each one. The algorithm that does this efficiently is called backpropagation. Each training step consists of four stages.
First comes the forward pass. A mini-batch of examples is fed into the network, and each layer computes its outputs from the previous layer's outputs, until the final layer produces predictions. Along the way, the intermediate values are stored, because they will be needed later.
Second, the loss is computed by comparing the predictions with the true targets, using mean squared error for regression or cross-entropy for classification. This single number summarises how badly the network did on this batch.
Third comes the backward pass, which is backpropagation itself. Using the chain rule, it works out how much each weight contributed to the loss. It starts at the output, where the effect of the predictions on the loss is easy to compute, and works backwards one layer at a time. At each layer, it uses the gradient already computed for the layer above, together with the values stored during the forward pass, to compute the gradient for the current layer. Because each layer reuses the work done for the layer above, the gradients for all the weights are obtained at roughly the cost of one extra forward pass.
Fourth, every weight is updated with a gradient-descent step, usually using an optimiser such as Adam or SGD with momentum. Then the next mini-batch is processed, and the cycle repeats thousands or millions of times.
You will rarely have to implement backpropagation yourself. Deep-learning frameworks such as PyTorch and TensorFlow provide automatic differentiation ("autograd"): you write only the forward pass, and the framework records the operations you used and computes all the gradients for you when you call loss.backward(). Understanding what happens inside is still valuable, though, especially for diagnosing why training fails, which is the subject of the next sections.
Interactive: this part runs in your browser.
Try it: train a neural network in your browser
Press Train and watch the decision boundary form as backpropagation and Adam update the weights. Try the spiral with 1 unit versus 16, sigmoid versus ReLU, and a learning rate that is far too high.
Interactive: this part runs in your browser.
Try it: activation functions
Compare each function (solid) and its derivative (dashed). During backprop, gradients get multiplied by these derivatives layer after layer.
Interactive: this part runs in your browser.
Making deep networks train well
Deep networks can be surprisingly hard to train, and over the years researchers have developed a toolkit of techniques that make training reliable. Each one addresses a specific problem.
The first problem is vanishing and exploding gradients. During backpropagation, the gradient is multiplied by a factor at every layer. If those factors are typically less than 1, the gradient shrinks exponentially as it travels back through many layers, and the early layers barely learn at all (vanishing gradients). If the factors are typically greater than 1, the gradient grows exponentially and training becomes unstable (exploding gradients). The fixes are: use ReLU-family activations, whose slope of 1 for positive inputs doesn't shrink the signal; initialise the weights carefully, at a scale chosen so that signals keep roughly the same size from layer to layer (He initialisation for ReLU networks, Xavier or Glorot initialisation for tanh networks); add normalisation layers (batch normalisation or layer normalisation), which keep each layer's activations at a healthy scale; add residual (skip) connections, which give the gradient a direct path around each block; and use gradient clipping, which caps the size of the gradient to prevent occasional explosions.
The second problem is overfitting, because large networks can easily memorise their training data. The standard remedies are dropout, weight decay (preferably via the AdamW optimiser), data augmentation and early stopping, all described in the regularisation topic.
The third problem is slow or unstable optimisation. Adaptive optimisers such as Adam handle poorly scaled problems well. A learning-rate schedule, typically a short warm-up from a small learning rate followed by a gradual decay, often makes a large difference. And the batch size should be chosen sensibly, adjusting the learning rate along with it.
Finally, a practical debugging recipe that experienced practitioners use. Before training on the full dataset, take a tiny batch of, say, ten examples and check that the network can overfit it completely, driving the training loss close to zero. A correctly implemented network with enough capacity should always manage this. If it can't, there is a bug, perhaps in the data pipeline, the loss function or the way the labels are fed in, and no amount of tuning will help until it is fixed. Once the tiny-batch test passes, scale up to the full data and add regularisation. Throughout, plot both the training loss and the validation loss, since the relationship between them tells you whether you are underfitting or overfitting.
Interactive: this part runs in your browser.
Backpropagation derived for a 2-layer network
Backpropagation answers "how much did each weight contribute to the error?" It starts at the output, where the error signal is simply (prediction − target), and passes that signal backwards through each layer using the chain rule: multiply by the layer's weights (transposed) and by the activation's slope. Each weight's gradient is then (its input) × (the error signal arriving at its output).
🔤 Symbols in this section
| pre-activation of layer ℓ (before φ) | |
| activation (after φ) | |
| error signal at layer ℓ | |
| element-wise multiplication | |
| derivative of the activation |
This section derives backpropagation for a network with one hidden layer, with every step shown. The same pattern extends to any number of layers.
The network takes an input row vector and computes, in order,
Here and are the pre-activations of the hidden and output layers, is the hidden layer's output, and holds the predicted class probabilities. With a one-hot target (a 1 for the correct class and 0 elsewhere), the loss is the cross-entropy . We want the gradient of with respect to all four sets of parameters. The key bookkeeping device is the error signal at each layer, defined as the gradient of the loss with respect to that layer's pre-activation, .
The general pattern for any depth is now clear: each layer's error signal is the next layer's error signal sent back through the transposed weights and multiplied by the local activation slope, , and each weight gradient is the layer's input (transposed) times its error signal, . For a mini-batch, exactly the same formulas hold with matrices in place of vectors, and the gradients are summed or averaged over the rows.
Why softmax with cross-entropy gives p − y
The derivative of the softmax is , where is 1 if and 0 otherwise. Using the chain rule, , because the one-hot labels add up to 1. ∎
Why this is efficient. Frameworks implement this idea in general form as reverse-mode automatic differentiation. During the forward pass they record every operation in a computation graph, and during the backward pass they apply the chain rule through that graph from the output back to the inputs. The total cost is only about two to three times the cost of the forward pass, regardless of how many parameters there are. Computing each of a billion partial derivatives separately would be hopeless; backpropagation gets them all at once, and that is what makes training very large models possible.
Vanishing gradients, initialisation, normalisation and dropout
In a deep network the backward signal is multiplied by something at every layer. Multiply by 0.25 twenty times and it vanishes; by 2 twenty times and it explodes. Every trick here keeps that signal a healthy size: careful initial weights, normalising activations, shortcut (residual) connections, and capping the gradient.
🔤 Symbols in this section
| inputs and outputs of a layer | |
| variance of initial weights | |
| batch mean and variance | |
| learned scale and shift | |
| dropout keep/drop mask | |
| gradient-clipping threshold |
This section explains the mathematics behind the training techniques listed earlier: why gradients vanish, how initialisation, normalisation and residual connections counteract that, and how dropout and clipping work.
Why gradients vanish or explode. By the chain rule, the gradient reaching the first layer of an -layer network is a product of one factor per layer:
Each factor combines the slope of the activation function with a weight matrix. If the typical "gain" of each factor is less than 1, the product shrinks exponentially with depth; if it is greater than 1, the product grows exponentially. The sigmoid's slope is at most , so in a 10-layer sigmoid network the activation slopes alone can shrink the gradient by a factor of roughly , about one millionth. That is why deep sigmoid networks were notoriously hard to train, and why ReLU, whose slope is exactly 1 for positive inputs, was such a breakthrough.
Initialisation keeps the signal at a constant size. Consider one neuron's pre-activation, , with independent, zero-mean weights and inputs. Using the variance rules from the probability topic, . For the signal to stay the same size from layer to layer, we want , which requires . Taking the backward pass into account too, and the effect of each activation function, gives the two standard schemes:
The extra factor of 2 for ReLU compensates for the fact that ReLU sets roughly half of its inputs to zero, halving the variance it passes on.
Batch normalisation enforces a healthy scale directly during training. For each feature, it subtracts the mean and divides by the standard deviation computed over the current mini-batch, and then applies a learned scale and shift so that the network can still choose whatever scale suits it best:
At test time, when there may be only one example, running averages of and collected during training are used instead. Layer normalisation applies the same idea across the features of each individual example rather than across the batch, so it doesn't depend on batch size at all, which is why transformers use it.
Residual connections change each block from to : the block learns a correction to its input rather than a complete replacement. Differentiating gives . The identity matrix means that, even if 's own gradient is small, the gradient can always flow straight through the "+" unchanged. This is what made it possible to train networks with hundreds of layers, starting with ResNet.
Inverted dropout. During training, each unit is kept with probability and set to zero with probability , and the survivors are divided by : , where each is 1 with probability . The division makes the expected value of equal to , so at test time, when dropout is switched off, nothing needs rescaling. Because a different random subset of units is dropped at each step, dropout can be viewed as training a huge collection of overlapping sub-networks ( of them for units) and averaging them.
Gradient clipping guards against occasional explosions: if the gradient's length exceeds a threshold , it is rescaled to have length exactly , keeping its direction: . It is standard practice when training recurrent networks and transformers.
Code: backprop from scratch, then a full PyTorch training loop
The NumPy network uses exactly the equations above, He initialisation, and a numerical gradient check.
import numpy as np
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=1000, noise=0.2, random_state=0)
Y = np.eye(2)[y] # one-hot
rng = np.random.default_rng(0)
d_in, d_h, d_out = 2, 32, 2
W1 = rng.normal(0, np.sqrt(2 / d_in), (d_in, d_h)); b1 = np.zeros(d_h) # He init
W2 = rng.normal(0, np.sqrt(2 / d_h), (d_h, d_out)); b2 = np.zeros(d_out)
def forward(X):
z1 = X @ W1 + b1
h = np.maximum(0, z1)
z2 = h @ W2 + b2
z2 = z2 - z2.max(1, keepdims=True) # stable softmax
p = np.exp(z2) / np.exp(z2).sum(1, keepdims=True)
return z1, h, p
def loss_fn(p, Y):
return -np.mean(np.sum(Y * np.log(p + 1e-12), axis=1))
lr = 0.5
for epoch in range(2001):
z1, h, p = forward(X)
n = len(X)
d2 = (p - Y) / n # delta at output
gW2, gb2 = h.T @ d2, d2.sum(0)
d1 = (d2 @ W2.T) * (z1 > 0) # back through ReLU
gW1, gb1 = X.T @ d1, d1.sum(0)
W1 -= lr * gW1; b1 -= lr * gb1; W2 -= lr * gW2; b2 -= lr * gb2
if epoch % 500 == 0:
print(f"epoch {epoch}: loss {loss_fn(p, Y):.4f} acc {np.mean(p.argmax(1) == y):.3f}")
# Gradient check on one weight: analytic vs finite difference
i, j, eps = 0, 3, 1e-5
_, h, p = forward(X); d2 = (p - Y) / len(X)
analytic = (X.T @ ((d2 @ W2.T) * (X @ W1 + b1 > 0)))[i, j]
W1[i, j] += eps; lp = loss_fn(forward(X)[2], Y)
W1[i, j] -= 2 * eps; lm = loss_fn(forward(X)[2], Y)
W1[i, j] += eps
print("grad check:", analytic, (lp - lm) / (2 * eps))import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader, random_split
torch.manual_seed(0)
Xt, yt = torch.tensor(X, dtype=torch.float32), torch.tensor(y)
train_ds, val_ds = random_split(TensorDataset(Xt, yt), [800, 200])
train_dl = DataLoader(train_ds, batch_size=64, shuffle=True)
val_dl = DataLoader(val_ds, batch_size=256)
model = nn.Sequential(nn.Linear(2, 64), nn.ReLU(), nn.Dropout(0.1),
nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 2))
opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=1e-2)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=200)
loss_fn = nn.CrossEntropyLoss()
best, patience, bad = float("inf"), 20, 0
for epoch in range(200):
model.train()
for xb, yb in train_dl:
opt.zero_grad()
loss = loss_fn(model(xb), yb)
loss.backward() # backprop via autograd
nn.utils.clip_grad_norm_(model.parameters(), 1.0) # gradient clipping
opt.step()
sched.step()
model.eval() # disables dropout
with torch.no_grad():
val_loss = sum(loss_fn(model(xb), yb).item() * len(yb) for xb, yb in val_dl) / len(val_ds)
if val_loss < best - 1e-4:
best, bad = val_loss, 0
torch.save(model.state_dict(), "best.pt")
else:
bad += 1
if bad >= patience:
print("early stop at epoch", epoch); break
model.load_state_dict(torch.load("best.pt"))
print("best val loss:", round(best, 4))Quiz
1. Backpropagation computes gradients using…
2. Why did ReLU help train deep networks?
3. Your network can't overfit a tiny batch of 10 examples. This suggests…
4. In backprop, δℓ = (δℓ₊₁ Wℓ₊₁ᵀ) ⊙ …
5. He initialisation for ReLU layers uses Var(w) =
6. Why do residual connections help very deep networks?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Dive into Deep Learning (free book with code) ↗ Interactive textbook in PyTorch, by Zhang, Lipton, Li & Smola
- Practical Deep Learning for Coders (fast.ai) ↗ Free, code-first course