Optimisers & Learning-Rate Schedules
Mini-batch SGD, momentum, RMSProp, Adam and AdamW, why the learning rate matters most, warm-up and cosine schedules, and gradient clipping.
Mini-batch stochastic gradient descent
Training minimises the average loss over the training set. Computing the exact gradient needs a pass over all the data, which is too slow, so networks use mini-batch SGD: take a random batch of B examples (typically 32 to 1,024), compute the gradient on it, and step . One pass over the data is an epoch; reshuffle every epoch.
The batch gradient is a noisy estimate of the true gradient. That noise is not all bad: it helps escape sharp minima and acts as a mild regulariser. Larger batches give less noisy gradients and use the GPU better, but each epoch has fewer steps, and very large batches often generalise slightly worse unless the learning rate and schedule are adjusted.
Momentum: rolling through ravines
Loss surfaces often look like long narrow ravines: steep across, shallow along. Plain SGD bounces between the walls and creeps along the floor. Momentum keeps a running "velocity", an exponentially weighted average of past gradients:
Components that keep pointing the same way (along the ravine) add up; components that flip sign (across it) cancel. Nesterov momentum evaluates the gradient slightly ahead, at , which is usually a little better. SGD with Nesterov momentum and a good schedule is still the optimiser of choice for many CNNs.
Adaptive learning rates: RMSProp, Adam, AdamW
Different parameters need different step sizes: rare features get small, infrequent gradients; others get large ones. RMSProp divides each parameter's step by a running root-mean-square of its recent gradients. Adam combines this with momentum and corrects both averages for starting at zero:
with defaults . Adam is forgiving and works well with little tuning, which makes it the default for transformers and most new projects (a learning rate of is a famous first guess).
AdamW fixes how Adam handles weight decay. Adding to the gradient (classic L2) gets divided by too, so heavily-updated weights are barely decayed. AdamW instead decouples it: . Use AdamW, typically with between 0.01 and 0.1, and don't decay biases and normalisation parameters.
Try it: an optimiser race
Four optimisers start from the same point on a narrow ravine (the minimum is the cross). Change the learning rate and add gradient noise like a mini-batch would. Which one is robust to the learning rate?
Interactive: this part runs in your browser.
The learning rate is the most important hyperparameter
Too high: the loss oscillates, spikes or becomes NaN. Too low: it decreases painfully slowly and may stall. If you only tune one thing, tune the learning rate, on a log scale (1e-4, 3e-4, 1e-3, 3e-3…).
A quick way to find a good range is the learning-rate range test: train for a few hundred steps while increasing the learning rate exponentially from 1e-7 to 1, and plot the loss. Pick a value about ten times smaller than where the loss is lowest, before it blows up.
The rate also should change during training: large early to make progress, small late to settle into a good minimum. Common schedules:
- Step decay: divide by 10 at fixed epochs (classic for ResNets).
- Cosine decay: smoothly from the peak to near zero along half a cosine wave.
- Linear warm-up for the first few percent of steps, then decay. Warm-up avoids huge, unstable early updates while Adam's statistics and normalisation layers settle; it is standard for transformers.
- One-cycle: warm up to a high rate, then anneal down: often trains fast.
Try it: learning-rate schedules
Compare the shapes of common schedules over a training run, and the effect of warm-up length.
Interactive: this part runs in your browser.
Gradient clipping and accumulation
A single bad batch can produce a huge gradient that throws the parameters far away (loss spikes, NaNs), especially in RNNs and transformers. Gradient clipping by global norm rescales the whole gradient when its norm exceeds a threshold c (often 1.0): . The direction is kept; only the length is capped.
Gradient accumulation simulates a larger batch when it does not fit in memory: run k small batches, add up their gradients (dividing each loss by k), and step once. Remember that batch-normalisation statistics still see only the small batches.
Code example: write SGD-momentum and Adam
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code example: warm-up, cosine decay and clipping
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: optimiser, schedule and clipping in PyTorch
The standard recipe for a transformer-style model.
import torch, math
decay, no_decay = [], []
for n, p in model.named_parameters():
(no_decay if p.ndim < 2 else decay).append(p) # no decay on biases / norm weights
opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.05},
{"params": no_decay, "weight_decay": 0.0}], lr=3e-4, betas=(0.9, 0.95))
total, warm = 10_000, 500
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: s / warm if s < warm
else 0.5 * (1 + math.cos(math.pi * (s - warm) / (total - warm))))
for step, (xb, yb) in zip(range(total), loader):
loss = loss_fn(model(xb), yb)
opt.zero_grad(set_to_none=True)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
opt.step(); sched.step()Quiz
1. What does momentum do in a narrow ravine?
2. Why does Adam divide by √ŝ?
3. What is the difference between Adam with L2 and AdamW?
4. Your loss becomes NaN in the first 50 steps. The most likely fix is…
5. Why use a learning-rate warm-up for transformers?
6. Clipping the gradient by global norm…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Dive into Deep Learning (free book with code) ↗ Interactive textbook in PyTorch, by Zhang, Lipton, Li & Smola
- Deep Learning Tuning Playbook (Google) ↗ How to tune hyperparameters systematically
- Understanding Deep Learning (Prince) ↗ Modern, clear and free textbook