Deep Learning
Topic 16 · Deep Learning · ⏱ about 8 h

Efficient Training & Deployment

Where GPU memory goes, mixed precision, gradient checkpointing, multi-GPU training, and making models small and fast: quantisation, pruning, distillation and export.

Show:
In this topic
Beginner

Where the memory goes

"CUDA out of memory" is a daily event in deep learning. Training memory has four parts:

  • Weights: N parameters × bytes per number (4 in float32, 2 in bf16/fp16).
  • Gradients: the same size as the weights.
  • Optimiser state: Adam keeps two extra numbers per parameter (m and s), usually in float32: 8 bytes per parameter. SGD with momentum keeps one.
  • Activations saved for the backward pass: grow with batch size × sequence length (or image size) × depth, and often dominate.

A rule of thumb for full training with Adam in mixed precision is about 16 bytes per parameter before activations: a 1-billion-parameter model needs ~16 GB just for weights, gradients and optimiser state. Inference needs only the weights (plus activations for one batch and, for LLMs, the KV cache): 2 bytes per parameter in bf16, 0.5 in 4-bit.

Beginner

Watch it: Quantisation: snapping weights to 8-bit integers

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Try it

Try it: will it fit on the GPU?

Estimate memory for training or serving a model. Change precision, optimiser, batch size and sequence length, and switch on gradient checkpointing.

Interactive: this part runs in your browser.

Beginner

Training faster and bigger

  • Mixed precision: compute in bf16 or fp16 while keeping a float32 master copy of the weights. It roughly halves activation memory and doubles throughput on modern GPUs. fp16 needs loss scaling to stop small gradients underflowing to zero; bf16 has float32's range and usually does not. In PyTorch: torch.autocast.
  • Gradient checkpointing: store only some activations and recompute the rest during the backward pass: much less memory for about 30% more compute.
  • Gradient accumulation for a larger effective batch (see Optimisers).
  • Data parallel training (PyTorch DDP): each GPU holds a full model copy and processes different batches; gradients are averaged across GPUs each step.
  • Sharded training (FSDP, DeepSpeed ZeRO) splits weights, gradients and optimiser state across GPUs when the model is too large for one; tensor and pipeline parallelism split individual layers or groups of layers for the very largest models.
  • Keep the GPU busy: enough DataLoader workers, pinned memory, and profiling (a GPU at 30% utilisation is usually waiting for data).
Beginner

Making models small and fast

A model is only useful if it is fast and cheap enough where it runs: a hospital server, a phone, a browser. The main tools:

  • Quantisation: store weights (and possibly activations) as 8-bit or 4-bit integers instead of 32-bit floats: 4 to 8 times smaller and often faster. Each tensor (or each output channel) gets a scale: q=round⁡(w/s)q=\operatorname{round}(w/s), s=max⁡∣w∣/127s=\max|w|/127. Post-training quantisation needs no retraining, only a small calibration set; quantisation-aware training recovers accuracy when PTQ loses too much.
  • Pruning: remove weights or whole channels/heads that matter least. Structured pruning (whole channels) gives real speed-ups; scattered zeros usually do not without special hardware.
  • Knowledge distillation: train a small "student" to match the softened output probabilities of a large "teacher". The student often keeps most of the teacher's accuracy at a fraction of the cost (DistilBERT keeps about 97% of BERT's quality with 40% fewer parameters).
  • Export and runtimes: ONNX, TorchScript / torch.compile, TensorRT, OpenVINO, and llama.cpp-style runtimes for LLMs.

Always measure accuracy on the real evaluation set after each compression step, plus latency at the 50th and 95th percentiles and throughput on the target hardware, with realistic batch sizes. Check that accuracy did not drop more for some groups than for others.

Try it

Code example: quantise a model to int8

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: mixed precision, checkpointing, quantisation, export

Common snippets.

Python · PyTorch
import torch
# mixed precision training (bf16 on recent GPUs)
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
    loss = loss_fn(model(xb), yb)
loss.backward(); opt.step()

# gradient checkpointing for a block
from torch.utils.checkpoint import checkpoint
y = checkpoint(block, x, use_reentrant=False)

# dynamic int8 quantisation of Linear layers (CPU inference)
qmodel = torch.ao.quantization.quantize_dynamic(model.eval(), {torch.nn.Linear}, dtype=torch.qint8)

# export for serving
torch.onnx.export(model.eval(), torch.randn(1, 3, 224, 224), "model.onnx",
                  input_names=["image"], output_names=["logits"], dynamic_axes={"image": {0: "batch"}})
compiled = torch.compile(model)          # graph compilation for faster training/inference
This code needs PyTorch, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. Roughly how much memory does full Adam training need per parameter, before activations, in mixed precision?

2. You get "CUDA out of memory" while training. Which change does NOT reduce memory?

3. Why does fp16 training need loss scaling while bf16 usually does not?

4. Knowledge distillation trains…

5. After int8 quantisation, what must you check?

6. DDP (data parallel) training means…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready