Deep Learning in Practice (PyTorch)
Tensors, autograd and the training loop; data loading, GPUs, transfer learning and fine-tuning, with three Colab notebooks.
Why a framework, and what PyTorch gives you
Everything in the deep-learning topics (layers, losses, backpropagation, optimisers) can be written in NumPy, as you did for gradient descent in the game. Real networks have millions of parameters, run on GPUs and change shape every week, so teams use a framework. PyTorch is the most widely used in research and increasingly in industry; TensorFlow/Keras and JAX are the main alternatives, and the ideas transfer directly.
PyTorch gives you three things. Tensors: n-dimensional arrays like NumPy's, which can live on a GPU (x.to("cuda")). Autograd: every operation on tensors that require gradients is recorded in a graph, and calling loss.backward() computes the gradient of the loss with respect to every parameter by the chain rule, automatically. Building blocks: layers (nn.Linear, nn.Conv2d), losses, optimisers (torch.optim.AdamW) and data utilities.
What PyTorch does not do is write the training loop for you. That is deliberate: the loop is where you control batching, evaluation, early stopping, logging and checkpoints, and where most bugs live.
The training loop, line by line
Almost every PyTorch project contains the same loop. For each epoch (one pass over the training data) and each mini-batch:
model.train()switches on training behaviour (dropout active, batch-norm statistics updating).optimizer.zero_grad()clears the gradients from the previous batch. PyTorch adds new gradients to old ones, so forgetting this line silently corrupts training.- Forward pass:
logits = model(xb), thenloss = loss_fn(logits, yb). - Backward pass:
loss.backward()fills.gradfor every parameter. optimizer.step()updates the parameters using those gradients.
After each epoch, evaluate on the validation set inside model.eval() and with torch.no_grad(): (no dropout, no graph, much less memory). Keep the weights from the best validation epoch, stop when validation stops improving, and save a checkpoint (torch.save(model.state_dict(), path)) together with everything needed to use it: preprocessing statistics, label names, a version.
Two loss details catch many people. For binary classification, output a raw logit and use BCEWithLogitsLoss, which applies the sigmoid internally in a numerically stable way. For multi-class, use CrossEntropyLoss on logits: it already contains the softmax, so adding your own softmax layer is a bug.
Try it: review a training loop
A teammate's training script has several classic bugs. Click every line you would comment on in code review, then submit.
Interactive: this part runs in your browser.
Data loading, GPUs and memory
A Dataset knows how to return one example (__getitem__) and how many there are (__len__). A DataLoader wraps it to produce batches, shuffle the training data each epoch, and load data in parallel with num_workers. Augmentations (random crops, flips, noise) belong in the training dataset's transforms only; validation and test data must be transformed deterministically.
To use a GPU, move both the model and every batch to the same device: model.to(device) and xb.to(device). "Expected all tensors to be on the same device" is the most common PyTorch error. GPU memory is used by the weights, the optimiser state (Adam keeps two extra numbers per parameter), and the activations saved for the backward pass, which grow with batch size. If you run out of memory: reduce the batch size, use mixed precision (torch.autocast, 16-bit activations), or use gradient accumulation (several small batches per optimiser step).
Reproducibility needs care: set seeds (torch.manual_seed, NumPy, Python), log library versions, and expect tiny differences on GPUs unless deterministic algorithms are forced.
Transfer learning and fine-tuning
Training large networks from scratch needs huge datasets and compute. In practice, almost every image and text project starts from a pretrained model and adapts it. For images, a CNN or vision transformer pretrained on ImageNet already detects edges, textures and object parts; you replace the final layer with one for your classes. For text, a pretrained transformer such as DistilBERT already understands language; you add a classification head.
A standard recipe: (1) replace the head; (2) freeze the backbone and train only the head for an epoch or two, so the random head does not send large, destructive gradients into good features; (3) unfreeze and fine-tune everything with a small learning rate (around 1e-4 for CNNs, 2e-5 to 5e-5 for transformers), often with warm-up and a decaying schedule; (4) evaluate per class and analyse errors. With a few thousand labelled examples this usually beats training from scratch by a wide margin.
The Hugging Face transformers library provides thousands of pretrained models and tokenizers with a consistent API (AutoTokenizer, AutoModelForSequenceClassification). Before using one, check its licence, what data it was trained on, its size and latency, and whether it can run where your data is allowed to go.
Hands-on: three Colab notebooks
Deep-learning code needs a GPU and libraries that do not run inside a web page, so these three notebooks run in Google Colab (free, in your browser). Copy a notebook's code with the button and paste it into a new Colab notebook, or upload the .ipynb file you downloaded with the course. Each notebook ends with "Try next" experiments.
Interactive: this part runs in your browser.
The maths: backpropagation through a layer
For a linear layer followed by a loss , let be the gradient arriving from above. The chain rule gives the three gradients autograd computes:
The last one is passed down to the previous layer, which is why the algorithm is called back-propagation. For a ReLU, is passed through where and zeroed elsewhere. For a mini-batch (one example per row), the weight gradient is averaged over the batch.
With softmax and cross-entropy combined, the gradient with respect to the logits is simply (predicted probabilities minus the one-hot label). That compact, stable form is why CrossEntropyLoss takes logits rather than probabilities.
AdamW keeps running averages of the gradient and its square and updates , where the weight-decay term is applied separately from the adaptive step (the "W" in AdamW).
Advanced: making training fast and reliable
Mixed precision (torch.autocast with a gradient scaler, or bfloat16 on recent GPUs) roughly halves memory and speeds up training. Gradient clipping (clip_grad_norm_) prevents rare huge gradients from wrecking training, especially in RNNs and transformers. Learning-rate schedules (warm-up, cosine or linear decay) are standard for transformers. Gradient accumulation simulates large batches on small GPUs.
Debugging recipe: first overfit a single small batch (the loss should go near zero; if not, there is a bug), check that the initial loss matches theory (about for balanced classification), plot learning curves, and look at predictions on real examples.
Scaling up: DistributedDataParallel trains on several GPUs; checkpoint regularly so long runs can resume; track every run (config, data version, code commit, metrics) with an experiment tracker such as MLflow or Weights & Biases. Export models for serving with TorchScript or ONNX, and measure latency on the target hardware.
Quiz
1. What happens if you forget optimizer.zero_grad()?
2. Which pair belongs in the validation loop?
3. Your model ends with nn.Softmax and you use nn.CrossEntropyLoss. This isโฆ
4. "Expected all tensors to be on the same device" meansโฆ
5. Why freeze the backbone for the first epochs of fine-tuning?
๐ Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- PyTorch: Learn the basics โ Official tutorials
- Practical Deep Learning for Coders (fast.ai) โ Free, code-first course