PyTorch Essentials
The library step by step: tensors and devices, autograd, nn.Module, Dataset and DataLoader, the training loop, evaluation mode, saving and loading, and the errors everyone hits.
What PyTorch gives you
PyTorch is NumPy with three extra powers: tensors can live on a GPU, operations are recorded so gradients are computed automatically (autograd), and it ships building blocks for networks, optimisers and data loading. Prerequisites: NumPy and the ideas in Neural Networks and Training Neural Networks.
PyTorch cannot run inside this web page, so the browser exercise below writes the same pattern in NumPy, and notebook 13 (at the end of this topic) runs real PyTorch in Google Colab with a free GPU.
Tensors: shape, dtype, device
import torch x = torch.tensor([[1., 2.], [3., 4.]]) # from data z = torch.zeros(3, 4); r = torch.randn(64, 10) # common constructors x.shape, x.dtype, x.device # check these constantly x.mean(dim=0), x.sum(dim=1, keepdim=True) # 'dim' is NumPy's 'axis' x @ x.T, x.view(4), x.unsqueeze(0).shape # matmul, reshape, add an axis -> (1, 2, 2) device = "cuda" if torch.cuda.is_available() else "cpu" x = x.to(device) # move to the GPU x.cpu().numpy(), torch.from_numpy(a) # to / from NumPy (CPU only)
Rules that prevent most errors: model and data must be on the same device; class labels for cross_entropy are torch.long; inputs are usually float32.
Autograd and nn.Module
Set requires_grad=True (parameters have it already) and call loss.backward(): every parameter's .grad receives d(loss)/d(parameter). Gradients accumulate, which is why each step starts with zero_grad().
from torch import nn
class MLP(nn.Module):
def __init__(self, d_in, hidden, n_out):
super().__init__() # always call this first
self.net = nn.Sequential(nn.Linear(d_in, hidden), nn.ReLU(), nn.Dropout(0.1),
nn.Linear(hidden, n_out))
def forward(self, x):
return self.net(x) # return logits; the loss applies softmax
model = MLP(20, 64, 3)
sum(p.numel() for p in model.parameters()) # parameter count
Layers assigned as attributes in __init__ are registered, so model.parameters(), model.to(device) and model.state_dict() find them. A plain Python list of layers is not registered: use nn.ModuleList.
Try it: build a network and track the shapes
Stack layers and see the output shape and parameter count of each one, as torchinfo.summary would show.
Interactive: this part runs in your browser.
Dataset, DataLoader and the training loop
from torch.utils.data import Dataset, DataLoader
class TableData(Dataset):
def __init__(self, X, y): self.X = torch.tensor(X, dtype=torch.float32); self.y = torch.tensor(y)
def __len__(self): return len(self.y)
def __getitem__(self, i): return self.X[i], self.y[i]
train_dl = DataLoader(TableData(X_tr, y_tr), batch_size=64, shuffle=True) # shuffle training only
val_dl = DataLoader(TableData(X_va, y_va), batch_size=256)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
for epoch in range(20):
model.train() # dropout / batch norm in training mode
for xb, yb in train_dl:
xb, yb = xb.to(device), yb.to(device)
loss = nn.functional.cross_entropy(model(xb), yb)
opt.zero_grad(); loss.backward(); opt.step()
model.eval() # evaluation mode
with torch.no_grad(): # no graph: faster, less memory
val_loss = sum(nn.functional.cross_entropy(model(x.to(device)), y.to(device), reduction="sum").item()
for x, y in val_dl) / len(val_dl.dataset)
Code example: the PyTorch pattern in NumPy
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Saving, loading and predicting
torch.save(model.state_dict(), "model.pt") # save the weights, not the object
model = MLP(20, 64, 3); model.load_state_dict(torch.load("model.pt", map_location="cpu")); model.eval()
with torch.inference_mode():
probs = torch.softmax(model(x_new), dim=1)
# to resume training, also save opt.state_dict() and the epoch number
The errors everyone hits
| Error message (shortened) | Usual cause | Fix |
|---|---|---|
| Expected all tensors to be on the same device | model on GPU, batch on CPU (or the reverse) | xb.to(device) inside the loop; model.to(device) once |
| mat1 and mat2 shapes cannot be multiplied (64x400 and 256x10) | wrong in_features after a reshape or conv | print x.shape before the layer; fix nn.Linear(400, …) |
| expected scalar type Long but found Float | class labels stored as floats | y.long() |
| CUDA out of memory | batch too big, or keeping graphs alive (losses.append(loss)) | smaller batch; loss.item(); mixed precision; torch.no_grad() in evaluation |
| Trying to backward through the graph a second time | reusing a graph across steps | recompute the forward pass each step; .detach() values you keep |
| (no error) loss stays flat or validation numbers change between runs | missing zero_grad, wrong learning rate, or forgot model.eval() | see Debugging Training |
Hands-on: PyTorch in Colab (notebook 13)
Tensors, autograd, a Dataset and DataLoader, an MLP, a training loop with validation curves, and saving and reloading, with exercises at the end. Copy the code into a new Colab notebook or upload the .ipynb.
Interactive: this part runs in your browser.
Cheat sheet: PyTorch
The commands you will use most, on one screen. Come back to it while you work.
Interactive: this part runs in your browser.
Quiz
1. Why call opt.zero_grad() every step?
2. What does model.eval() change?
3. Your labels are float tensors and cross_entropy fails. Fix?
4. What should you save to reload a model later?
5. Layers stored in a plain Python list inside a Module are…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- PyTorch: Learn the Basics (official tutorial) ↗ tensors, datasets, models, autograd, optimisation, saving
- PyTorch: Datasets & DataLoaders ↗ writing your own Dataset
- Dive into Deep Learning (free book with code) ↗ Interactive textbook in PyTorch, by Zhang, Lipton, Li & Smola