Image Classification in Practice
Choosing and fine-tuning a backbone, imbalanced and multi-label data, evaluation and calibration, and seeing what the model looks at with Grad-CAM.
The standard recipe
- Split correctly: by patient, site or document, not by image, so near-duplicates don't leak across splits.
- Start from a pretrained backbone (ResNet-50, EfficientNet, ConvNeXt or a ViT from timm), replace the head, use its preprocessing.
- Train the head, then fine-tune with AdamW, a small learning rate, warm-up and cosine decay, augmentation, label smoothing and early stopping.
- Evaluate with the right metric, per class and per subgroup, and look at the errors.
- Iterate on data (relabel confusing classes, add examples of failure cases) more than on architecture.
Picking a backbone is a trade-off between accuracy, speed and memory; timm lists ImageNet accuracy and throughput for each model. For small datasets, a mid-sized model with a strong pretrained checkpoint usually beats a giant one.
Imbalance, multi-label and calibration
Imbalanced classes are the norm in medicine and industry (most images are normal). Report per-class recall and precision, PR-AUC or balanced accuracy rather than accuracy; use a weighted loss or oversample the rare class; and choose the decision threshold from costs.
Multi-label problems (an image can show several findings) use one sigmoid per label with BCEWithLogitsLoss instead of softmax, and a threshold per label.
Calibration: modern networks are often over-confident: a predicted 0.95 may be right only 80% of the time. Check with a reliability diagram or expected calibration error, and fix it with temperature scaling (divide the logits by a single T fitted on validation data). Calibrated probabilities matter whenever someone acts on the number, such as a triage threshold.
What is the model looking at? Grad-CAM
A CNN can be right for the wrong reason: a famous pneumonia model learned to recognise the hospital's portable X-ray machine markers rather than the lungs. Class activation maps show which regions drove a prediction. Grad-CAM takes the last convolutional feature maps A (K channels), weights each channel by the average gradient of the class score with respect to it, sums, and applies ReLU:
The map is upsampled and overlaid on the image. Use it to audit models (is it looking at the lesion, or at a ruler, a watermark, a text label?), not as proof that a prediction is correct: saliency maps can be misleading, and they should be checked across many images.
Try it: is the model looking at the right thing?
Two models classify the same lesion photos with similar accuracy. Compare their class-activation maps: one has learned a shortcut.
Interactive: this part runs in your browser.
Code example: Grad-CAM and calibration
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: fine-tune a timm model
The core of notebook 09.
import timm, torch
model = timm.create_model("convnext_tiny.fb_in22k", pretrained=True, num_classes=4)
cfg = timm.data.resolve_data_config({}, model=model) # the model's own preprocessing
train_tf = timm.data.create_transform(**cfg, is_training=True, auto_augment="rand-m7-mstd0.5")
eval_tf = timm.data.create_transform(**cfg)
class_w = torch.tensor([1.0, 1.0, 3.0, 5.0]) # rarer classes weigh more
loss_fn = torch.nn.CrossEntropyLoss(weight=class_w, label_smoothing=0.1)
opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.05)
# after training: fit temperature T on validation logits, then probs = softmax(logits / T)Quiz
1. Your X-ray dataset has several images per patient. You should splitโฆ
2. An image may show several findings at once. The output layer should useโฆ
3. Temperature scalingโฆ
4. Grad-CAM shows a skin-lesion model focusing on a ruler next to malignant lesions. This meansโฆ
5. With 2% positives, which metric is most informative?
๐ Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- timm documentation (Hugging Face) โ Pretrained image models with benchmarks
- Grad-CAM (Selvaraju et al., 2016) โ Visual explanations from deep networks
- CS231n: Deep Learning for Computer Vision (Stanford) โ Notes on backprop, CNNs and training