Computer Vision
Topic 3 ยท Computer Vision ยท โฑ about 8 h

Image Classification in Practice

Choosing and fine-tuning a backbone, imbalanced and multi-label data, evaluation and calibration, and seeing what the model looks at with Grad-CAM.

Show:
In this topic
Beginner

The standard recipe

  1. Split correctly: by patient, site or document, not by image, so near-duplicates don't leak across splits.
  2. Start from a pretrained backbone (ResNet-50, EfficientNet, ConvNeXt or a ViT from timm), replace the head, use its preprocessing.
  3. Train the head, then fine-tune with AdamW, a small learning rate, warm-up and cosine decay, augmentation, label smoothing and early stopping.
  4. Evaluate with the right metric, per class and per subgroup, and look at the errors.
  5. Iterate on data (relabel confusing classes, add examples of failure cases) more than on architecture.

Picking a backbone is a trade-off between accuracy, speed and memory; timm lists ImageNet accuracy and throughput for each model. For small datasets, a mid-sized model with a strong pretrained checkpoint usually beats a giant one.

Beginner

Imbalance, multi-label and calibration

Imbalanced classes are the norm in medicine and industry (most images are normal). Report per-class recall and precision, PR-AUC or balanced accuracy rather than accuracy; use a weighted loss or oversample the rare class; and choose the decision threshold from costs.

Multi-label problems (an image can show several findings) use one sigmoid per label with BCEWithLogitsLoss instead of softmax, and a threshold per label.

Calibration: modern networks are often over-confident: a predicted 0.95 may be right only 80% of the time. Check with a reliability diagram or expected calibration error, and fix it with temperature scaling (divide the logits by a single T fitted on validation data). Calibrated probabilities matter whenever someone acts on the number, such as a triage threshold.

Beginner

What is the model looking at? Grad-CAM

A CNN can be right for the wrong reason: a famous pneumonia model learned to recognise the hospital's portable X-ray machine markers rather than the lungs. Class activation maps show which regions drove a prediction. Grad-CAM takes the last convolutional feature maps A (K channels), weights each channel by the average gradient of the class score with respect to it, sums, and applies ReLU:

ฮฑk=1HWโˆ‘i,jโˆ‚ycโˆ‚Aijk,Lc=ReLUโก(โˆ‘kฮฑkAk).\alpha_k=\frac{1}{HW}\sum_{i,j}\frac{\partial y^c}{\partial A^k_{ij}},\qquad L^c=\operatorname{ReLU}\Big(\sum_k\alpha_kA^k\Big).

The map is upsampled and overlaid on the image. Use it to audit models (is it looking at the lesion, or at a ruler, a watermark, a text label?), not as proof that a prediction is correct: saliency maps can be misleading, and they should be checked across many images.

Try it

Try it: is the model looking at the right thing?

Two models classify the same lesion photos with similar accuracy. Compare their class-activation maps: one has learned a shortcut.

Interactive: this part runs in your browser.

Try it

Code example: Grad-CAM and calibration

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: fine-tune a timm model

The core of notebook 09.

Python ยท timm
import timm, torch
model = timm.create_model("convnext_tiny.fb_in22k", pretrained=True, num_classes=4)
cfg = timm.data.resolve_data_config({}, model=model)          # the model's own preprocessing
train_tf = timm.data.create_transform(**cfg, is_training=True, auto_augment="rand-m7-mstd0.5")
eval_tf = timm.data.create_transform(**cfg)
class_w = torch.tensor([1.0, 1.0, 3.0, 5.0])                  # rarer classes weigh more
loss_fn = torch.nn.CrossEntropyLoss(weight=class_w, label_smoothing=0.1)
opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.05)
# after training: fit temperature T on validation logits, then probs = softmax(logits / T)
This code needs timm, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab โ†— (free; Runtime โ†’ Change runtime type โ†’ GPU for deep learning).
Check yourself

Quiz

1. Your X-ray dataset has several images per patient. You should splitโ€ฆ

2. An image may show several findings at once. The output layer should useโ€ฆ

3. Temperature scalingโ€ฆ

4. Grad-CAM shows a skin-lesion model focusing on a ruler next to malignant lesions. This meansโ€ฆ

5. With 2% positives, which metric is most informative?

Go deeper

๐Ÿ“š Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready