Computer Vision
Topic 2 ยท Computer Vision ยท โฑ about 5 h

Images as Data, Preprocessing & Augmentation

Pixels, channels and colour spaces, resizing and normalisation, file formats including medical images, and data augmentation that keeps labels correct.

Show:
In this topic
Beginner

An image is an array

A greyscale image is a 2-D array of brightness values; a colour image adds a channel axis, usually red, green and blue. Files store 8-bit integers from 0 to 255; models want floats. Watch the layout: PIL and OpenCV give (H, W, C), PyTorch wants (C, H, W), and OpenCV loads colours as BGR, not RGB, a classic bug that silently swaps red and blue.

Other colour spaces are sometimes useful: HSV separates colour (hue) from brightness, which helps when lighting varies; LAB approximates human perception. Medical images are different again: X-rays and CT come as DICOM files with 12โ€“16-bit values and metadata (patient, scanner, pixel spacing); CT values are in Hounsfield units and are "windowed" (clipped to a range such as soft tissue or bone) before training. Microscopy may have many channels.

Beginner

Resizing and normalisation

Networks expect a fixed input size (224 ร— 224 is common for classification). Resizing choices matter: squashing changes aspect ratios (a round lesion becomes an oval); resize the short side then centre-crop, or letterbox (pad to a square), keeps shapes. Small objects can vanish when a large image is downscaled, so detection on high-resolution scans often uses tiles.

Normalise pixels the same way the pretrained model was trained: typically scale to [0, 1], then subtract the ImageNet mean (0.485, 0.456, 0.406) and divide by the std (0.229, 0.224, 0.225) per channel. Use exactly the same preprocessing at inference; a mismatch is one of the commonest production bugs in vision.

Beginner

Data augmentation

Augmentation creates new training images with transformations that do not change the label. It is the most effective regulariser in vision. Common families:

  • Geometric: random resized crops, flips, small rotations, scaling, perspective.
  • Photometric: brightness, contrast, saturation, hue, blur, noise, JPEG compression.
  • Occlusion and mixing: Cutout / random erasing, MixUp and CutMix (blend two images and their labels).
  • Automatic policies: RandAugment, TrivialAugment (random choices from a list of transforms).

Two rules. First, transforms must be valid for your task: horizontal flips are fine for photos of clothes but swap left and right anatomy in a chest X-ray; colour jitter can destroy the signal in skin images where colour is diagnostic. Second, for detection and segmentation, the boxes and masks must be transformed together with the image (Albumentations does this). Augment training data only; validation and test use the deterministic resize and normalise.

Try it

Try it: augmentation playground

Apply augmentations to a small image and its bounding box. Notice how the box must move with the image, and which transforms would change the meaning of a medical image.

Interactive: this part runs in your browser.

Try it

Code example: augmentation that moves the boxes too

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: torchvision and Albumentations

Training and evaluation transforms for classification, and a box-aware pipeline for detection.

Python
from torchvision import transforms as T
train_tf = T.Compose([T.RandomResizedCrop(224, scale=(0.7, 1.0)), T.RandomHorizontalFlip(),
                      T.ColorJitter(0.2, 0.2, 0.1), T.ToTensor(),
                      T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
eval_tf = T.Compose([T.Resize(256), T.CenterCrop(224), T.ToTensor(),
                     T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

import albumentations as A                     # boxes and masks move with the image
det_tf = A.Compose([A.RandomSizedBBoxSafeCrop(512, 512), A.HorizontalFlip(), A.RandomBrightnessContrast()],
                   bbox_params=A.BboxParams(format="pascal_voc", label_fields=["labels"]))
out = det_tf(image=img, bboxes=boxes, labels=labels)

import cv2
img = cv2.cvtColor(cv2.imread("scan.jpg"), cv2.COLOR_BGR2RGB)   # OpenCV loads BGR!
This code needs torchvision, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab โ†— (free; Runtime โ†’ Change runtime type โ†’ GPU for deep learning).
Check yourself

Quiz

1. You load an image with OpenCV and feed it to an ImageNet model without conversion. What is wrong?

2. Which augmentation is risky for chest X-rays?

3. For object detection, a random crop mustโ€ฆ

4. At inference, images should be normalisedโ€ฆ

5. Why is resizing a 4000 ร— 3000 slide straight to 224 ร— 224 a problem for detecting small cells?

Go deeper

๐Ÿ“š Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready