Image Segmentation
Labelling every pixel: semantic and instance segmentation, the U-Net encoder-decoder, Dice and IoU, losses for small structures, and medical-imaging practice.
Semantic and instance segmentation
Semantic segmentation assigns a class to every pixel (wound, healthy skin, background). The output has the same height and width as the input, with one channel per class. Instance segmentation also separates individual objects of the same class (each cell, each wound); Mask R-CNN does it by predicting a mask inside each detected box. Panoptic segmentation combines both. Foundation models such as SAM (Segment Anything) produce masks from a click or a box prompt and are increasingly used to speed up labelling.
Watch it: U-Net: down to understand, up to locate, skips for detail
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
U-Net: an encoder-decoder with skip connections
Classification networks shrink the image to a single vector; segmentation must output a full-resolution map. The U-Net (2015, designed for biomedical images) solves this with two halves:
- an encoder (a CNN, often pretrained) that downsamples step by step, learning what is in the image;
- a decoder that upsamples back to full size (transposed convolutions or interpolation + convolution);
- skip connections that copy encoder feature maps at each resolution and concatenate them to the decoder's, restoring the fine detail of where boundaries are.
The last layer is a 1 × 1 convolution giving one logit per class per pixel. U-Net and its descendants (with ResNet or EfficientNet encoders, nnU-Net's self-configuring version) remain the default for medical segmentation; transformer-based models (SegFormer, Mask2Former) are common on natural images.
Dice, IoU and segmentation losses
Pixel accuracy is misleading: if the wound covers 3% of the image, predicting "background" everywhere is 97% accurate. Segmentation is measured by overlap between the predicted mask P and the true mask G:
Report them per class and per image (averaged over images), and also check boundary errors (Hausdorff distance) when outlines matter clinically. For training, plain cross-entropy is dominated by the many background pixels; Dice loss (1 − soft Dice on the predicted probabilities) or focal loss focus on the small foreground, and a sum of cross-entropy and Dice loss is a robust default.
Try it: threshold a probability map
A model outputs a probability for every pixel of a wound photo. Move the threshold and compare pixel accuracy, IoU and Dice against the nurse's outline.
Interactive: this part runs in your browser.
Segmentation in medical imaging
- Labels are expensive and vary between experts: two clinicians outline a wound differently. Measure inter-rater agreement (Dice between annotators); the model cannot be expected to beat it.
- Images come from different devices and sites: evaluate on data from a site not used in training.
- 3-D volumes (CT, MRI) use 3-D convolutions or slice-by-slice models; patches are used to fit memory.
- Physical units matter: convert pixels to mm² using the pixel spacing (from DICOM metadata, or a calibration marker in photos) before reporting areas.
Code example: Dice, IoU and U-Net shapes
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: a U-Net with a pretrained encoder
Using the segmentation-models-pytorch library.
import segmentation_models_pytorch as smp, torch
model = smp.Unet(encoder_name="resnet34", encoder_weights="imagenet", in_channels=3, classes=1)
dice = smp.losses.DiceLoss(mode="binary"); bce = torch.nn.BCEWithLogitsLoss()
loss_fn = lambda logits, mask: bce(logits, mask) + dice(logits, mask)
x = torch.randn(2, 3, 256, 256); print(model(x).shape) # (2, 1, 256, 256): one logit per pixel
# metrics on thresholded predictions
tp, fp, fn, tn = smp.metrics.get_stats(model(x).sigmoid(), torch.randint(0, 2, (2, 1, 256, 256)), mode="binary", threshold=0.5)
print(smp.metrics.iou_score(tp, fp, fn, tn, reduction="micro-imagewise"))Quiz
1. A lesion covers 2% of each image. A model predicts "background" everywhere. Its pixel accuracy and Dice are…
2. What do U-Net's skip connections do?
3. IoU is 0.5. What is Dice?
4. Two nurses outlining the same wounds agree with Dice 0.82. A model scoring Dice 0.80 against one nurse is…
5. Why is Dice loss often added to cross-entropy for small structures?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- U-Net (Ronneberger et al., 2015) ↗ The original U-Net paper
- MONAI ↗ Open-source framework for medical imaging AI