Vision Transformers, Self-Supervision & CLIP
Images as sequences of patches, how ViTs compare with CNNs, self-supervised pretraining (DINO, masked autoencoders), and CLIP for zero-shot classification and search.
An image is worth 16 × 16 words
A Vision Transformer (ViT) cuts the image into fixed-size patches (e.g. 16 × 16 pixels: a 224 × 224 image gives 14 × 14 = 196 patches), flattens each patch and maps it with a linear layer to a vector: a "token". A learned position embedding is added, a special [CLS] token is prepended, and the sequence goes through ordinary transformer blocks. The [CLS] token's final vector (or the average of all tokens) feeds the classifier.
Compared with CNNs, ViTs have weaker built-in assumptions (no locality or weight sharing over space), so they need more data or strong pretraining, but they scale very well and attention can relate distant parts of the image from the first layer. In practice: with a good pretrained checkpoint, ViTs and modern CNNs (ConvNeXt) perform similarly; CNNs are often cheaper at high resolution and on small devices. Hybrid and hierarchical designs (Swin) bring locality back.
Watch it: CLIP: pulling matching images and captions together
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Try it: patches and zero-shot labels
See how an image becomes a sequence of patch tokens, then classify images zero-shot by comparing an image embedding with text embeddings of label prompts, as CLIP does.
Interactive: this part runs in your browser.
Self-supervised pretraining
Labels are the bottleneck in vision, especially in medicine. Self-supervised learning pretrains on unlabelled images with a task made from the data itself:
- Contrastive (SimCLR, MoCo): two augmented views of the same image should have similar embeddings, and views of different images dissimilar ones.
- Self-distillation (DINO, DINOv2): a student network matches a slowly-updated teacher's output on different views; its features are excellent for retrieval, clustering and linear probes.
- Masked autoencoders (MAE): hide 75% of the patches and reconstruct them.
A hospital with 100,000 unlabelled scans and 1,000 labelled ones can pretrain (or continue pretraining) on the unlabelled set, then fine-tune on the labelled one: often a large gain over ImageNet initialisation alone.
CLIP: images and text in one space
CLIP trains an image encoder and a text encoder together on hundreds of millions of image–caption pairs, so that each image's embedding is close (high cosine similarity) to its own caption's and far from the others. The result is a shared space for images and language, which enables:
- Zero-shot classification: embed prompts such as "a photo of a {label}" for each class and pick the one most similar to the image, with no training.
- Text-to-image search and image-to-image search over large collections.
- Vision encoders for multimodal LLMs, and conditioning for image generators.
Zero-shot accuracy depends heavily on the prompt wording and on whether the domain was well covered in the web training data: it is often poor on specialised medical images. Measure it on your data, and fall back to a linear probe on CLIP embeddings with a few labelled examples, which is usually much better.
Code example: patchify and zero-shot with embeddings
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: zero-shot classification with CLIP
Hugging Face transformers.
from transformers import CLIPModel, CLIPProcessor
from PIL import Image
import torch
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32"); proc = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
labels = ["a referral letter", "a pathology report", "a prescription", "a handwritten note"]
inputs = proc(text=[f"a scanned image of {l}" for l in labels], images=Image.open("scan.png"), return_tensors="pt", padding=True)
with torch.no_grad():
probs = model(**inputs).logits_per_image.softmax(dim=-1)[0]
print({l: round(p.item(), 3) for l, p in zip(labels, probs)})
# better with labels: embed images with model.get_image_features(...) and train a logistic regression on themQuiz
1. A 224 × 224 image with 16 × 16 patches gives how many patch tokens?
2. Compared with CNNs, ViTs…
3. What does a masked autoencoder learn from?
4. How does CLIP classify zero-shot?
5. CLIP zero-shot accuracy on your dermatology photos is poor. Next step?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- An Image is Worth 16x16 Words (Dosovitskiy et al., 2020) ↗ The ViT paper
- CLIP (Radford et al., 2021) ↗ Learning transferable visual models from natural language
- DINOv2 (Oquab et al., 2023) ↗ Self-supervised visual features