Transfer Learning, Fine-Tuning & Embeddings
Reuse what a pretrained model already knows: feature extraction vs fine-tuning, choosing a strategy, practical fine-tuning, LoRA and other parameter-efficient methods, and embeddings.
Why pretrained models transfer
Training a large network from scratch needs millions of labelled examples and a lot of compute. Most teams have neither. Fortunately, what a network learns on a big general dataset (ImageNet photos, web text) is largely reusable: early CNN layers detect edges and textures that appear in X-rays and scanned forms too, and a language model's understanding of grammar and meaning helps with clinic messages. Transfer learning starts from such a pretrained model and adapts it to your task with far less data.
Two basic ways to adapt it:
- Feature extraction (linear probing): freeze the pretrained network, use its penultimate-layer outputs as features, and train only a new small head (a linear layer or logistic regression). Fast, needs very little data, hard to overfit.
- Fine-tuning: replace the head and continue training the whole network (or its top layers) on your data with a small learning rate. Usually more accurate when you have enough data, but easier to overfit and more expensive.
Watch it: Transfer learning: freeze, replace the head, then fine-tune
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Choosing a strategy
| Your data is similar to the pretraining data | Your data is quite different (e.g. medical scans vs photos) | |
|---|---|---|
| Few labels (hundreds) | Feature extraction + linear head | Feature extraction from earlier layers; consider a domain-specific pretrained model; augment heavily |
| Many labels (tens of thousands+) | Fine-tune the whole network | Fine-tune the whole network (pretraining still helps optimisation); domain pretraining if available |
Before any of this, check whether a domain-specific pretrained model exists (for example models pretrained on medical images or clinical text): it usually beats a general one. And always compare with a simple baseline trained on the same split.
Try it: which strategy, how many labels?
Set how many labelled examples you have and how close your data is to the pretraining data. The chart shows typical (illustrative) learning curves for each strategy.
Interactive: this part runs in your browser.
Fine-tuning well
- Small learning rate: typically 10 to 100 times smaller than training from scratch (e.g. 1e-5 to 1e-4 with AdamW), with warm-up. Large rates destroy the pretrained features ("catastrophic forgetting").
- Train the new head first, then unfreeze: a randomly initialised head sends large, noisy gradients into the pretrained layers. Train the head for an epoch with the backbone frozen, then unfreeze (gradually, top layers first, if data is small).
- Discriminative learning rates: smaller rates for early layers (general features), larger for later ones.
- Use the same preprocessing as in pretraining: image size, normalisation mean/std, tokenizer. Mismatches quietly cost accuracy.
- BatchNorm with small batches: keep it in eval mode (frozen statistics) during fine-tuning.
- Check the licence and model card of the pretrained model: allowed uses, training data, known limitations.
Parameter-efficient fine-tuning: LoRA
Fully fine-tuning a model with billions of parameters needs memory for all their gradients and optimiser states, and a full copy of the model per task. Parameter-efficient fine-tuning (PEFT) freezes the original weights and trains a small number of new ones.
LoRA (low-rank adaptation) is the most popular. For a frozen weight matrix W (d × d), it learns a low-rank update made of two thin matrices, B (d × r) and A (r × d), with r much smaller than d (often 8 to 64):
B starts at zero, so training begins exactly at the pretrained model. With d = 4,096 and r = 16, that is 131,072 trainable parameters instead of 16.8 million per matrix (0.8%). Adapters for many tasks can share one base model, and after training BA can be merged into W, so inference costs nothing extra. QLoRA additionally stores the frozen base model in 4-bit precision, so large models can be fine-tuned on a single GPU. Other PEFT methods include adapter layers and prompt tuning.
Embeddings: a pretrained model as a feature extractor
The vector a network produces just before its final layer is an embedding: a compact representation in which similar inputs are close together. Embeddings from pretrained models power semantic search and retrieval-augmented generation (see the LLM topic), duplicate detection, clustering, recommendation and few-shot classification with a simple classifier on top.
Compare embeddings with cosine similarity (usually after normalising them to length 1), and use a vector index (FAISS, a vector database) when there are millions. An embedding model should be evaluated on your data: retrieve examples, label whether the neighbours are truly similar, and compare models.
Code example: few labels, pretrained features
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: fine-tune a pretrained ResNet, and LoRA with PEFT
The two most common fine-tuning set-ups.
import torch, torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights
w = ResNet50_Weights.DEFAULT
model = resnet50(weights=w)
preprocess = w.transforms() # SAME preprocessing as pretraining
for p in model.parameters(): p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, 4) # new head: 4 document types
# stage 1: train only model.fc for an epoch or two (lr 1e-3)
# stage 2: unfreeze, smaller lr for the backbone
for p in model.parameters(): p.requires_grad = True
opt = torch.optim.AdamW([{"params": model.fc.parameters(), "lr": 1e-3},
{"params": [p for n, p in model.named_parameters() if not n.startswith("fc")], "lr": 1e-5}],
weight_decay=0.01)from transformers import AutoModelForSequenceClassification
from peft import LoraConfig, get_peft_model
base = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=3)
cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.1, target_modules=["q_lin", "v_lin"])
model = get_peft_model(base, cfg)
model.print_trainable_parameters() # roughly 1% of parameters are trainableQuiz
1. You have 300 labelled X-ray images and a general ImageNet model. A sensible first approach is…
2. Why use a much smaller learning rate when fine-tuning?
3. Why train the new head before unfreezing the backbone?
4. With LoRA rank r = 8 on a 1,024 × 1,024 matrix, how many trainable parameters are added?
5. Your model was pretrained with images normalised by ImageNet mean/std at 224 px. In production you feed 0–255 pixels at 512 px. Expect…
6. How should two embeddings usually be compared?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- PyTorch tutorials ↗ Official tutorials
- LoRA (Hu et al., 2021) ↗ Low-rank adaptation paper
- Hugging Face PEFT docs ↗ LoRA and other parameter-efficient methods