Text Classification & Sequence Labelling
Baselines first (TF-IDF + linear models), fine-tuning encoders, imbalanced and multi-label text, named-entity recognition with BIO tags and span-level F1, and when to use an LLM instead.
Always start with a strong baseline
TF-IDF + logistic regression is fast, cheap, interpretable and often surprisingly strong. TF-IDF represents a text by its words (and word pairs) weighted by how characteristic they are: frequent in this text, rare across the corpus. A linear model on top shows exactly which words drive each class. Train it in minutes, and only then ask whether a transformer is worth its cost.
Fine-tuning a pretrained encoder (DistilBERT, RoBERTa, DeBERTa, or a domain model such as a clinical BERT) adds a classification head on the [CLS] vector and trains the whole model with a small learning rate (2e-5 to 5e-5) for a few epochs. It handles negation, word order, typos and paraphrases far better, and usually wins when you have a few thousand labelled examples. LLMs with few-shot prompts need no training and are a quick option when labels are scarce, but are slower, costlier, can drift between model versions, and must be evaluated the same way.
Metrics, imbalance and labels
- Use macro-F1 (average F1 over classes) and per-class recall: accuracy hides failure on rare classes. For a safety-critical class ("urgent"), fix a recall target and report precision at it.
- Imbalance: class weights, oversampling, and threshold tuning on validation data.
- Multi-label: one sigmoid per label, a threshold per label.
- Labels: write labelling guidelines with examples; measure agreement (Cohen's κ); review disagreements and confusing pairs. Noisy labels cap any model's accuracy.
- Leakage: near-duplicate messages (templates, copy-paste) across train and test inflate scores; deduplicate first.
Named-entity recognition with BIO tags
NER labels each token: is it part of a person's name, a medicine, a date? Entities can span several tokens, so tags use the BIO scheme: B-MED begins an entity, I-MED continues it, O is outside any entity.
A token classifier (an encoder with a linear layer per token) predicts tags; a decoding step turns tags into spans, fixing invalid sequences (an I- after O is treated as a new entity). NER is evaluated at the span level: a predicted entity counts only if its boundaries and type match exactly (seqeval's F1). Token accuracy is misleading because most tokens are O.
NER is central to de-identification of clinical text (find and redact names, addresses, dates), and to extracting structured data (medicines and doses) from notes.
Try it: what does the baseline rely on?
A TF-IDF + logistic regression message router. Type a message and see the predicted category and the words that pushed it there; try negation and paraphrases.
Interactive: this part runs in your browser.
Code example: a message router baseline
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code example: BIO tags to spans, and span-level F1
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: fine-tune a classifier with Hugging Face
The core of notebook 11.
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
import numpy as np, evaluate
name = "distilbert-base-uncased"; tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=5)
ds = ds.map(lambda b: tok(b["text"], truncation=True, max_length=128), batched=True) # a datasets.DatasetDict
f1 = evaluate.load("f1")
def metrics(p): return f1.compute(predictions=np.argmax(p.predictions, -1), references=p.label_ids, average="macro")
args = TrainingArguments("out", learning_rate=3e-5, num_train_epochs=3, per_device_train_batch_size=32,
warmup_ratio=0.1, weight_decay=0.01, eval_strategy="epoch", save_strategy="epoch",
load_best_model_at_end=True, metric_for_best_model="f1")
Trainer(model=model, args=args, train_dataset=ds["train"], eval_dataset=ds["validation"], tokenizer=tok, compute_metrics=metrics).train()Quiz
1. Why train TF-IDF + logistic regression first?
2. Tags B-MED I-MED O B-PER describe…
3. A NER model predicts "metformin" as MED but misses "500mg" (true span: "metformin 500mg"). At the span level this counts as…
4. Your router has 94% accuracy but misses 30% of urgent messages. Which metric should have warned you?
5. Fine-tuning learning rate for BERT-style encoders is typically…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Hugging Face LLM / NLP Course ↗ Free course on transformers, tokenizers and fine-tuning
- seqeval ↗ Span-level evaluation for sequence labelling
- Speech and Language Processing (Jurafsky & Martin, 3rd ed. draft) ↗ The standard NLP textbook, free online