Text as Data: Normalisation & Tokenisation
From characters to tokens: normalisation, word and subword tokenisers (BPE, WordPiece, SentencePiece), vocabularies and special tokens, and why tokenisation affects cost, languages and bugs.
Cleaning text without destroying it
Raw text is messy: casing, accents, emojis, typos, HTML, abbreviations ("pt c/o SOB" in clinical notes), mixed languages. Classic pipelines normalised heavily (lowercase, strip punctuation, remove stop words, stem "running" → "run"). With pretrained transformers, do much less: their tokenizer expects text like the text it was trained on, and casing, punctuation and negation ("no chest pain") carry meaning. Do normalise encoding (Unicode NFC), whitespace and obvious markup; do de-identify personal information where required; keep everything else.
Watch it: Byte-pair encoding: merging the most frequent pair
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Tokenisation: words, characters, or subwords
Models work on integer token ids. How text is split matters:
- Words: intuitive, but the vocabulary explodes, and unseen words ("atorvastatin", typos) become an unknown token
[UNK]. - Characters: no unknowns, but sequences become very long.
- Subwords (the modern standard): frequent words are single tokens, rare words are split into frequent pieces: "atorvastatin" → "ator", "vast", "atin". No unknown words, manageable length.
Byte-pair encoding (BPE) learns subwords from a corpus: start with characters, repeatedly merge the most frequent adjacent pair into a new symbol, until the vocabulary reaches a target size (GPT models use 50k–200k). Byte-level BPE starts from bytes, so any text can be encoded. WordPiece (BERT) is similar with a likelihood-based merge rule; SentencePiece (T5, Llama) treats the text as a raw stream including spaces (shown as ▁), which suits languages without spaces.
Try it: watch BPE learn
Train byte-pair encoding on a small corpus of patient messages, merge by merge, and see how new words get tokenised.
Interactive: this part runs in your browser.
Why tokenisation matters in practice
- Cost and limits: LLM prices and context windows are counted in tokens. English averages about 0.75 words per token; many other languages need far more tokens for the same meaning, so they cost more and fit less.
- Domain words: medicine names, codes and numbers split into many pieces, which can hurt accuracy; domain-specific tokenizers or vocabulary extensions help.
- Special tokens:
[CLS],[SEP], padding, beginning/end of sequence, chat-template markers. Using a model with the wrong tokenizer or chat template is a silent, serious bug. - Alignment with labels: for NER, word labels must be mapped onto subword tokens (usually label the first piece and ignore the rest in the loss).
- Numbers and spelling: models see tokens, not characters, which is why counting letters or doing arithmetic on long numbers is hard for them.
Code example: byte-pair encoding from scratch
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: tokenizers in practice
Inspect how a real tokenizer splits text, and how word labels align to subwords.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
enc = tok("Repeat script for atorvastatin please", return_offsets_mapping=True)
print(tok.convert_ids_to_tokens(enc["input_ids"]))
# ['[CLS]', 'repeat', 'script', 'for', 'at', '##or', '##vas', '##tat', '##in', 'please', '[SEP]']
words = ["Repeat", "script", "for", "atorvastatin", "please"]
enc = tok(words, is_split_into_words=True)
print(enc.word_ids()) # [None, 0, 1, 2, 3, 3, 3, 3, 3, 4, None]: label only the first piece of each word
gpt = AutoTokenizer.from_pretrained("gpt2")
for s in ["Hello, how are you?", "नमस्ते, तपाईंलाई कस्तो छ?"]:
print(len(s.split()), "words →", len(gpt(s)["input_ids"]), "tokens")Quiz
1. Why are subword tokenisers the standard?
2. How does BPE build its vocabulary?
3. Lowercasing and removing stop words before fine-tuning BERT is…
4. For NER with subword tokens, word labels are usually…
5. The same message in Nepali uses 3× more tokens than in English. Consequences?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Hugging Face tokenizers: summary ↗ BPE, WordPiece and SentencePiece explained
- Speech and Language Processing (Jurafsky & Martin, 3rd ed. draft) ↗ The standard NLP textbook, free online