Natural Language Processing
Topic 2 · NLP · ⏱ about 5 h

Text as Data: Normalisation & Tokenisation

From characters to tokens: normalisation, word and subword tokenisers (BPE, WordPiece, SentencePiece), vocabularies and special tokens, and why tokenisation affects cost, languages and bugs.

Show:
In this topic
Beginner

Cleaning text without destroying it

Raw text is messy: casing, accents, emojis, typos, HTML, abbreviations ("pt c/o SOB" in clinical notes), mixed languages. Classic pipelines normalised heavily (lowercase, strip punctuation, remove stop words, stem "running" → "run"). With pretrained transformers, do much less: their tokenizer expects text like the text it was trained on, and casing, punctuation and negation ("no chest pain") carry meaning. Do normalise encoding (Unicode NFC), whitespace and obvious markup; do de-identify personal information where required; keep everything else.

Beginner

Watch it: Byte-pair encoding: merging the most frequent pair

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

Tokenisation: words, characters, or subwords

Models work on integer token ids. How text is split matters:

  • Words: intuitive, but the vocabulary explodes, and unseen words ("atorvastatin", typos) become an unknown token [UNK].
  • Characters: no unknowns, but sequences become very long.
  • Subwords (the modern standard): frequent words are single tokens, rare words are split into frequent pieces: "atorvastatin" → "ator", "vast", "atin". No unknown words, manageable length.

Byte-pair encoding (BPE) learns subwords from a corpus: start with characters, repeatedly merge the most frequent adjacent pair into a new symbol, until the vocabulary reaches a target size (GPT models use 50k–200k). Byte-level BPE starts from bytes, so any text can be encoded. WordPiece (BERT) is similar with a likelihood-based merge rule; SentencePiece (T5, Llama) treats the text as a raw stream including spaces (shown as ▁), which suits languages without spaces.

Try it

Try it: watch BPE learn

Train byte-pair encoding on a small corpus of patient messages, merge by merge, and see how new words get tokenised.

Interactive: this part runs in your browser.

Beginner

Why tokenisation matters in practice

  • Cost and limits: LLM prices and context windows are counted in tokens. English averages about 0.75 words per token; many other languages need far more tokens for the same meaning, so they cost more and fit less.
  • Domain words: medicine names, codes and numbers split into many pieces, which can hurt accuracy; domain-specific tokenizers or vocabulary extensions help.
  • Special tokens: [CLS], [SEP], padding, beginning/end of sequence, chat-template markers. Using a model with the wrong tokenizer or chat template is a silent, serious bug.
  • Alignment with labels: for NER, word labels must be mapped onto subword tokens (usually label the first piece and ignore the rest in the loss).
  • Numbers and spelling: models see tokens, not characters, which is why counting letters or doing arithmetic on long numbers is hard for them.
Try it

Code example: byte-pair encoding from scratch

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: tokenizers in practice

Inspect how a real tokenizer splits text, and how word labels align to subwords.

Python · transformers
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
enc = tok("Repeat script for atorvastatin please", return_offsets_mapping=True)
print(tok.convert_ids_to_tokens(enc["input_ids"]))
# ['[CLS]', 'repeat', 'script', 'for', 'at', '##or', '##vas', '##tat', '##in', 'please', '[SEP]']
words = ["Repeat", "script", "for", "atorvastatin", "please"]
enc = tok(words, is_split_into_words=True)
print(enc.word_ids())     # [None, 0, 1, 2, 3, 3, 3, 3, 3, 4, None]: label only the first piece of each word
gpt = AutoTokenizer.from_pretrained("gpt2")
for s in ["Hello, how are you?", "नमस्ते, तपाईंलाई कस्तो छ?"]:
    print(len(s.split()), "words →", len(gpt(s)["input_ids"]), "tokens")
This code needs Hugging Face transformers, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. Why are subword tokenisers the standard?

2. How does BPE build its vocabulary?

3. Lowercasing and removing stop words before fine-tuning BERT is…

4. For NER with subword tokens, word labels are usually…

5. The same message in Nepali uses 3× more tokens than in English. Consequences?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready