Transformers from Scratch
Scaled dot-product attention, masks, multi-head attention, positional encodings, the transformer block, encoder vs decoder models, and the cost of attention.
Attention: every token looks at every other
An RNN passes information along the sequence one step at a time. Attention lets every position look directly at every other position in a single step. Each token produces three vectors with learned linear layers: a query q ("what am I looking for?"), a key k ("what do I contain?") and a value v ("what will I pass on?"). A token's output is a weighted average of all values, weighted by how well its query matches each key:
with Q, K, V of shape (T, d_k) for a sequence of T tokens. is the (T ร T) table of match scores; softmax turns each row into weights that sum to 1. Dividing by keeps the scores from growing with the vector size: without it, softmax saturates into a near one-hot, and gradients vanish.
M is a mask added before the softmax: where attention is forbidden, so those weights become exactly 0. A causal mask hides future tokens (for models that generate text left to right); a padding mask hides padding.
Watch it: Data flowing through a transformer block
A short narrated animation. Press โถ Play, or step through it with โฎ โญ. Tick ๐ narrate to hear each step.
Interactive: this part runs in your browser.
Try it: an attention matrix
Each row is one token's query; each column a key. Toggle the causal mask and the โd scaling, and compare two heads that have learned different relationships.
Interactive: this part runs in your browser.
Multi-head attention and positions
One attention pattern can only express one kind of relationship at a time. Multi-head attention runs h attention operations in parallel, each on its own d/h-dimensional slice (its own learned Q, K, V projections), then concatenates the results and mixes them with an output projection. Different heads learn different relationships: the previous token, the subject of the verb, a matching bracket.
Attention itself ignores order: shuffle the tokens and each output just moves with its token. So position information is added. The original transformer adds fixed sinusoidal positional encodings to the embeddings: , . Many models learn a position embedding instead, and most recent language models use rotary embeddings (RoPE), which rotate q and k by position-dependent angles so the score depends on the relative distance.
The transformer block
A transformer is a stack of identical blocks. The modern ("pre-norm") block, for an input X of shape (B, T, d):
Two residual connections carry the signal (and gradients) through every block; LayerNorm keeps it in range. Attention is the only place where tokens talk to each other; the MLP, which holds about two thirds of the parameters, works on each token separately. Input tokens are first turned into vectors by an embedding table; a final linear layer maps each vector to logits over the vocabulary or to class scores.
Encoders, decoders and how they are trained
| Type | Attention | Pre-training objective | Examples ยท good at |
|---|---|---|---|
| Encoder-only | Bidirectional (sees all tokens) | Masked language modelling: predict hidden words | BERT, RoBERTa ยท classification, search embeddings, tagging |
| Decoder-only | Causal (left to right) | Next-token prediction | GPT, Llama, Claude ยท generation, chat, almost everything at scale |
| Encoder-decoder | Encoder bidirectional; decoder causal + cross-attention to the encoder | Span corruption / translation | T5, original transformer ยท translation, summarisation |
The same block also works on images split into patches (Vision Transformers) and on audio. The LLM topic covers how decoder models are instruction-tuned and used; the NLP track goes further.
The cost of attention
The score matrix is T ร T per head, so attention's compute and memory grow with the square of the sequence length: doubling the context makes attention four times more expensive. Three engineering ideas matter in practice:
- FlashAttention computes exact attention in tiles without storing the full T ร T matrix in GPU memory: much faster, and the default in PyTorch's
scaled_dot_product_attention. - When generating, a decoder reuses the keys and values of earlier tokens: the KV cache. It makes each new token cheap but uses a lot of memory for long contexts.
- Sparse, sliding-window and linear attention variants trade some quality for lower cost on very long inputs.
Code example: attention with masks and heads
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code example: a whole transformer block
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: a transformer block in PyTorch
Using PyTorch's fused attention (FlashAttention when available).
import torch, torch.nn as nn, torch.nn.functional as F
class Block(nn.Module):
def __init__(self, d=256, h=8, p=0.1):
super().__init__()
self.h, self.ln1, self.ln2 = h, nn.LayerNorm(d), nn.LayerNorm(d)
self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d), nn.Dropout(p))
def forward(self, x, causal=True): # x: (B, T, d)
B, T, d = x.shape
q, k, v = self.qkv(self.ln1(x)).split(d, dim=-1)
q, k, v = (t.view(B, T, self.h, d // self.h).transpose(1, 2) for t in (q, k, v)) # (B, h, T, d/h)
a = F.scaled_dot_product_attention(q, k, v, is_causal=causal)
x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
return x + self.mlp(self.ln2(x))
print(Block()(torch.randn(2, 16, 256)).shape) # torch.Size([2, 16, 256])Quiz
1. Why divide attention scores by โd_k?
2. What does a causal mask do?
3. Why do transformers need positional information?
4. In a transformer block, where do tokens exchange information?
5. Which model type is BERT?
6. Doubling the context length makes standard attentionโฆ
๐ Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Attention Is All You Need (Vaswani et al., 2017) โ The original transformer paper
- The Illustrated Transformer (Alammar) โ A visual walk-through
- Neural Networks: Zero to Hero (Karpathy) โ Video course building backprop, MLPs and GPT from scratch