Maths for AI
Topic 15 · Information & computation · ⏱ about 3 h

Information Theory: Entropy, Cross-Entropy & KL Divergence

Measure surprise and uncertainty in bits, see why cross-entropy is the loss function of classifiers and language models, and compare distributions with KL divergence.

Show:
In this topic
Beginner

Surprise and entropy

An unlikely event is more surprising (more informative) than a likely one. Information theory measures the surprise of an outcome with probability pp as −log⁡2p-\log_2 p bits: a fair coin flip gives 1 bit, an event with probability 1/8 gives 3 bits, a certain event gives 0.

Entropy is the average surprise of a distribution, a measure of its uncertainty:

H(p)=−∑ipilog⁡piH(p) = -\sum_i p_i \log p_i

It is largest when all outcomes are equally likely and zero when one outcome is certain. With the natural log the unit is nats instead of bits; machine-learning libraries use natural logs.

Try it

Try it: entropy of a distribution

Drag the sliders to share probability between four outcomes. Watch the entropy rise as the distribution spreads out, and fall to zero as one outcome becomes certain.

Interactive: this part runs in your browser.

Beginner

Cross-entropy: the loss of classifiers

Cross-entropy measures the average surprise when the true distribution is pp but you predicted qq:

H(p,q)=−∑ipilog⁡qiH(p, q) = -\sum_i p_i \log q_i

For classification, the true distribution puts all its probability on the correct class yy, so cross-entropy becomes simply −log⁡qy-\log q_y: minus the log of the probability the model gave to the right answer. Predicting 0.9 for the right class costs 0.105; predicting 0.1 costs 2.3; predicting 0.001 costs 6.9. Confident wrong answers are punished hard.

This is the loss behind logistic regression (log loss), neural network classifiers and every language model, which is trained to minimise the cross-entropy of the next token. A language model's perplexity is just ecross-entropye^{\text{cross-entropy}}. Minimising cross-entropy is also the same as maximising likelihood (see MLE).

Try it

Try it: the cost of a prediction

Set the model's predicted probabilities for three classes and choose the true class. See the loss, and how fast it grows for confident mistakes.

Interactive: this part runs in your browser.

Beginner

KL divergence

The Kullback–Leibler divergence is the extra surprise from using qq when the truth is pp:

DKL(p ∥ q)=∑ipilog⁡piqi=H(p,q)−H(p)D_{KL}(p\,\|\,q) = \sum_i p_i \log \frac{p_i}{q_i} = H(p, q) - H(p)

It is zero only when p=qp = q, always non-negative, and not symmetric: DKL(p∥q)≠DKL(q∥p)D_{KL}(p\|q) \ne D_{KL}(q\|p), so it is not a true distance. You will meet it in variational autoencoders, in knowledge distillation (a small model learning a large model's probabilities), in monitoring data drift, and in fine-tuning LLMs with RLHF or DPO, where a KL penalty keeps the tuned model close to the original.

A related idea, mutual information, measures how much knowing one variable reduces uncertainty about another; decision trees choose splits by the closely related information gain.

Beginner

Code: entropy, cross-entropy and KL in NumPy

Compute them by hand and check against scikit-learn and SciPy.

Python
import numpy as np
from sklearn.metrics import log_loss
from scipy.stats import entropy as sp_entropy

def entropy(p):
    p = np.asarray(p, float)
    p = p[p > 0]                       # 0 * log 0 is taken as 0
    return -np.sum(p * np.log2(p))

print("fair coin:", entropy([0.5, 0.5]), "bits")
print("biased coin:", round(entropy([0.9, 0.1]), 3), "bits")
print("fair die:", round(entropy([1/6] * 6), 3), "bits")

# cross-entropy loss for 4 predictions over 3 classes
y_true = np.array([0, 2, 1, 2])
probs = np.array([[0.7, 0.2, 0.1],
                  [0.1, 0.3, 0.6],
                  [0.2, 0.5, 0.3],
                  [0.8, 0.1, 0.1]])     # the last one is confidently wrong
per_example = -np.log(probs[np.arange(len(y_true)), y_true])
print("loss per example:", per_example.round(3))
print("mean cross-entropy:", per_example.mean().round(4), "sklearn:", round(log_loss(y_true, probs), 4))

p = np.array([0.5, 0.3, 0.2]); q = np.array([0.4, 0.4, 0.2])
kl_pq = np.sum(p * np.log(p / q)); kl_qp = np.sum(q * np.log(q / p))
print("KL(p||q) =", round(kl_pq, 4), " KL(q||p) =", round(kl_qp, 4), " scipy:", round(sp_entropy(p, q), 4))
Open in playground ↗
Beginner

Practice: check yourself

Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.

Question 1.
What is the entropy, in bits, of a fair 8-sided die?
Question 2.
A classifier gives the correct class probability 0.8. What is the cross-entropy loss (natural log)? Give 3 decimal places.
Question 3.
What is the entropy in bits of a coin with P(heads) = 0.5?
Check yourself

Quiz

1. Which distribution over 4 outcomes has the highest entropy?

2. A classifier gives the true class probability 0.01. Its cross-entropy loss for that example is about…

3. Which statement about KL divergence is true?

4. Minimising cross-entropy on training labels is equivalent to…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready