Information Theory: Entropy, Cross-Entropy & KL Divergence
Measure surprise and uncertainty in bits, see why cross-entropy is the loss function of classifiers and language models, and compare distributions with KL divergence.
Surprise and entropy
An unlikely event is more surprising (more informative) than a likely one. Information theory measures the surprise of an outcome with probability as bits: a fair coin flip gives 1 bit, an event with probability 1/8 gives 3 bits, a certain event gives 0.
Entropy is the average surprise of a distribution, a measure of its uncertainty:
It is largest when all outcomes are equally likely and zero when one outcome is certain. With the natural log the unit is nats instead of bits; machine-learning libraries use natural logs.
Try it: entropy of a distribution
Drag the sliders to share probability between four outcomes. Watch the entropy rise as the distribution spreads out, and fall to zero as one outcome becomes certain.
Interactive: this part runs in your browser.
Cross-entropy: the loss of classifiers
Cross-entropy measures the average surprise when the true distribution is but you predicted :
For classification, the true distribution puts all its probability on the correct class , so cross-entropy becomes simply : minus the log of the probability the model gave to the right answer. Predicting 0.9 for the right class costs 0.105; predicting 0.1 costs 2.3; predicting 0.001 costs 6.9. Confident wrong answers are punished hard.
This is the loss behind logistic regression (log loss), neural network classifiers and every language model, which is trained to minimise the cross-entropy of the next token. A language model's perplexity is just . Minimising cross-entropy is also the same as maximising likelihood (see MLE).
Try it: the cost of a prediction
Set the model's predicted probabilities for three classes and choose the true class. See the loss, and how fast it grows for confident mistakes.
Interactive: this part runs in your browser.
KL divergence
The Kullback–Leibler divergence is the extra surprise from using when the truth is :
It is zero only when , always non-negative, and not symmetric: , so it is not a true distance. You will meet it in variational autoencoders, in knowledge distillation (a small model learning a large model's probabilities), in monitoring data drift, and in fine-tuning LLMs with RLHF or DPO, where a KL penalty keeps the tuned model close to the original.
A related idea, mutual information, measures how much knowing one variable reduces uncertainty about another; decision trees choose splits by the closely related information gain.
Code: entropy, cross-entropy and KL in NumPy
Compute them by hand and check against scikit-learn and SciPy.
import numpy as np
from sklearn.metrics import log_loss
from scipy.stats import entropy as sp_entropy
def entropy(p):
p = np.asarray(p, float)
p = p[p > 0] # 0 * log 0 is taken as 0
return -np.sum(p * np.log2(p))
print("fair coin:", entropy([0.5, 0.5]), "bits")
print("biased coin:", round(entropy([0.9, 0.1]), 3), "bits")
print("fair die:", round(entropy([1/6] * 6), 3), "bits")
# cross-entropy loss for 4 predictions over 3 classes
y_true = np.array([0, 2, 1, 2])
probs = np.array([[0.7, 0.2, 0.1],
[0.1, 0.3, 0.6],
[0.2, 0.5, 0.3],
[0.8, 0.1, 0.1]]) # the last one is confidently wrong
per_example = -np.log(probs[np.arange(len(y_true)), y_true])
print("loss per example:", per_example.round(3))
print("mean cross-entropy:", per_example.mean().round(4), "sklearn:", round(log_loss(y_true, probs), 4))
p = np.array([0.5, 0.3, 0.2]); q = np.array([0.4, 0.4, 0.2])
kl_pq = np.sum(p * np.log(p / q)); kl_qp = np.sum(q * np.log(q / p))
print("KL(p||q) =", round(kl_pq, 4), " KL(q||p) =", round(kl_qp, 4), " scipy:", round(sp_entropy(p, q), 4))Practice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. Which distribution over 4 outcomes has the highest entropy?
2. A classifier gives the true class probability 0.01. Its cross-entropy loss for that example is about…
3. Which statement about KL divergence is true?
4. Minimising cross-entropy on training labels is equivalent to…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Visual Information Theory (Chris Olah) ↗ a famous visual explanation of entropy, cross-entropy and KL
- Information Theory, Inference, and Learning Algorithms (David MacKay) ↗ free textbook; chapters 1–2 cover entropy and information