Home / Maths for AI / Symbols / Probability and statistics

Cross-entropy

Say it: "the cross-entropy of p and q"

What it means

H(p,q)=−∑xp(x)log⁡q(x)H(p, q) = -\sum_x p(x) \log q(x): the average surprise when the truth is pp and you predicted qq. For one labelled example it is −log⁡qtrue class-\log q_{\text{true class}}.

Example

qtrue=0.8  ⇒  −ln⁡0.8≈0.223q_{\text{true}} = 0.8 \;\Rightarrow\; -\ln 0.8 \approx 0.223

Confident wrong answers cost a lot.

In code

import numpy as np
for q in [0.99, 0.8, 0.1, 0.001]:
    print(q, round(-np.log(q), 3))

Output

0.99 0.01
0.8 0.223
0.1 2.303
0.001 6.908

Where it appears in machine learning

The standard loss for classification and for training language models.

Related symbols

Learn it properly in the free lesson Information Theory, with practice questions that check your answers.