Cross-entropy
Say it: "the cross-entropy of p and q"
What it means
: the average surprise when the truth is and you predicted . For one labelled example it is .
Example
Confident wrong answers cost a lot.
In code
import numpy as np for q in [0.99, 0.8, 0.1, 0.001]: print(q, round(-np.log(q), 3))
Output
0.99 0.01 0.8 0.223 0.1 2.303 0.001 6.908
Where it appears in machine learning
The standard loss for classification and for training language models.
Related symbols
Learn it properly in the free lesson Information Theory, with practice questions that check your answers.