Home / Maths for AI / Symbols / Functions

Softmax

Say it: "softmax"

What it means

Turns a list of scores into probabilities that add up to 1: softmax(z)i=ezi∑jezj\mathrm{softmax}(z)_i = \frac{e^{z_i}}{\sum_j e^{z_j}}.

Example

softmax(2,1,0)≈(0.665,0.245,0.090)\mathrm{softmax}(2, 1, 0) \approx (0.665, 0.245, 0.090)

The largest score gets the largest share.

In code

import numpy as np
z = np.array([2.0, 1.0, 0.0])
p = np.exp(z - z.max()) / np.exp(z - z.max()).sum()
print(p.round(3), p.sum())

Output

[0.665 0.245 0.09 ] 0.9999999999999999

Where it appears in machine learning

The output layer of every classifier with several classes, and the attention weights in transformers.

Related symbols

Learn it properly in the free lesson Numerical Stability, with practice questions that check your answers.