Home / Maths for AI / Symbols / Machine learning notation

Lambda (regularisation strength)

Say it: "lambda"

What it means

In ML, λ\lambda usually weights a penalty term: L+λ∥w∥2L + \lambda \|\mathbf{w}\|^2. (In linear algebra the same letter names an eigenvalue.)

Example

J(w)=MSE+λ∥w∥22J(\mathbf{w}) = \text{MSE} + \lambda \|\mathbf{w}\|_2^2

Bigger lambda, smaller weights.

In code

from sklearn.linear_model import Ridge
import numpy as np
X = np.random.default_rng(0).normal(size=(50, 3)); y = X @ [3.0, -2.0, 0.5]
for lam in [0.1, 100]:
    print(lam, Ridge(alpha=lam).fit(X, y).coef_.round(2))

Output

0.1 [ 2.99 -1.99  0.5 ]
100 [ 1.11 -0.45  0.28]

Where it appears in machine learning

Ridge, lasso and weight decay all control overfitting with a λ\lambda (called alpha in scikit-learn).

Related symbols

Learn it properly in the free lesson Optimisation, with practice questions that check your answers.