Machine Learning
Topic 12 · Model quality · ⏱ about 4 h

Regularization

Penalising complexity so models generalise better.

Show:
In this topic
Beginner

The idea: add a penalty

Regularisation is any technique that deliberately limits how complex a model can become, in order to make it generalise better. The most common form changes the objective that training minimises. Instead of minimising the loss alone, we minimise the loss plus a penalty on the size of the weights:

best λoverfitunderfitregularisation strength λ (log scale) →trainvalidation
As λ grows, training error rises steadily, while validation error first falls (less overfitting) and then rises (underfitting).
J(w)=Loss(w)+λ⋅Ω(w). J(w) = \text{Loss}(w) + \lambda\cdot\Omega(w).

The first term rewards fitting the training data well. The second term, Ω(w)\Omega(w), measures how large or complicated the weights are, and penalises that. The number λ\lambda (lambda) controls how strongly the penalty counts relative to the fit.

The value of λ\lambda moves the model along the bias–variance scale. With λ=0\lambda = 0, there is no penalty at all, and a flexible model is free to overfit. With a very large λ\lambda, the penalty dominates, every weight is pushed almost to zero, and the model becomes too simple and underfits. The best value lies somewhere in between, and since it depends on the data, we find it by trying a range of values and choosing the one with the best cross-validation score.

In terms of the bias–variance trade-off, regularisation knowingly accepts a little extra bias (the weights are no longer free to fit the training data as closely as possible) in exchange for a large reduction in variance (the fitted model no longer swings wildly with every quirk of the training sample). When the model is prone to overfitting, this trade almost always improves performance on new data.

Interactive: this part runs in your browser.

Breakdown. The regularised loss split into its two parts, data fit and penalty, as λ changes.
Beginner

L2 (Ridge), L1 (Lasso), Elastic Net

Three penalties dominate practice, and they behave quite differently.

L2 regularisation, known as Ridge regression for linear models and as weight decay in neural networks, penalises the sum of the squared weights, λ∑jwj2\lambda\sum_j w_j^2. Because the penalty grows with the square, large weights are punished heavily while small ones cost very little. The effect is to shrink all the weights smoothly towards zero, though rarely exactly to zero. When two features are strongly correlated, ridge tends to share the weight between them rather than picking one, which makes the fitted model stable. Ridge also has an exact solution, w^=(X⊤X+λI)−1X⊤y\hat w = (X^\top X + \lambda I)^{-1}X^\top y, and adding λI\lambda I guarantees that the matrix can always be inverted. In gradient descent, the penalty adds 2λw2\lambda w to the gradient, so every step multiplies the weights by a factor slightly below 1 before the usual update; they "decay", hence the name weight decay. From a Bayesian point of view, ridge is what you get from a Gaussian prior belief that the weights are probably small.

L1 regularisation, called Lasso, penalises the sum of the absolute values of the weights, λ∑j∣wj∣\lambda\sum_j |w_j|. This small change has a big consequence: lasso drives many weights to exactly zero, producing a sparse model that uses only a subset of the features. In effect it performs feature selection automatically, which is very useful when you suspect that only a few of many features really matter. Its weakness is with correlated features, where it tends to keep one and drop the others somewhat arbitrarily. Because the absolute value has a sharp corner at zero, lasso is solved with special methods such as coordinate descent. Its Bayesian interpretation is a Laplace prior, which is sharply peaked at zero.

Elastic Net combines the two penalties, λ[α∑j∣wj∣+(1−α)∑jwj2]\lambda\big[\alpha\sum_j|w_j| + (1-\alpha)\sum_j w_j^2\big], where α\alpha (called l1_ratio in scikit-learn) sets the mix. It inherits lasso's ability to set weights to zero and ridge's stability with correlated features, tending to keep or drop correlated features together as a group. It is popular for high-dimensional data, such as genetics or text, where there can be more features than examples.

L2 / RidgeL1 / LassoElastic Net
Penaltyλ∑wj2\lambda\sum w_j^2λ∑∣wj∣\lambda\sum |w_j|a mix of both
Weights set to exactly 0?noyesyes
Correlated featuresshares the weightpicks onekeeps the group
Bayesian priorGaussianLaplacea mix

Why does L1 produce exact zeros when L2 doesn't? Picture a model with just two weights. Regularisation is equivalent to finding the best-fitting weights inside a fixed "budget" region. For L2, the region of weights with w12+w22w_1^2 + w_2^2 below the budget is a circle; for L1, the region with ∣w1∣+∣w2∣|w_1| + |w_2| below the budget is a diamond, whose corners lie exactly on the axes. The loss has elliptical contour lines centred on the unregularised solution, and the regularised solution is where the smallest of these ellipses first touches the budget region. A smooth circle is usually touched at a point where both weights are non-zero. A diamond, because of its pointed corners, is very often touched first at a corner, and at a corner one of the weights is exactly zero. The picture below shows the two cases.

Interactive: this part runs in your browser.

Interactive. Why Lasso gives exact zeros and Ridge does not: the shape of the allowed region.
L1 diamondw₁ = 0 L2 circleboth ≠ 0
Try it

Try it: slide λ

A degree-12 polynomial with Ridge. Tiny λ → wiggly overfit with enormous weights. Large λ → flat underfit. Find the sweet spot.

Interactive: this part runs in your browser.

Beginner

Beyond L1/L2

Penalising the size of the weights is only one way to control complexity, and several other techniques are used just as widely.

In neural networks, dropout randomly switches off a fraction of the neurons during each training step. Because the network can never rely on any particular neuron being present, it is forced to spread what it learns across many neurons, which makes it more robust. Early stopping watches the error on a validation set during training and stops as soon as that error starts to rise, which is the moment the network begins to memorise noise. Data augmentation creates extra, slightly altered copies of the training examples, for instance flipped, cropped or rotated images, so the model sees more variety and finds it harder to memorise individual examples.

Tree-based models are regularised by limiting their structure: capping the maximum depth, requiring a minimum number of examples in each leaf, or growing a full tree and then pruning back branches that don't improve validation performance.

Two practical warnings complete the picture. First, beware of how scikit-learn parameterises some models. In LogisticRegression and in support vector machines, the regularisation strength is controlled by a parameter C that is the inverse of λ\lambda, roughly C=1/λC = 1/\lambda. So a smaller C means stronger regularisation, the opposite of what many people expect. Second, always standardise your features before using L1 or L2 penalties. The penalty acts on the size of the weights, and a weight's size depends on the units of its feature: a feature measured in millimetres needs a much smaller weight than the same feature in kilometres. Without scaling, features are penalised unfairly according to their units. Finally, the intercept is normally left unpenalised, since shifting all predictions up or down doesn't make a model more complex; scikit-learn handles this automatically.

Intermediate · Maths

Ridge: closed form and shrinkage through the SVD

📐 Maths you'll need: m-matrices · m-gradients · m-eigen
💡 The idea in plain words

Ridge adds "keep the weights small" to the loss. Mathematically this adds λ to the diagonal of XᵀX, which makes it always invertible. Looking through the SVD, ridge turns down the volume on the directions in the data that have little spread, which are exactly the directions where noise gets amplified the most.

🔤 Symbols in this section
λ\lambdaregularisation strength
IIidentity matrix
X=UDV⊤X = UDV^\topSVD of the data
djd_jj-th singular value (spread along direction j)
uju_jj-th left singular vector
df⁡(λ)\operatorname{df}(\lambda)effective number of parameters

This section derives the ridge solution and then uses the singular value decomposition to see exactly what ridge does to the data, which turns out to be very intuitive.

Ridge regression minimises the squared error plus an L2 penalty, ∥y−Xw∥2+λ∥w∥2\lVert y - Xw\rVert^2 + \lambda\lVert w\rVert^2. Its gradient is the least-squares gradient from the linear regression topic plus the gradient of the penalty, 2λw2\lambda w. Setting the total to zero:

−2X⊤(y−Xw)+2λw=0⟹(X⊤X+λI) w=X⊤y -2X^\top(y - Xw) + 2\lambda w = 0 \quad\Longrightarrow\quad (X^\top X + \lambda I)\,w = X^\top y
w^ridge=(X⊤X+λI)−1X⊤y \boxed{\hat w_{\text{ridge}} = (X^\top X + \lambda I)^{-1}X^\top y}

The only difference from ordinary least squares is the extra λI\lambda I. Since X⊤XX^\top X has non-negative eigenvalues, adding λI\lambda I raises every eigenvalue by λ\lambda, so all of them become at least λ>0\lambda > 0. The matrix is therefore always invertible, even when features are perfectly correlated or there are more features than examples.

To see what ridge actually does, write the data using its singular value decomposition, X=UDV⊤X = UDV^\top, where the singular values djd_j measure how spread out the data are along each principal direction uju_j. After some algebra, the ridge predictions can be written as

Xw^ridge=∑juj dj2dj2+λ uj⊤y. X\hat w_{\text{ridge}} = \sum_{j} u_j \, \frac{d_j^2}{d_j^2 + \lambda}\, u_j^\top y.

Compare this with ordinary least squares, which gives the same sum but with every factor equal to 1. Ridge multiplies the contribution of each direction by a shrinkage factor dj2dj2+λ\frac{d_j^2}{d_j^2 + \lambda}, which is always between 0 and 1. For a direction where the data are very spread out (large djd_j), the factor is close to 1 and that direction is barely touched. For a direction where the data hardly vary (small djd_j), the factor is close to 0 and that direction is heavily suppressed. This is exactly the right thing to do, because estimating a slope along a direction with very little spread is like estimating the slope of a line from points that are all bunched together: tiny amounts of noise produce huge swings in the estimate. Ridge quietly turns down the volume on the unreliable directions.

Adding up the shrinkage factors gives a useful measure of the model's complexity, the effective degrees of freedom:

df⁡(λ)=∑jdj2dj2+λ. \operatorname{df}(\lambda) = \sum_j \frac{d_j^2}{d_j^2 + \lambda}.

With λ=0\lambda = 0, every factor is 1 and the model uses all dd of its parameters freely. As λ\lambda increases, df⁡\operatorname{df} falls smoothly towards zero, so λ\lambda works like a dial that continuously adjusts the number of parameters the model effectively uses.

Finally, the bias–variance trade-off can be seen directly. Under the usual noise assumptions, the average ridge estimate is (X⊤X+λI)−1X⊤X w∗(X^\top X + \lambda I)^{-1}X^\top X\,w^*, which is not the true w∗w^*, so ridge is biased. But its variance is strictly smaller than that of least squares. In 1970, Hoerl and Kennard proved that there is always some positive λ\lambda for which this trade is worthwhile, giving ridge a lower overall mean squared error than ordinary least squares.

Key takeaway. ŵ_ridge = (XᵀX + λI)⁻¹Xᵀy shrinks each principal direction by d²/(d² + λ): it adds a little bias to remove a lot of variance.
Advanced

Lasso: soft-thresholding, coordinate descent and the Bayesian view

📐 Maths you'll need: m-calculus · m-optim · m-stats
💡 The idea in plain words

Lasso's penalty |w| has a sharp corner at zero. Optimising one weight at a time shows what that corner does: if a feature can't reduce the error by more than λ, its weight snaps to exactly zero; otherwise it's shrunk by λ. That's how lasso selects features automatically.

🔤 Symbols in this section
∥w∥1\lVert w\rVert_1L1 norm: sum of |w_j|
∂∣w∣\partial|w|subgradient of |w| (any value in [−1, 1] at 0)
rjr_jresidual ignoring feature j
ρj=xj⊤rj\rho_j = x_j^\top r_jhow much feature j could explain
S(ρ,λ)S(\rho, \lambda)soft-thresholding operator
ttL1 budget in the constrained form

This section explains, mathematically, how the lasso sets weights to exactly zero, and then connects regularisation to probability and to modern deep-learning optimisers.

The lasso minimises 12∥y−Xw∥2+λ∥w∥1\frac{1}{2}\lVert y - Xw\rVert^2 + \lambda\lVert w\rVert_1. The difficulty is that the absolute value ∣w∣|w| has a sharp corner at zero, so it has no ordinary derivative there. We replace the derivative with a subgradient: away from zero it is the usual slope sign⁡(w)\operatorname{sign}(w), which is +1+1 or −1-1, and at zero it is allowed to be any value between −1 and 1, reflecting all the slopes of lines that touch the corner.

Rather than solving for all weights at once, the standard algorithm, coordinate descent, optimises one weight at a time while holding the others fixed, which turns the problem into a simple one-dimensional one.

Step 1: isolate one feature. Fix every weight except wjw_j. Let rj=y−∑k≠jxkwkr_j = y - \sum_{k\ne j} x_k w_k be the partial residual, the part of yy not yet explained by the other features. Let ρj=xj⊤rj\rho_j = x_j^\top r_j measure how strongly feature jj is correlated with that leftover part.
Step 2: write the optimality condition. The best wjw_j must make the subgradient contain zero: 0∈−ρj+∥xj∥2wj+λ ∂∣wj∣0 \in -\rho_j + \lVert x_j\rVert^2 w_j + \lambda\,\partial|w_j|.
Step 3: solve the three cases. If ρj>λ\rho_j > \lambda, the solution is positive, wj=(ρj−λ)/∥xj∥2w_j = (\rho_j - \lambda)/\lVert x_j\rVert^2. If ρj<−λ\rho_j < -\lambda, it is negative, wj=(ρj+λ)/∥xj∥2w_j = (\rho_j + \lambda)/\lVert x_j\rVert^2. And if −λ≤ρj≤λ-\lambda \le \rho_j \le \lambda, the only way to satisfy the condition is wj=0w_j = 0, using the freedom of the subgradient at zero.

These three cases combine into a single formula using the soft-thresholding operator SS:

w^j=S(ρj,λ)∥xj∥2,S(ρ,λ)=sign⁡(ρ) max⁡(∣ρ∣−λ, 0). \hat w_j = \frac{S(\rho_j, \lambda)}{\lVert x_j\rVert^2}, \qquad S(\rho, \lambda) = \operatorname{sign}(\rho)\,\max(|\rho| - \lambda,\ 0).

In words: if a feature can't explain more than λ\lambda of what's left, its weight is set to exactly zero; otherwise its weight is shrunk towards zero by λ\lambda. Cycling through the features with this update, again and again until nothing changes, is what scikit-learn's Lasso does. For comparison, the same calculation for ridge gives wj=ρj/(∥xj∥2+λ)w_j = \rho_j/(\lVert x_j\rVert^2 + \lambda), which shrinks the weight but can never reach exactly zero unless ρj\rho_j is already zero. That is the algebraic version of the diamond-and-circle picture.

There is also an equivalent way to state the lasso problem. Using Lagrange multipliers, minimising the error plus λ∥w∥1\lambda\lVert w\rVert_1 gives the same answer as minimising the error subject to a budget, ∥w∥1≤t\lVert w\rVert_1 \le t, for a value of tt that corresponds to each λ\lambda. That constrained form is the "diamond" in the beginner picture.

From a Bayesian point of view, both penalties are prior beliefs about the weights (see the likelihood topic). A Laplace prior, p(wj)∝e−∣wj∣/bp(w_j)\propto e^{-|w_j|/b}, has a sharp peak at zero, expressing a belief that many weights are probably exactly zero; its MAP estimate is the lasso. A Gaussian prior has a smooth, rounded peak, expressing a belief that weights are probably small; its MAP estimate is ridge.

Finally, a practical point about deep learning. With plain SGD, adding an L2 penalty to the loss and applying "weight decay" are the same thing. With Adam they are not. Adam divides every gradient by a running estimate of its typical size, and that includes the gradient of the penalty, so weights that receive large gradients end up barely regularised at all. AdamW fixes this by applying the decay directly to the weights, separately from the adaptive step: w←w−η(m^/(v^+ϵ)+λw)w \leftarrow w - \eta\big(\hat m/(\sqrt{\hat v}+\epsilon) + \lambda w\big). For this reason, AdamW is the standard optimiser for training modern networks such as transformers.

Key takeaway. Lasso = soft-thresholding: weights with |ρⱼ| ≤ λ become exactly 0; it is MAP with a Laplace prior; use AdamW for true weight decay in deep learning.
Code

Code: ridge closed form, lasso by coordinate descent, and scikit-learn

The lasso implementation is a direct translation of the soft-thresholding formula. Note how it produces exact zeros.

Python · NumPy from scratch
import numpy as np

rng = np.random.default_rng(0)
n, d = 100, 10
X = rng.normal(size=(n, d))
w_true = np.array([3, -2, 1.5, 0, 0, 0, 0, 0, 0, 0.0])   # only 3 features matter
y = X @ w_true + rng.normal(scale=1.0, size=n)
X = (X - X.mean(0)) / X.std(0); y = y - y.mean()          # standardise, centre

# Ridge: closed form
lam = 10.0
w_ridge = np.linalg.solve(X.T @ X + lam * np.eye(d), X.T @ y)

# Lasso: cyclic coordinate descent with soft-thresholding
def soft(rho, lam):
    return np.sign(rho) * np.maximum(np.abs(rho) - lam, 0.0)

def lasso_cd(X, y, lam, iters=200):
    w = np.zeros(X.shape[1])
    col_sq = (X ** 2).sum(0)
    for _ in range(iters):
        for j in range(X.shape[1]):
            r_j = y - X @ w + X[:, j] * w[j]       # residual without feature j
            w[j] = soft(X[:, j] @ r_j, lam) / col_sq[j]
    return w

w_lasso = lasso_cd(X, y, lam=30.0)
np.set_printoptions(precision=2, suppress=True)
print("ridge:", w_ridge)
print("lasso:", w_lasso)        # irrelevant features are exactly 0
Open in playground ↗
Python · scikit-learn (tune λ by cross-validation)
from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV, lasso_path

ridge = RidgeCV(alphas=np.logspace(-3, 3, 50)).fit(X, y)
lasso = LassoCV(cv=5, random_state=0).fit(X, y)
enet = ElasticNetCV(l1_ratio=[0.2, 0.5, 0.8], cv=5, random_state=0).fit(X, y)
print("best ridge alpha:", ridge.alpha_)
print("best lasso alpha:", round(lasso.alpha_, 4), "non-zero:", np.sum(lasso.coef_ != 0))
print("elastic net l1_ratio:", enet.l1_ratio_)

# Regularisation path: how each coefficient enters as alpha decreases
alphas, coefs, _ = lasso_path(X, y, n_alphas=20)
print(coefs.shape)   # (n_features, n_alphas)

# Note: sklearn's Lasso minimises (1/2n)||y - Xw||^2 + alpha*||w||_1,
# so alpha = lam / n compared with the scratch version above.
Open in playground ↗
Before it runs, the code block above runs first (silently), because this one continues it.
Check yourself

Quiz

1. Which penalty can set weights to exactly zero?

2. Increasing λ a lot makes both train and test error rise. That's…

3. In scikit-learn LogisticRegression, stronger regularization means…

4. Soft-thresholding sets wⱼ to exactly zero when…

5. Ridge shrinks most strongly along directions with…

6. Why use AdamW instead of adding L2 to Adam's gradient?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready