Regularization
Penalising complexity so models generalise better.
The idea: add a penalty
Regularisation is any technique that deliberately limits how complex a model can become, in order to make it generalise better. The most common form changes the objective that training minimises. Instead of minimising the loss alone, we minimise the loss plus a penalty on the size of the weights:
The first term rewards fitting the training data well. The second term, , measures how large or complicated the weights are, and penalises that. The number (lambda) controls how strongly the penalty counts relative to the fit.
The value of moves the model along the bias–variance scale. With , there is no penalty at all, and a flexible model is free to overfit. With a very large , the penalty dominates, every weight is pushed almost to zero, and the model becomes too simple and underfits. The best value lies somewhere in between, and since it depends on the data, we find it by trying a range of values and choosing the one with the best cross-validation score.
In terms of the bias–variance trade-off, regularisation knowingly accepts a little extra bias (the weights are no longer free to fit the training data as closely as possible) in exchange for a large reduction in variance (the fitted model no longer swings wildly with every quirk of the training sample). When the model is prone to overfitting, this trade almost always improves performance on new data.
Interactive: this part runs in your browser.
L2 (Ridge), L1 (Lasso), Elastic Net
Three penalties dominate practice, and they behave quite differently.
L2 regularisation, known as Ridge regression for linear models and as weight decay in neural networks, penalises the sum of the squared weights, . Because the penalty grows with the square, large weights are punished heavily while small ones cost very little. The effect is to shrink all the weights smoothly towards zero, though rarely exactly to zero. When two features are strongly correlated, ridge tends to share the weight between them rather than picking one, which makes the fitted model stable. Ridge also has an exact solution, , and adding guarantees that the matrix can always be inverted. In gradient descent, the penalty adds to the gradient, so every step multiplies the weights by a factor slightly below 1 before the usual update; they "decay", hence the name weight decay. From a Bayesian point of view, ridge is what you get from a Gaussian prior belief that the weights are probably small.
L1 regularisation, called Lasso, penalises the sum of the absolute values of the weights, . This small change has a big consequence: lasso drives many weights to exactly zero, producing a sparse model that uses only a subset of the features. In effect it performs feature selection automatically, which is very useful when you suspect that only a few of many features really matter. Its weakness is with correlated features, where it tends to keep one and drop the others somewhat arbitrarily. Because the absolute value has a sharp corner at zero, lasso is solved with special methods such as coordinate descent. Its Bayesian interpretation is a Laplace prior, which is sharply peaked at zero.
Elastic Net combines the two penalties, , where (called l1_ratio in scikit-learn) sets the mix. It inherits lasso's ability to set weights to zero and ridge's stability with correlated features, tending to keep or drop correlated features together as a group. It is popular for high-dimensional data, such as genetics or text, where there can be more features than examples.
| L2 / Ridge | L1 / Lasso | Elastic Net | |
|---|---|---|---|
| Penalty | a mix of both | ||
| Weights set to exactly 0? | no | yes | yes |
| Correlated features | shares the weight | picks one | keeps the group |
| Bayesian prior | Gaussian | Laplace | a mix |
Why does L1 produce exact zeros when L2 doesn't? Picture a model with just two weights. Regularisation is equivalent to finding the best-fitting weights inside a fixed "budget" region. For L2, the region of weights with below the budget is a circle; for L1, the region with below the budget is a diamond, whose corners lie exactly on the axes. The loss has elliptical contour lines centred on the unregularised solution, and the regularised solution is where the smallest of these ellipses first touches the budget region. A smooth circle is usually touched at a point where both weights are non-zero. A diamond, because of its pointed corners, is very often touched first at a corner, and at a corner one of the weights is exactly zero. The picture below shows the two cases.
Interactive: this part runs in your browser.
Try it: slide λ
A degree-12 polynomial with Ridge. Tiny λ → wiggly overfit with enormous weights. Large λ → flat underfit. Find the sweet spot.
Interactive: this part runs in your browser.
Beyond L1/L2
Penalising the size of the weights is only one way to control complexity, and several other techniques are used just as widely.
In neural networks, dropout randomly switches off a fraction of the neurons during each training step. Because the network can never rely on any particular neuron being present, it is forced to spread what it learns across many neurons, which makes it more robust. Early stopping watches the error on a validation set during training and stops as soon as that error starts to rise, which is the moment the network begins to memorise noise. Data augmentation creates extra, slightly altered copies of the training examples, for instance flipped, cropped or rotated images, so the model sees more variety and finds it harder to memorise individual examples.
Tree-based models are regularised by limiting their structure: capping the maximum depth, requiring a minimum number of examples in each leaf, or growing a full tree and then pruning back branches that don't improve validation performance.
Two practical warnings complete the picture. First, beware of how scikit-learn parameterises some models. In LogisticRegression and in support vector machines, the regularisation strength is controlled by a parameter C that is the inverse of , roughly . So a smaller C means stronger regularisation, the opposite of what many people expect. Second, always standardise your features before using L1 or L2 penalties. The penalty acts on the size of the weights, and a weight's size depends on the units of its feature: a feature measured in millimetres needs a much smaller weight than the same feature in kilometres. Without scaling, features are penalised unfairly according to their units. Finally, the intercept is normally left unpenalised, since shifting all predictions up or down doesn't make a model more complex; scikit-learn handles this automatically.
Ridge: closed form and shrinkage through the SVD
Ridge adds "keep the weights small" to the loss. Mathematically this adds λ to the diagonal of XᵀX, which makes it always invertible. Looking through the SVD, ridge turns down the volume on the directions in the data that have little spread, which are exactly the directions where noise gets amplified the most.
🔤 Symbols in this section
| regularisation strength | |
| identity matrix | |
| SVD of the data | |
| j-th singular value (spread along direction j) | |
| j-th left singular vector | |
| effective number of parameters |
This section derives the ridge solution and then uses the singular value decomposition to see exactly what ridge does to the data, which turns out to be very intuitive.
Ridge regression minimises the squared error plus an L2 penalty, . Its gradient is the least-squares gradient from the linear regression topic plus the gradient of the penalty, . Setting the total to zero:
The only difference from ordinary least squares is the extra . Since has non-negative eigenvalues, adding raises every eigenvalue by , so all of them become at least . The matrix is therefore always invertible, even when features are perfectly correlated or there are more features than examples.
To see what ridge actually does, write the data using its singular value decomposition, , where the singular values measure how spread out the data are along each principal direction . After some algebra, the ridge predictions can be written as
Compare this with ordinary least squares, which gives the same sum but with every factor equal to 1. Ridge multiplies the contribution of each direction by a shrinkage factor , which is always between 0 and 1. For a direction where the data are very spread out (large ), the factor is close to 1 and that direction is barely touched. For a direction where the data hardly vary (small ), the factor is close to 0 and that direction is heavily suppressed. This is exactly the right thing to do, because estimating a slope along a direction with very little spread is like estimating the slope of a line from points that are all bunched together: tiny amounts of noise produce huge swings in the estimate. Ridge quietly turns down the volume on the unreliable directions.
Adding up the shrinkage factors gives a useful measure of the model's complexity, the effective degrees of freedom:
With , every factor is 1 and the model uses all of its parameters freely. As increases, falls smoothly towards zero, so works like a dial that continuously adjusts the number of parameters the model effectively uses.
Finally, the bias–variance trade-off can be seen directly. Under the usual noise assumptions, the average ridge estimate is , which is not the true , so ridge is biased. But its variance is strictly smaller than that of least squares. In 1970, Hoerl and Kennard proved that there is always some positive for which this trade is worthwhile, giving ridge a lower overall mean squared error than ordinary least squares.
Lasso: soft-thresholding, coordinate descent and the Bayesian view
Lasso's penalty |w| has a sharp corner at zero. Optimising one weight at a time shows what that corner does: if a feature can't reduce the error by more than λ, its weight snaps to exactly zero; otherwise it's shrunk by λ. That's how lasso selects features automatically.
🔤 Symbols in this section
| L1 norm: sum of |w_j| | |
| subgradient of |w| (any value in [−1, 1] at 0) | |
| residual ignoring feature j | |
| how much feature j could explain | |
| soft-thresholding operator | |
| L1 budget in the constrained form |
This section explains, mathematically, how the lasso sets weights to exactly zero, and then connects regularisation to probability and to modern deep-learning optimisers.
The lasso minimises . The difficulty is that the absolute value has a sharp corner at zero, so it has no ordinary derivative there. We replace the derivative with a subgradient: away from zero it is the usual slope , which is or , and at zero it is allowed to be any value between −1 and 1, reflecting all the slopes of lines that touch the corner.
Rather than solving for all weights at once, the standard algorithm, coordinate descent, optimises one weight at a time while holding the others fixed, which turns the problem into a simple one-dimensional one.
These three cases combine into a single formula using the soft-thresholding operator :
In words: if a feature can't explain more than of what's left, its weight is set to exactly zero; otherwise its weight is shrunk towards zero by . Cycling through the features with this update, again and again until nothing changes, is what scikit-learn's Lasso does. For comparison, the same calculation for ridge gives , which shrinks the weight but can never reach exactly zero unless is already zero. That is the algebraic version of the diamond-and-circle picture.
There is also an equivalent way to state the lasso problem. Using Lagrange multipliers, minimising the error plus gives the same answer as minimising the error subject to a budget, , for a value of that corresponds to each . That constrained form is the "diamond" in the beginner picture.
From a Bayesian point of view, both penalties are prior beliefs about the weights (see the likelihood topic). A Laplace prior, , has a sharp peak at zero, expressing a belief that many weights are probably exactly zero; its MAP estimate is the lasso. A Gaussian prior has a smooth, rounded peak, expressing a belief that weights are probably small; its MAP estimate is ridge.
Finally, a practical point about deep learning. With plain SGD, adding an L2 penalty to the loss and applying "weight decay" are the same thing. With Adam they are not. Adam divides every gradient by a running estimate of its typical size, and that includes the gradient of the penalty, so weights that receive large gradients end up barely regularised at all. AdamW fixes this by applying the decay directly to the weights, separately from the adaptive step: . For this reason, AdamW is the standard optimiser for training modern networks such as transformers.
Code: ridge closed form, lasso by coordinate descent, and scikit-learn
The lasso implementation is a direct translation of the soft-thresholding formula. Note how it produces exact zeros.
import numpy as np
rng = np.random.default_rng(0)
n, d = 100, 10
X = rng.normal(size=(n, d))
w_true = np.array([3, -2, 1.5, 0, 0, 0, 0, 0, 0, 0.0]) # only 3 features matter
y = X @ w_true + rng.normal(scale=1.0, size=n)
X = (X - X.mean(0)) / X.std(0); y = y - y.mean() # standardise, centre
# Ridge: closed form
lam = 10.0
w_ridge = np.linalg.solve(X.T @ X + lam * np.eye(d), X.T @ y)
# Lasso: cyclic coordinate descent with soft-thresholding
def soft(rho, lam):
return np.sign(rho) * np.maximum(np.abs(rho) - lam, 0.0)
def lasso_cd(X, y, lam, iters=200):
w = np.zeros(X.shape[1])
col_sq = (X ** 2).sum(0)
for _ in range(iters):
for j in range(X.shape[1]):
r_j = y - X @ w + X[:, j] * w[j] # residual without feature j
w[j] = soft(X[:, j] @ r_j, lam) / col_sq[j]
return w
w_lasso = lasso_cd(X, y, lam=30.0)
np.set_printoptions(precision=2, suppress=True)
print("ridge:", w_ridge)
print("lasso:", w_lasso) # irrelevant features are exactly 0from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV, lasso_path
ridge = RidgeCV(alphas=np.logspace(-3, 3, 50)).fit(X, y)
lasso = LassoCV(cv=5, random_state=0).fit(X, y)
enet = ElasticNetCV(l1_ratio=[0.2, 0.5, 0.8], cv=5, random_state=0).fit(X, y)
print("best ridge alpha:", ridge.alpha_)
print("best lasso alpha:", round(lasso.alpha_, 4), "non-zero:", np.sum(lasso.coef_ != 0))
print("elastic net l1_ratio:", enet.l1_ratio_)
# Regularisation path: how each coefficient enters as alpha decreases
alphas, coefs, _ = lasso_path(X, y, n_alphas=20)
print(coefs.shape) # (n_features, n_alphas)
# Note: sklearn's Lasso minimises (1/2n)||y - Xw||^2 + alpha*||w||_1,
# so alpha = lam / n compared with the scratch version above.Quiz
1. Which penalty can set weights to exactly zero?
2. Increasing λ a lot makes both train and test error rise. That's…
3. In scikit-learn LogisticRegression, stronger regularization means…
4. Soft-thresholding sets wⱼ to exactly zero when…
5. Ridge shrinks most strongly along directions with…
6. Why use AdamW instead of adding L2 to Adam's gradient?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- scikit-learn: Linear models ↗ Official guide
- An Introduction to Statistical Learning (free book) ↗ James, Witten, Hastie, Tibshirani (Python edition available)