Linear Regression
The simplest predictive model — and the foundation for many others.
The model
Linear regression is the simplest predictive model, and it is worth understanding thoroughly because so many other methods build on it. It predicts a number by giving each feature a weight, multiplying, and adding the results, plus a constant:
Each weight has a direct interpretation. The weight says how much the prediction changes when feature increases by one unit while all the other features stay the same. For a house-price model, a weight of 0.5 on "floor area in square metres" would mean that each extra square metre adds about $500 to the predicted price, other things being equal. The constant is called the intercept or bias; it is the prediction when every feature equals zero, and it lets the line sit at the right height.
To decide which weights are best, we need a way to measure how wrong a set of weights is. For each training example, the residual is the difference between the true value and the prediction. We square each residual and average them, giving the mean squared error:
Squaring does two useful things. It makes every error count as positive, so that over-predictions and under-predictions can't cancel each other out. And it penalises large errors much more heavily than small ones: an error of 10 counts 100 times as much as an error of 1. As the likelihood topic shows, this choice is also exactly what maximum likelihood recommends when the noise in the data is Gaussian. "Fitting" linear regression means finding the weights that make this MSE as small as possible.
Interactive: this part runs in your browser.
Watch it: Linear regression: finding the best straight line
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Try it: fit a line by hand
Move the sliders to make the red residual lines as short as possible, watching the MSE. Then press Snap to best fit to see the exact least-squares answer.
Interactive: this part runs in your browser.
Solving it
There are two main ways to find the weights that minimise the squared error, and both are used in practice.
The first is an exact formula called the normal equation. If we stack the training examples into a data matrix (with a column of 1s for the intercept) and the targets into a vector , the best weights are
This gives the exact answer in one calculation, with no iteration and no learning rate to tune. Its drawbacks appear at scale: inverting the matrix takes time roughly proportional to , which becomes slow with many features, and the matrix cannot be inverted at all if some features are exact combinations of others. The next section derives this formula step by step.
The second is gradient descent, which starts from some initial weights and repeatedly takes small steps downhill on the squared-error surface. It needs many steps, but each one is cheap, and it scales to enormous datasets because it can work on small batches of data at a time. It is also the method that generalises to models without a neat formula, such as neural networks, which is why it has a topic of its own.
Linear regression relies on a few assumptions, and knowing them helps you recognise when it is the wrong tool. It assumes the relationship between features and target is roughly linear; that the errors for different examples are independent of each other; that the errors have about the same spread everywhere (called homoscedasticity); and that the features are not too strongly correlated with one another (otherwise the individual weights become unstable). Plotting the residuals against the predictions is a quick way to check: a healthy model shows a shapeless cloud, while a curve or a funnel shape signals that an assumption is violated.
The usual summary of fit is , the coefficient of determination:
The denominator measures how much the targets vary around their average, and the numerator measures how much error remains after using the model. So is the fraction of the variation that the model explains. A value of 1 means perfect predictions, and a value of 0 means the model is no better than always predicting the average.
Finally, "linear" refers to the weights, not to the features. If you add new features such as and , the model can fit curves, but it is still a weighted sum, so all of the methods above still apply unchanged. This is called polynomial regression, and it reappears in the bias–variance topic.
Interactive: this part runs in your browser.
Deriving the normal equation
We want the weights that make the total squared error smallest. The error is a bowl-shaped function of the weights, and at the bottom of a bowl the slope is zero in every direction. Setting the gradient to zero gives a system of linear equations, and solving it gives the best weights directly, with no iteration needed.
🔤 Symbols in this section
| design matrix: n examples (rows), d features (columns) | |
| vector of targets | |
| weight vector | |
| squared length = sum of squares | |
| transpose of X | |
| matrix inverse | |
| the "hat" matrix that maps y to predictions |
Here we derive the normal equation carefully, using the matrix calculus from the gradients topic, and then look at what it means geometrically.
We collect the training data into a design matrix , with one row per example and one column per feature (including a column of 1s so that the intercept is just another weight), and the targets into a vector . The predictions for all examples are , and the total squared error is the squared length of the residual vector:
Multiplying out the brackets, and using the facts that and that a single number equals its own transpose, gives three terms:
Now we take the gradient with respect to , one term at a time. The first term does not depend on , so its gradient is zero. The second is linear in , and by the rule it contributes . The third is a quadratic form with the symmetric matrix , and by the rule it contributes . So
At the minimum, the gradient must be zero. Setting it equal to zero and dividing by −2 gives the normal equation, . If can be inverted (which is true whenever no feature is an exact combination of the others), multiplying both sides by its inverse gives the solution:
How do we know this stationary point is a minimum and not a maximum? The Hessian of is , and as shown in the eigenvectors topic, is always positive semi-definite. The loss is therefore a bowl (it is convex), so any point where the gradient is zero is a global minimum.
The normal equation also has a lovely geometric meaning. The predictions can be written as , where is called the hat matrix (because it puts the hat on ). The vector is the orthogonal projection of onto the space spanned by the columns of : out of all the vectors the model could possibly produce, it is the one closest to the true targets. Rewriting the normal equation as says that the residual vector is perpendicular, or "normal", to every feature column. That is where the name comes from, and it makes sense: if the residual still had some component along a feature, we could reduce the error further by adjusting that feature's weight.
Statistical properties, R² and numerical stability
If you collected a different sample of data, you'd get slightly different weights. This section asks: on average, are the weights right (unbiased)? And how much do they wobble from sample to sample (variance)? The wobble is large when features are strongly correlated, because the data can't tell which feature deserves the credit.
🔤 Symbols in this section
| the true weights | |
| random noise | |
| noise variance | |
| covariance matrix of the estimated weights | |
| variance inflation factor for feature j | |
| adjusted R² | |
| QR decomposition (orthogonal × triangular) |
If you collected a fresh sample of data and refitted the model, you would get slightly different weights. This section asks how good the least-squares weights are as estimates: are they right on average, and how much do they vary?
To answer this, assume the data really come from a linear model, , where are the true weights and is random noise with mean zero, variance , and no correlation between examples. Substituting this into the normal equation gives
The estimate equals the true weights plus a term caused by the noise. Since the noise averages to zero, so does that term, and therefore : least squares is unbiased. The variability of the estimate comes entirely from the noise term, and working out its covariance gives
The Gauss–Markov theorem adds that, among all estimators that are linear in and unbiased, least squares has the smallest variance. It is often summarised by saying that least squares is the BLUE, the best linear unbiased estimator.
The covariance formula reveals an important practical danger. When two features are strongly correlated, such as house size in square metres and number of rooms, is close to being non-invertible: its smallest eigenvalue is tiny, so its inverse has a huge entry, and the variance of the weights becomes enormous. Intuitively, the data can't tell which of the two correlated features deserves the credit, so their weights swing wildly from sample to sample even though the predictions stay sensible. This problem is called multicollinearity. It is measured for each feature by the variance inflation factor, , where is how well feature can be predicted from the other features. Ridge regression fixes it by adding to .
In practice is unknown, but it can be estimated from the residuals: . We divide by rather than because fitting weights uses up "degrees of freedom" and makes the residuals slightly smaller than the true noise. Plugging this estimate into the covariance formula gives a standard error for each weight, which statistics packages use to produce confidence intervals and to test whether a weight is significantly different from zero.
A word on comparing models with : adding any feature, even pure noise, can never make the training go down, so it always favours bigger models. The adjusted corrects for this by penalising the number of features :
Finally, a numerical point. Although the formula contains , good software never computes that inverse explicitly. Forming squares the condition number of the problem, which can magnify rounding errors badly. Instead, libraries use a QR decomposition of or the singular value decomposition, which solve the same least-squares problem accurately and also cope with columns that are exact combinations of others. NumPy's np.linalg.lstsq does this for you.
Code: three ways to fit linear regression
The same model fitted by the normal equation, by gradient descent, and by scikit-learn. All three agree.
import numpy as np
rng = np.random.default_rng(0)
n, d = 200, 3
X = rng.normal(size=(n, d))
true_w, true_b = np.array([2.0, -1.0, 0.5]), 4.0
y = X @ true_w + true_b + rng.normal(scale=0.5, size=n)
Xb = np.c_[np.ones(n), X] # add intercept column
# 1) Normal equation, solved stably (QR/SVD under the hood)
w_ls, *_ = np.linalg.lstsq(Xb, y, rcond=None)
print("lstsq: ", w_ls.round(3))
# 2) Batch gradient descent on MSE = (1/n)||y - Xw||^2
w = np.zeros(d + 1)
lr = 0.1
for epoch in range(500):
grad = -2 / n * Xb.T @ (y - Xb @ w) # derived above
w -= lr * grad
print("grad descent:", w.round(3))
# Goodness of fit
y_hat = Xb @ w_ls
r2 = 1 - np.sum((y - y_hat) ** 2) / np.sum((y - y.mean()) ** 2)
print("R^2 =", round(r2, 4))from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
reg = LinearRegression().fit(X, y) # intercept handled for you
print("intercept:", reg.intercept_.round(3), "coef:", reg.coef_.round(3))
print("MSE:", round(mean_squared_error(y, reg.predict(X)), 4),
" R^2:", round(r2_score(y, reg.predict(X)), 4))
# Polynomial regression = PolynomialFeatures + LinearRegression in a pipelineQuiz
1. What does MSE stand for and do?
2. R² = 0 means the model is…
3. Polynomial regression (x, x², x³) is…
4. The normal equation says the residual vector is…
5. Under the standard assumptions, Cov(ŵ) equals…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- scikit-learn: Linear models ↗ Official guide
- An Introduction to Statistical Learning (free book) ↗ James, Witten, Hastie, Tibshirani (Python edition available)