Machine Learning
Topic 21 · Unsupervised learning · ⏱ about 4 h

Dimensionality Reduction & PCA

Compressing many features into a few that keep most of the information.

Show:
In this topic
Beginner

Why reduce dimensions?

Many datasets have far more features than are really needed to describe them. An image might have thousands of pixels, a gene-expression dataset tens of thousands of genes, a survey hundreds of questions, and many of those features are strongly related to one another. Dimensionality reduction means representing the data with fewer features while keeping as much of the useful information as possible.

There are three main reasons to do this. The first is visualisation: we can only look at data in two or three dimensions, so reducing many features down to two lets us plot the data and see its structure. The second is to make learning easier and faster: fewer features means faster training, and it eases the curse of dimensionality that troubles distance-based methods. The third is to remove noise and redundancy: when many features measure nearly the same thing, combining them into one more reliable feature can help a model.

There are two broad strategies. Feature selection keeps a subset of the original features and discards the rest, which keeps the result easy to interpret. Feature extraction instead creates a smaller number of brand-new features, each a combination of the originals. Principal component analysis, the subject of this topic, is the most important feature-extraction method.

Beginner

Watch it: PCA: finding the directions that matter

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

Principal Component Analysis

Principal component analysis (PCA) finds a new set of axes for the data, chosen so that the first axis captures as much of the data's variation as possible, the second captures as much of the remaining variation as possible, and so on. These new axes are called the principal components.

PC1: direction of most variancePC2: next most, at 90°
PCA finds new axes: PC1 along the longest spread of the data, PC2 perpendicular to it. Keeping only PC1 keeps most of the information.

Imagine a cloud of points shaped like a tilted rugby ball. The first principal component is the direction along the length of the ball, the direction in which the points are most spread out. The second principal component is the direction of greatest remaining spread, and it is always at right angles to the first; for the rugby ball, it is across its width. With more dimensions this continues, each new component at right angles to all the previous ones and capturing less variation than the one before. If most of the variation lies along the first few components, we can describe each point by its positions along just those few axes and lose very little information.

PCA is carried out in four steps. First, standardise the features. This step is essential, because PCA looks for directions of large variance, and a feature measured in large units (say, salary in dollars) would otherwise dominate simply because its numbers are bigger. Second, compute the covariance matrix of the standardised data, which records how every pair of features varies together. Third, find the covariance matrix's eigenvectors and eigenvalues: the eigenvectors are the principal component directions, and each eigenvalue is the amount of variance along its direction. (Software actually uses the closely related singular value decomposition, which is more accurate.) Fourth, keep the kk components with the largest eigenvalues and project the data onto them; each point's new coordinates are its positions along those kk axes.

How many components should you keep? The explained variance ratio of each component is its eigenvalue divided by the sum of all the eigenvalues, the fraction of the total variation it captures. A common rule is to keep just enough components to explain, say, 95% of the variance. Another is to plot the ratios in decreasing order, a scree plot, and keep the components before the curve flattens out.

Interactive: this part runs in your browser.

Breakdown. How the total variance splits across principal components, and what each component is made of.
Try it

Try it: find PC1

Rotate the orange axis. Each blue point is projected onto it (grey lines). Find the angle that maximises the spread of the projections, then snap to PC1 to check.

Interactive: this part runs in your browser.

Beginner

Beyond PCA

PCA is the right tool for many problems, but it has limitations, and several other methods address them.

PCA's strengths are that it is fast, well understood, and reversible: from the reduced representation you can reconstruct an approximation of the original data, which makes it useful for compression and noise removal. Its main limitation is that it is linear. It can only find flat directions, so if the data lie along a curved surface, such as a spiral or a rolled-up sheet, PCA can't "unroll" it. A second limitation is that its components are mixtures of all the original features, which can make them hard to interpret.

t-SNE is a non-linear method designed specifically for visualisation. It places points in two dimensions so that points that were close neighbours in the original space remain close neighbours in the picture. It is excellent at revealing clusters, but two cautions apply: the sizes of clusters and the distances between clusters in a t-SNE plot are not meaningful, and the result changes with its settings, so it should be used for exploration, not as input to other models. UMAP works in a similar spirit but is faster and tends to preserve more of the global arrangement of the clusters.

Autoencoders are neural networks trained to compress their input into a small "bottleneck" representation and then reconstruct the input from it. Because they are built from non-linear layers, they can learn curved structure, and they can be thought of as a non-linear generalisation of PCA. (An autoencoder with only linear layers learns essentially the same subspace as PCA.)

Finally, linear discriminant analysis (LDA) is a supervised alternative. PCA ignores labels and looks for directions of greatest overall variance; LDA uses the labels and looks for the directions that best separate the classes. When the goal is classification, those can be quite different directions.

MethodTypeBest used for
PCAlinear, unsupervisedcompression, preprocessing, removing redundancy
t-SNE / UMAPnon-linear, unsupervised2-D visualisation of clusters
Autoencodernon-linear, unsupervisedlearned compression of images, audio, etc.
LDAlinear, supervisedfinding directions that separate known classes
Intermediate · Maths

Deriving PCA as an eigenvalue problem

📐 Maths you'll need: m-eigen · m-optim · m-vectors
💡 The idea in plain words

PCA looks for the direction in which the data is most spread out. Spread along a direction is a simple formula, uᵀSu, and maximising it while keeping u a unit vector (Lagrange multipliers!) leads straight to an eigenvector equation. So the principal components are the eigenvectors of the covariance matrix, and the eigenvalues are how much spread each one captures.

🔤 Symbols in this section
SScovariance matrix of the (centred) data
uucandidate direction (unit vector)
λ\lambdaLagrange multiplier = eigenvalue = variance captured
UkU_kmatrix of the top k eigenvectors
Z=XUkZ = XU_kdata in the new k-dimensional coordinates

This section derives PCA from first principles. The derivation shows that "the direction of greatest variance" leads directly to an eigenvector problem, which is why eigenvectors appear in PCA.

We start with data that have been centred, so that every feature has mean zero, and write S=1n−1X⊤XS = \frac{1}{n-1}X^\top X for the covariance matrix. Consider any direction, given by a unit vector uu (with u⊤u=1u^\top u = 1). Projecting every point onto this direction gives one number per point, xi⊤ux_i^\top u, and collecting them gives the vector XuXu. Because the data are centred, these projections have mean zero, so their variance is the average of their squares:

Var⁡(Xu)=1n−1(Xu)⊤(Xu)=u⊤(1n−1X⊤X)u=u⊤Su. \operatorname{Var}(Xu) = \frac{1}{n-1}(Xu)^\top(Xu) = u^\top\Big(\frac{1}{n-1}X^\top X\Big)u = u^\top S u.

PCA's first principal component is the direction that makes this variance as large as possible:

max⁡u u⊤Susubject tou⊤u=1. \max_u\ u^\top S u \quad\text{subject to}\quad u^\top u = 1.

The constraint is essential: without it, we could make the variance as large as we like simply by making uu longer. This is a constrained optimisation problem, so we use a Lagrange multiplier λ\lambda. The Lagrangian is L=u⊤Su−λ(u⊤u−1)\mathcal L = u^\top S u - \lambda(u^\top u - 1), and setting its gradient with respect to uu to zero (using the rule ∇u(u⊤Su)=2Su\nabla_u(u^\top Su) = 2Su) gives 2Su−2λu=02Su - 2\lambda u = 0, that is,

Su=λu. \boxed{S u = \lambda u}.

So the best direction must be an eigenvector of the covariance matrix. Which one? Multiplying both sides on the left by u⊤u^\top and using u⊤u=1u^\top u = 1 gives u⊤Su=λu^\top S u = \lambda: the variance captured by an eigenvector is exactly its eigenvalue. To capture the most variance, we pick the eigenvector with the largest eigenvalue. That is the first principal component. If we then ask for the best direction that is perpendicular to the first, the same argument gives the eigenvector with the second-largest eigenvalue, and so on. Because SS is symmetric, its eigenvectors are automatically perpendicular to each other and its eigenvalues are real and non-negative, so everything fits together neatly.

The fraction of the total variance explained by component kk, and the new reduced features, are

explained variance ratiok=λk∑jλj,Z=X Uk, \text{explained variance ratio}_k = \frac{\lambda_k}{\sum_j \lambda_j}, \qquad Z = X\,U_k,

where UkU_k is the matrix whose columns are the top kk eigenvectors, so ZZ has one row per point and kk columns.

There is a second, equally natural way to arrive at PCA. Instead of maximising the variance kept, we could minimise the information lost: project each point onto a kk-dimensional subspace, map it back, and measure the squared reconstruction error ∑i∥xi−UkUk⊤xi∥2\sum_i\lVert x_i - U_kU_k^\top x_i\rVert^2. The best subspace for this is exactly the span of the top kk eigenvectors, and the minimum error equals (n−1)(n-1) times the sum of the discarded eigenvalues, ∑j>kλj\sum_{j>k}\lambda_j. Since the total variance is fixed, keeping the most variance and losing the least information are the same goal, seen from opposite sides.

Key takeaway. Maximising uᵀSu with ‖u‖ = 1 gives Su = λu: PCs are eigenvectors of the covariance matrix, and keeping the top ones minimises reconstruction error.
Advanced

SVD, low-rank approximation, kernel PCA and whitening

📐 Maths you'll need: m-eigen · m-matrices
💡 The idea in plain words

In practice, PCA is computed with the SVD of the data matrix itself, which is more accurate than building the covariance matrix. The SVD also gives the best possible low-rank compression of any matrix. Kernel PCA and probabilistic PCA extend the idea to curved structure and to probabilistic models.

🔤 Symbols in this section
X=UΣV⊤X = U\Sigma V^\topSVD of centred data
σj\sigma_jj-th singular value
XkX_kbest rank-k approximation
∥⋅∥F\lVert\cdot\rVert_FFrobenius norm (square root of sum of squared entries)
K~\tilde Kcentred kernel matrix
z∼N(0,I)z\sim\mathcal N(0, I)latent variable in probabilistic PCA

This section connects PCA to the singular value decomposition, which is how PCA is actually computed, and then describes three extensions.

PCA via the SVD. Any matrix can be factorised by the singular value decomposition. For the centred data matrix, write X=UΣV⊤X = U\Sigma V^\top, where the columns of UU and VV are perpendicular unit vectors and Σ\Sigma is diagonal with the singular values σ1≥σ2≥…\sigma_1 \ge \sigma_2 \ge \dots. Substituting this into the covariance matrix, and using U⊤U=IU^\top U = I:

S=1n−1X⊤X=1n−1VΣU⊤UΣV⊤=V Σ2n−1 V⊤. S = \frac{1}{n-1}X^\top X = \frac{1}{n-1}V\Sigma U^\top U\Sigma V^\top = V\,\frac{\Sigma^2}{n-1}\,V^\top.

This is exactly the eigen-decomposition of SS. So the principal directions are the columns of VV, the eigenvalues are λj=σj2/(n−1)\lambda_j = \sigma_j^2/(n-1), and the projected data are XVk=UkΣkXV_k = U_k\Sigma_k. Computing the SVD of XX directly is preferred in practice because forming X⊤XX^\top X first would square the condition number and lose numerical accuracy, and for very large matrices, randomised SVD algorithms compute just the top components quickly.

Best low-rank approximation. The Eckart–Young theorem states that, among all matrices of rank kk, the one closest to XX (measured by the Frobenius norm, the square root of the sum of squared entries) is the truncated SVD, which keeps only the top kk singular values: Xk=UkΣkVk⊤X_k = U_k\Sigma_kV_k^\top. Its error is exactly the sum of the discarded squared singular values, ∥X−Xk∥F2=∑j>kσj2\lVert X - X_k\rVert_F^2 = \sum_{j>k}\sigma_j^2. This single theorem underlies PCA compression, latent semantic analysis of text, recommender systems built on matrix factorisation, and low-rank adaptation methods for fine-tuning large models.

Whitening. After projecting onto the principal components, we can also divide each component by the square root of its eigenvalue: Z=XVΛ−1/2Z = XV\Lambda^{-1/2}. The resulting features are uncorrelated and all have variance 1, so their covariance matrix is the identity. This "whitening" is sometimes used as a preprocessing step.

Kernel PCA. Just as SVMs use kernels to work in a richer feature space, PCA can be carried out in a feature space ϕ(x)\phi(x) without ever computing ϕ\phi. The computation needs only the kernel matrix of all pairwise similarities, centred appropriately (call it K~\tilde K): we solve the eigenproblem K~a=nλa\tilde K a = n\lambda a, and a new point's projection onto a component is ∑iai k(xi,x)\sum_i a_i\,k(x_i, x). With a non-linear kernel such as the RBF kernel, kernel PCA can capture curved structure, such as separating two concentric circles, that ordinary PCA cannot.

Probabilistic PCA. PCA can also be phrased as a probability model, in which each observed point is generated from a small number of hidden variables: x=Wz+μ+εx = Wz + \mu + \varepsilon, where zz is drawn from a standard Gaussian and ε\varepsilon is small Gaussian noise. Fitting this model by maximum likelihood recovers the principal subspace. This view links PCA to factor analysis, allows it to handle missing values in a principled way, and, when the linear map WW is replaced by a neural network, leads to the variational autoencoder.

Key takeaway. Principal directions = columns of V, eigenvalues = σ²/(n−1); truncated SVD is the optimal low-rank approximation (Eckart–Young).
Code

Code: PCA from scratch with SVD, and in scikit-learn

The scratch version checks that the eigenvalues from the covariance matrix match the squared singular values.

Python · NumPy from scratch
import numpy as np
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True)          # 1797 images, 64 pixel features
Xc = X - X.mean(0)                           # centre (scaling optional: pixels share units)

# Route 1: eigen-decomposition of the covariance matrix
S = Xc.T @ Xc / (len(X) - 1)
eigvals, eigvecs = np.linalg.eigh(S)         # ascending order
eigvals, eigvecs = eigvals[::-1], eigvecs[:, ::-1]

# Route 2: SVD of the data (preferred numerically)
U, sing, Vt = np.linalg.svd(Xc, full_matrices=False)
print("eigvals match sigma^2/(n-1):", np.allclose(eigvals[:10], sing[:10] ** 2 / (len(X) - 1)))

ratio = eigvals / eigvals.sum()
k = np.searchsorted(np.cumsum(ratio), 0.95) + 1
print(f"{k} of 64 components keep 95% of the variance")

Z = Xc @ Vt[:k].T                            # project: n x k
X_rec = Z @ Vt[:k] + X.mean(0)               # reconstruct
err = np.sum((X - X_rec) ** 2)
print("reconstruction error:", round(err), " sum of discarded sigma^2:", round(np.sum(sing[k:] ** 2)))
Open in playground ↗
Python · scikit-learn
from sklearn.decomposition import PCA, KernelPCA
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.manifold import TSNE

pca = PCA(n_components=0.95, svd_solver="full").fit(X)   # keep 95% variance
print("components kept:", pca.n_components_)
print("first 5 ratios:", pca.explained_variance_ratio_[:5].round(3))

# PCA as a preprocessing step inside a pipeline
clf = make_pipeline(StandardScaler(), PCA(n_components=30), LogisticRegression(max_iter=2000))
print("CV accuracy with PCA(30):", cross_val_score(clf, X, y, cv=5).mean().round(3))

# Non-linear alternatives
Z_kpca = KernelPCA(n_components=2, kernel="rbf", gamma=0.01).fit_transform(X)
Z_tsne = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(X)   # visualisation only
print(Z_kpca.shape, Z_tsne.shape)
Open in playground ↗
Before it runs, the code block above runs first (silently), because this one continues it.
Check yourself

Quiz

1. The first principal component is the direction of…

2. Why standardise before PCA?

3. t-SNE is mainly used for…

4. The variance captured by a principal component equals…

5. If X = UΣVᵀ (centred), the principal directions are…

6. PCA's reconstruction error equals…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready