Eigenvectors, Covariance & SVD
The linear algebra behind PCA, conditioning and convexity.
Eigenvectors: directions a matrix only stretches
When you multiply a vector by a matrix, the result usually points in a different direction from the original vector: the matrix both stretches and turns it. But for most matrices there are a few special directions that are only stretched, never turned. A vector in one of those directions is called an eigenvector, and the amount it gets stretched by is its eigenvalue, written (lambda). In symbols,
Let's see this with the matrix . Multiply it by the vector : the result is , which is exactly . The vector kept its direction and was stretched by a factor of 3, so is an eigenvector with eigenvalue 3. Now try : the result is , unchanged, so it is an eigenvector with eigenvalue 1. By contrast, becomes , which points in a different direction, so is not an eigenvector.
How do we find eigenvalues without guessing? Start from and move everything to one side: , which can be written . We want a non-zero vector that this matrix sends to zero. That can only happen if the matrix squashes space flat, which (as we saw in the matrices topic) means its determinant is zero:
This quadratic factorises as , so the eigenvalues are 3 and 1, which matches what we found by hand. For each eigenvalue, you can then find its eigenvector by solving .
Symmetric matrices, those that equal their own transpose, are especially well behaved, and they are the ones that matter most in machine learning, because covariance matrices and Hessians are always symmetric. A symmetric matrix always has real eigenvalues, and its eigenvectors are perpendicular to each other (in our example, and are at right angles). This means a symmetric matrix can be understood as a simple three-step action: rotate so that the eigenvectors line up with the axes, stretch along each axis by the corresponding eigenvalue, then rotate back. In symbols, , where the columns of are the eigenvectors and is a diagonal matrix of the eigenvalues.
Two quick checks are useful when you compute eigenvalues. Their sum always equals the sum of the diagonal entries (the trace): here . Their product always equals the determinant: here .
Try it: find the eigenvectors
Start with the symmetric preset (this matrix). The dashed lines are the eigenvectors: points on them stay on their line. Try rotate: no real eigenvectors, because every direction turns.
Interactive: this part runs in your browser.
Covariance and the covariance matrix
Many datasets have features that tend to move together: taller people tend to be heavier, and bigger houses tend to have more bedrooms. Covariance measures this tendency. For two features and measured on examples,
To see what this formula is doing, look at a single term. If, for some example, both and are above their averages, both brackets are positive and their product is positive. If both are below average, both brackets are negative and their product is again positive. If one is above and the other below, the product is negative. So the covariance is positive when the two features tend to be above or below average together, negative when one tends to be high while the other is low, and close to zero when there is no consistent linear relationship. Because covariance depends on the units of measurement, we often divide by both standard deviations to get the correlation, , which always lies between −1 and 1.
Let's compute everything for a tiny dataset of three points, , and . The means are and . Subtracting them ("centring" the data) gives , and . Now each quantity is just an average of products, dividing by :
The correlation is , the maximum possible, which makes sense because the three points lie exactly on the line .
With many features, we collect all the variances and covariances into a single covariance matrix . The variances go on the diagonal and the covariances go off the diagonal:
There is a neat way to compute it in one step. If is the centred data matrix (one row per example), then , because each entry of is exactly a sum of products of centred values.
The covariance matrix is symmetric, so it has perpendicular eigenvectors, and these have a very useful meaning: they point along the directions in which the data is most and least spread out, and the eigenvalues measure how much spread there is in each direction. For our example, the eigenvectors are with eigenvalue 5 and with eigenvalue 0. The first points along the line the data lies on; the second is perpendicular to it, and its eigenvalue of zero tells us there is no spread at all in that direction. Those eigenvectors are precisely the principal components that PCA uses to compress data.
Interactive: this part runs in your browser.
Positive definiteness, condition number and the SVD
This section introduces three ideas that appear throughout the advanced material: positive definiteness, the condition number, and the singular value decomposition.
A symmetric matrix is called positive semi-definite (PSD) if for every vector . It can be shown that this is the same as saying that all of its eigenvalues are zero or positive. The geometric meaning is the important part. The expression is a quadratic function of , and when is PSD this function is shaped like a bowl: there is no direction in which it slopes downwards forever. If the inequality is strict ( for every non-zero , equivalently all eigenvalues positive), the matrix is called positive definite, the bowl has a single lowest point, and the matrix is invertible.
A particularly important fact is that is always positive semi-definite, for any matrix . The proof is one line: , and a squared length can never be negative. This has two big consequences. First, the least-squares loss in linear regression, whose curvature is , is always a bowl, so it has no misleading local minima. Second, covariance matrices (which are divided by a positive number) never have negative eigenvalues, which matches the common-sense fact that variances can't be negative.
The condition number of a symmetric positive definite matrix is the ratio of its largest eigenvalue to its smallest, . For a loss function, the eigenvalues of the Hessian describe how steeply the bowl curves in each direction. If one direction curves 100 times more steeply than another, , and the bowl is a long, narrow valley. As the gradient descent topic shows, the algorithm then zig-zags between the steep walls while making slow progress along the flat floor. Scaling the features so they have similar ranges brings down and makes training much faster.
Eigenvectors only make sense for square matrices, but a data matrix usually has many more rows than columns. The singular value decomposition (SVD) is the tool that works for any matrix. It says that every matrix can be written as a product of three simpler matrices:
Here and are rotation-like matrices (their columns are perpendicular unit vectors), and is a diagonal matrix of non-negative numbers called the singular values. In words, every matrix acts by rotating, then stretching along the axes by the singular values, then rotating again.
The SVD is closely connected to eigenvalues. Substituting it into and using the fact that gives . Comparing this with the form from the first section, we see that the eigenvectors of are the columns of , and its eigenvalues are the squared singular values .
Two applications make the SVD indispensable. First, if you keep only the largest singular values and set the rest to zero, you get the best possible approximation of using a matrix of rank , which is the mathematical basis of PCA, image compression and recommender systems. Second, the pseudo-inverse (inverting only the non-zero singular values) solves least-squares problems reliably even when cannot be inverted, which is what NumPy's lstsq does behind the scenes.
Code: eigenvalues, covariance and SVD
import numpy as np
A = np.array([[2.0, 1.0], [1.0, 2.0]])
vals, vecs = np.linalg.eigh(A) # eigh: for symmetric matrices, sorted ascending
print("eigenvalues:", vals) # [1, 3]
print("eigenvectors (columns):\n", vecs.round(3))
v = vecs[:, 1]
print("A v =", A @ v, " lambda v =", vals[1] * v) # same vector
print("trace", np.trace(A), "= sum", vals.sum(), "| det", round(np.linalg.det(A), 6), "= product", vals.prod())
pts = np.array([[1, 2], [2, 4], [3, 6]], dtype=float)
S = np.cov(pts, rowvar=False) # rows are observations
print("covariance matrix:\n", S) # [[1, 2], [2, 4]]
print("its eigenvalues:", np.linalg.eigvalsh(S).round(6)) # [0, 5]
rng = np.random.default_rng(0)
X = rng.normal(size=(50, 3))
print("X^T X eigenvalues all >= 0 (PSD):", np.all(np.linalg.eigvalsh(X.T @ X) >= -1e-10))
U, sing, Vt = np.linalg.svd(X, full_matrices=False)
print("X == U diag(s) V^T ?", np.allclose(X, U @ np.diag(sing) @ Vt))
print("sigma^2:", (sing ** 2).round(3), " eig(X^T X):", np.linalg.eigvalsh(X.T @ X)[::-1].round(3))
X1 = sing[0] * np.outer(U[:, 0], Vt[0]) # best rank-1 approximation
print("rank-1 approximation error:", round(np.linalg.norm(X - X1) ** 2, 3), "= sigma2^2 + sigma3^2 =", round(sing[1] ** 2 + sing[2] ** 2, 3))Practice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. If Av = 4v, then v is an eigenvector with eigenvalue…
2. Eigenvectors of a symmetric matrix (with distinct eigenvalues) are…
3. Why is XᵀX always positive semi-definite?
4. The eigenvalues of XᵀX equal…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- 3Blue1Brown: Essence of linear algebra ↗ Visual intuition
- Mathematics for Machine Learning (Deisenroth, Faisal, Ong) ↗ free textbook covering linear algebra, calculus, probability and optimisation for ML