Matrices & Matrix Multiplication
A dataset is a matrix; a layer of a neural net is a matrix multiply.
Matrices and the matrix–vector product
A matrix is a rectangular table of numbers. A matrix with rows and columns is called an " by " matrix, written . In machine learning, the most important matrix is the data matrix , which stacks all the examples on top of each other: each row is one example and each column is one feature. Here is a tiny dataset of three houses, each described by its floor area and its number of bedrooms, together with a weight vector:
is a matrix (three houses, two features) and is a column vector with two entries, one weight per feature.
The most useful thing we can do with these is the matrix–vector product . The rule is simple: take the dot product of each row of with the vector , and stack the answers into a new column. Because each row is one house, each dot product is the prediction for that house:
A single matrix–vector product therefore computes the predictions for every example at once. This is why linear regression is written compactly as , and it is also why GPUs, which are extremely fast at multiplying matrices, made deep learning practical.
Matrix multiplication only works when the shapes fit together, and the rule for this is the most useful debugging tool you will have when writing ML code. An matrix can multiply an matrix, and the result is :
The two "inner" numbers must match, because each row of the first matrix (length ) has to be dotted with each column of the second (also length ). The "outer" numbers give the shape of the answer. In our example, : three predictions, one per house. Whenever your code crashes with a "shape mismatch" error, writing out the shapes like this almost always shows you what went wrong.
Matrix × matrix, transpose, identity and inverse
Multiplying two matrices extends the same idea. The entry in row and column of the product is the dot product of row of with column of . Let's work through a example carefully:
The top-left entry, 19, comes from the first row dotted with the first column ; the top-right entry, 22, comes from the same row dotted with the second column ; and so on. An important difference from ordinary numbers is that order matters: in general . If you multiply these two matrices the other way round you get a completely different answer, which you can check in the code section.
The transpose of a matrix, written , is obtained by flipping it so that its rows become columns:
Transposing twice gets you back where you started, , and the transpose of a product reverses the order, ; this second rule is used in nearly every derivation in the course. Transposes also explain the notation for the dot product: if and are column vectors, then is a row, and a row times a column is exactly the dot product. In particular, .
The identity matrix has 1s along its diagonal and 0s everywhere else. Multiplying by it changes nothing, , so it plays the role that the number 1 plays in ordinary arithmetic.
That leads to the inverse. For ordinary numbers, dividing by 5 undoes multiplying by 5. For a square matrix , the inverse is the matrix that undoes , in the sense that . For a matrix there is a simple formula:
The number in the denominator is called the determinant. If the determinant is zero, the formula would divide by zero, so the inverse does not exist. Geometrically, a zero determinant means the matrix squashes the plane flat onto a line, and once two different points have been squashed onto the same point, there is no way to tell them apart again. In linear regression, this happens when two features are perfectly correlated (for example, a temperature recorded in both Celsius and Fahrenheit).
Inverses let us solve systems of linear equations. Consider the pair of equations and . We can write them as one matrix equation,
The determinant of the coefficient matrix is , which is not zero, so an inverse exists. Using the formula and multiplying both sides by the inverse:
We can check the answer by substituting back: and , so both equations are satisfied. The famous normal equation of linear regression, , is exactly this idea applied to a larger system. (In real code we ask the computer to solve the system directly rather than computing an inverse, which is faster and more accurate, but the mathematics is the same.)
Try it: step through a matrix multiplication
Press Next entry to compute the product one entry at a time. Each entry is one row of A dotted with one column of B.
Interactive: this part runs in your browser.
Try it: a matrix is a transformation
A matrix moves every point in space. The columns of are where the basis arrows and land. Try the presets, and type your own numbers.
Interactive: this part runs in your browser.
Code: matrices in NumPy
import numpy as np
X = np.array([[120, 3], [80, 2], [200, 4]], dtype=float) # 3 x 2
w = np.array([0.5, 10.0]) # length 2
print("X shape:", X.shape, " predictions Xw:", X @ w) # [90, 60, 140]
A, B = np.array([[1, 2], [3, 4]]), np.array([[5, 6], [7, 8]])
print("AB =\n", A @ B, "\nBA =\n", B @ A) # different!
print("A transpose:\n", A.T)
print("(AB)^T == B^T A^T ?", np.array_equal((A @ B).T, B.T @ A.T))
M = np.array([[2.0, 1.0], [1.0, 3.0]])
print("det:", np.linalg.det(M), "\ninverse:\n", np.linalg.inv(M))
print("solve 2x+y=5, x+3y=10 ->", np.linalg.solve(M, [5, 10])) # [1, 3]
# Prefer np.linalg.solve over inv(M) @ b: faster and more accurate.
try:
np.ones((3, 2)) @ np.ones((3, 1)) # inner dims 2 and 3 don't match
except ValueError as e:
print("shape error:", e)Practice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. A (4×3) matrix times a (3×2) matrix gives a…
2. (AB)ᵀ equals…
3. A matrix with determinant 0…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- 3Blue1Brown: Essence of linear algebra ↗ Visual intuition
- Mathematics for Machine Learning (Deisenroth, Faisal, Ong) ↗ free textbook covering linear algebra, calculus, probability and optimisation for ML