Partial Derivatives, Gradients & the Hessian
Calculus with many weights at once.
Partial derivatives and the gradient
Real models have many weights, not just one, so their loss is a function of many variables at once, written . We still want to know which way to adjust each weight, so we need a version of the derivative that works with several inputs.
The idea is to change one variable at a time. The partial derivative (the curly is read "partial") measures how the loss changes when we nudge only and hold every other weight fixed. In practice this means you differentiate as usual, but treat all the other variables as if they were constants.
Let's work through an example with two variables,
To find , we pretend is just a fixed number. The first term gives . The middle term is " times ", and since is now a constant, its derivative is . The last term doesn't involve at all, so it is a constant and contributes 0. Putting these together, and then doing the same with the roles swapped,
The gradient, written (the upside-down triangle is called "nabla"), simply collects all the partial derivatives into one vector. At the point :
The gradient has a beautiful geometric meaning. Picture the loss as a landscape of hills and valleys, drawn on a map with contour lines (lines of equal height). At any point, the gradient vector points in the direction of steepest ascent, straight uphill, and it is always perpendicular to the contour line through that point. Its length tells you how steep the hill is there. So the negative gradient, , points straight downhill, which is exactly where we want to go when minimising a loss.
That gives us gradient descent in many dimensions: subtract a small multiple of the gradient from the current weights. Starting from with a learning rate ,
Let's check that this step actually helped. At the starting point, . At the new point, . One step reduced the function from 11 to about 2.5. Training a neural network does exactly this, except that the gradient has millions of entries instead of two.
Try it: gradient descent on a 2-D landscape
The shading shows the height of the function (darker = lower). Arrows show the downhill direction everywhere. Click to choose a starting point, then step. Increase the stretch κ to make a narrow valley and watch the zig-zag; tick momentum to smooth it out.
Interactive: this part runs in your browser.
Matrix calculus: the three formulas behind every derivation
Several derivations later in the course (linear regression, ridge regression, logistic regression) use three formulas from matrix calculus. They can look intimidating, but each one is just a familiar single-variable rule written for vectors. Here we derive each one and check it with a small 2-D example, so that you can trust it when it appears later.
Formula 1: the gradient of a linear function. For a fixed vector ,
To see why, write it out in two dimensions: . The partial derivative with respect to is (the other term is a constant), and with respect to it is . Stacking these gives the vector , which is just . This is the vector version of the familiar fact that the derivative of is .
Formula 2: the gradient of a quadratic form. For a square matrix ,
Let's check it with the symmetric matrix . Multiplying out, . The partial derivatives are and . Meanwhile , which is exactly the same. This is the vector version of "the derivative of is ".
Formula 3: the gradient of the squared error. This is the one that matters most. We first expand the squared length using and the transpose rules:
Now differentiate each term with respect to . The first term, , doesn't contain , so its gradient is zero. The middle term is linear in with , so Formula 1 gives . The last term is a quadratic form with the symmetric matrix , so Formula 2 gives . Adding them,
It helps to read this result in words: "minus two, times the features transposed, times the errors". It has exactly the same shape as the single-example result from the chain-rule section, just applied to all the examples at once.
Two more objects complete the toolkit. When a function produces a vector rather than a single number (as a layer of a neural network does), its derivative is a whole matrix of partial derivatives, called the Jacobian, with entries . The chain rule for such functions multiplies Jacobians together, and that is precisely what backpropagation computes.
Finally, the Hessian is the matrix of second derivatives, . Where the gradient describes slope, the Hessian describes curvature: how the slope itself is changing. For our earlier function , differentiating the gradient once more gives
If the Hessian is "positive semi-definite" everywhere (a property explained in the eigenvector topic), the function is shaped like a bowl and has no misleading local minima. This particular function is not a bowl: it curves upwards in one direction and downwards in another, which makes it a saddle. The optimisation topic explains why that distinction matters so much.
Code: verify the gradient of the squared error
import numpy as np
f = lambda w: w[0] ** 2 + 3 * w[0] * w[1] + w[1] ** 2
grad_f = lambda w: np.array([2 * w[0] + 3 * w[1], 3 * w[0] + 2 * w[1]])
w = np.array([1.0, 2.0])
print("gradient at (1,2):", grad_f(w)) # [8, 7]
w_new = w - 0.1 * grad_f(w)
print("after one step:", w_new, " f:", f(w), "->", round(f(w_new), 3)) # 11 -> 2.5
def numerical_grad(F, w, h=1e-6):
g = np.zeros_like(w)
for j in range(len(w)):
e = np.zeros_like(w); e[j] = h
g[j] = (F(w + e) - F(w - e)) / (2 * h) # nudge one coordinate at a time
return g
rng = np.random.default_rng(0)
X, y, w = rng.normal(size=(20, 3)), rng.normal(size=20), rng.normal(size=3)
loss = lambda w: np.sum((y - X @ w) ** 2)
analytic = -2 * X.T @ (y - X @ w) # identity 3
print("analytic: ", analytic.round(5))
print("numerical:", numerical_grad(loss, w).round(5))
print("Hessian eigenvalues of f:", np.linalg.eigvalsh(np.array([[2, 3], [3, 2]]))) # [-1, 5]: a saddlePractice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. For f(a, b) = a²b, ∂f/∂a is…
2. The gradient points in the direction of…
3. ∇_w ‖y − Xw‖² equals…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- 3Blue1Brown: Essence of calculus ↗ Visual intuition
- Mathematics for Machine Learning (Deisenroth, Faisal, Ong) ↗ free textbook covering linear algebra, calculus, probability and optimisation for ML