Derivatives & the Chain Rule
Slopes tell a model which way to adjust its weights.
A derivative is a slope
Calculus enters machine learning through a single idea: the derivative, which measures how quickly a function's output changes when you nudge its input. Graphically, the derivative at a point is the slope of the curve there, the steepness of the line that just touches the curve at that point (called the tangent).
How do we measure a slope on a curve, when the steepness keeps changing? We start with something we already know how to do: the slope of a straight line between two points, "rise over run". Take the point and a second point a small distance further along. The run is and the rise is , so the slope between them is . As we make smaller and smaller, the two points move together, and this slope settles down to the slope of the curve itself. That limiting value is the derivative:
The derivative is written (read "f-prime of x") or ; both mean the same thing.
Let's watch this happen for at the point , where . With a fairly large step, , the rise-over-run slope is . With it is . With it is . The numbers are clearly closing in on 6, so . Repeating this at other points reveals a pattern, , and the derivative rules in the next section let us write that down immediately without any limits.
Why does machine learning care about slopes? Suppose is the loss of a model, a number measuring how wrong it is, as a function of one of its weights . The derivative tells us what would happen to the loss if we nudged that weight. If is positive, the loss rises as increases, so we should decrease . If is negative, the loss falls as increases, so we should increase . And if , the curve is flat, which is what happens at the bottom of a valley.
Notice that in both cases the right move is in the direction opposite to the slope. That observation is the whole of gradient descent, the algorithm that trains almost every model in this course:
where (the Greek letter eta) is a small positive number called the learning rate that controls how big each step is.
The derivative rules (so you never need limits)
Computing every derivative from the limit definition would be painfully slow, so mathematicians have worked out a small set of rules that cover almost every function we need. The table summarises them; the paragraphs after it explain how they are used.
| Function | Derivative | Example |
|---|---|---|
| a constant | ||
| (power rule) | ||
| (sum rule) | ||
| it is its own slope | ||
| at the slope is | ||
| (product rule) | ||
| (ReLU) | 1 if , 0 if | used in neural networks |
Each rule has a simple reason behind it. A constant never changes, so its slope is zero. The power rule says that to differentiate you bring the power down in front and reduce the power by one, which is why becomes , matching what we found numerically in the previous section. A constant multiplier simply comes along for the ride, and the derivative of a sum is the sum of the derivatives, because the two parts change independently. The exponential is special because its rate of growth is always equal to its current size, so is its own derivative. The logarithm grows more and more slowly as gets larger, and its slope is exactly . The ReLU function used in neural networks is flat (slope 0) for negative inputs and a 45° line (slope 1) for positive inputs.
Let's combine several rules in one example. To differentiate , we handle each term separately using the sum rule. The term gives by the power and constant rules. The term (which is ) gives . The constant 5 gives 0. Adding these up,
Now we can use this to find the lowest point of the curve. The bottom of a U-shaped curve is where the slope is zero, so we solve , which gives . This is the basic recipe for minimising a function by hand: differentiate, set the derivative equal to zero, and solve. Linear regression's normal equation is found in exactly this way, just with many variables at once.
Try it: watch the secant become the tangent
Shrink and watch the orange "rise over run" line approach the green tangent. Move to find where the slope is zero.
Interactive: this part runs in your browser.
The chain rule: the engine of deep learning
Machine learning models are built by feeding the output of one function into another. A neural network, for instance, passes its input through one layer, feeds that result into the next layer, and so on. To train such a model we need derivatives of these compositions of functions, and the tool for that is the chain rule.
Suppose : first acts on , then acts on the result. It is convenient to give the intermediate value a name, , so that . The chain rule says
The intuition is that rates of change multiply. Imagine three gears: if gear A turns 3 times for every turn of gear B, and gear B turns 2 times for every turn of gear C, then gear A turns times for every turn of C. In the same way, if changes 3 times as fast as , and changes 2 times as fast as , then changes 6 times as fast as .
Example 1. Let's differentiate . The inner function is and the outer function is . Each piece is easy on its own: and . Multiplying them and then replacing by gives
Example 2: the loss for one data point. Suppose a very simple model predicts , the true answer is , and we measure the error by . We want , because that tells gradient descent how to change . The inner function is the error , whose derivative with respect to is (remember that and are fixed data values here). The outer function is , with derivative . By the chain rule,
Let's check that this makes sense with numbers. Take , and a current weight . The model predicts , which is too small, since the answer is 5. The derivative is . It is negative, so gradient descent, which moves against the slope, will increase , pushing the prediction up towards 5. That is exactly the right thing to do.
Example 3: the slope of the sigmoid. The sigmoid can be written as with inner function . The outer derivative is by the power rule. The inner derivative needs the chain rule a second time: is the exponential of , so its derivative is . Multiplying the outer and inner derivatives, the two minus signs cancel:
This can be rewritten in a much more useful form. Split the fraction into two factors, . The first factor is just . For the second, notice that . Therefore
At , where , the slope is , the steepest the sigmoid ever gets. Far from zero, is close to 0 or 1 and the slope is almost zero, a fact that will matter when we study why deep networks can be hard to train.
For longer chains the rule simply extends: if , then is the product of the three local derivatives, each evaluated at the right point. A neural network is a long chain of layers, and the algorithm that trains it, backpropagation, is nothing more than the chain rule applied systematically from the output back towards the input.
Code: checking derivatives numerically
The definition of the derivative gives a free way to test any formula you derive. This exact trick is how deep-learning libraries test their gradients.
import numpy as np
def numerical_derivative(f, x, h=1e-5):
return (f(x + h) - f(x - h)) / (2 * h) # centred difference: more accurate
f = lambda x: x ** 2
for h in [1, 0.1, 0.01, 0.001]:
print(f"h={h}: slope = {(f(3 + h) - f(3)) / h:.4f}")
print("rule says f'(3) = 2*3 =", 2 * 3)
# Chain rule check: squared-error loss for one example
x, y = 2.0, 5.0
L = lambda w: (w * x - y) ** 2
dL_dw = lambda w: 2 * (w * x - y) * x
print("analytic:", dL_dw(1.0), " numerical:", round(numerical_derivative(L, 1.0), 6)) # -12
# Sigmoid derivative: sigma * (1 - sigma)
sig = lambda z: 1 / (1 + np.exp(-z))
for z in [-2.0, 0.0, 3.0]:
print(f"z={z}: formula {sig(z) * (1 - sig(z)):.5f} numerical {numerical_derivative(sig, z):.5f}")Practice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. d/dx (4x³) =
2. If L′(w) is negative, gradient descent will…
3. d/dx (3x + 1)² =
4. σ′(z) equals…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- 3Blue1Brown: Essence of calculus ↗ Visual intuition
- Mathematics for Machine Learning (Deisenroth, Faisal, Ong) ↗ free textbook covering linear algebra, calculus, probability and optimisation for ML