Maths for AI
Topic 9 · Calculus & optimisation · ⏱ about 4 h

Derivatives & the Chain Rule

Slopes tell a model which way to adjust its weights.

Show:
In this topic
Beginner

A derivative is a slope

Calculus enters machine learning through a single idea: the derivative, which measures how quickly a function's output changes when you nudge its input. Graphically, the derivative at a point is the slope of the curve there, the steepness of the line that just touches the curve at that point (called the tangent).

run hrisesecant (rise/run)tangent: slope f′(x) as h → 0
The derivative is the limit of "rise over run" as the run h shrinks: the secant line turns into the tangent.

How do we measure a slope on a curve, when the steepness keeps changing? We start with something we already know how to do: the slope of a straight line between two points, "rise over run". Take the point xx and a second point a small distance hh further along. The run is hh and the rise is f(x+h)−f(x)f(x + h) - f(x), so the slope between them is f(x+h)−f(x)h\frac{f(x+h) - f(x)}{h}. As we make hh smaller and smaller, the two points move together, and this slope settles down to the slope of the curve itself. That limiting value is the derivative:

f′(x)=lim⁡h→0f(x+h)−f(x)h. f'(x) = \lim_{h\to 0}\frac{f(x+h) - f(x)}{h}.

The derivative is written f′(x)f'(x) (read "f-prime of x") or dfdx\frac{df}{dx}; both mean the same thing.

Let's watch this happen for f(x)=x2f(x) = x^2 at the point x=3x = 3, where f(3)=9f(3) = 9. With a fairly large step, h=1h = 1, the rise-over-run slope is f(4)−f(3)1=16−91=7\frac{f(4) - f(3)}{1} = \frac{16 - 9}{1} = 7. With h=0.1h = 0.1 it is 9.61−90.1=6.1\frac{9.61 - 9}{0.1} = 6.1. With h=0.01h = 0.01 it is 9.0601−90.01=6.01\frac{9.0601 - 9}{0.01} = 6.01. The numbers are clearly closing in on 6, so f′(3)=6f'(3) = 6. Repeating this at other points reveals a pattern, f′(x)=2xf'(x) = 2x, and the derivative rules in the next section let us write that down immediately without any limits.

Why does machine learning care about slopes? Suppose L(w)L(w) is the loss of a model, a number measuring how wrong it is, as a function of one of its weights ww. The derivative L′(w)L'(w) tells us what would happen to the loss if we nudged that weight. If L′(w)L'(w) is positive, the loss rises as ww increases, so we should decrease ww. If L′(w)L'(w) is negative, the loss falls as ww increases, so we should increase ww. And if L′(w)=0L'(w) = 0, the curve is flat, which is what happens at the bottom of a valley.

Notice that in both cases the right move is in the direction opposite to the slope. That observation is the whole of gradient descent, the algorithm that trains almost every model in this course:

w←w−η L′(w), w \leftarrow w - \eta\, L'(w),

where η\eta (the Greek letter eta) is a small positive number called the learning rate that controls how big each step is.

Beginner

The derivative rules (so you never need limits)

Computing every derivative from the limit definition would be painfully slow, so mathematicians have worked out a small set of rules that cover almost every function we need. The table summarises them; the paragraphs after it explain how they are used.

FunctionDerivativeExample
a constant cc00ddx7=0\frac{d}{dx}7 = 0
xnx^n (power rule)nxn−1n x^{n-1}ddxx3=3x2\frac{d}{dx}x^3 = 3x^2
c f(x)c\,f(x)c f′(x)c\,f'(x)ddx5x2=10x\frac{d}{dx}5x^2 = 10x
f+gf + g (sum rule)f′+g′f' + g'ddx(x2+x)=2x+1\frac{d}{dx}(x^2 + x) = 2x + 1
exe^xexe^xit is its own slope
log⁡x\log x1/x1/xat x=2x = 2 the slope is 0.50.5
f⋅gf\cdot g (product rule)f′g+fg′f'g + fg'ddx(xex)=ex+xex\frac{d}{dx}\big(x e^x\big) = e^x + xe^x
max⁡(0,x)\max(0, x) (ReLU)1 if x>0x > 0, 0 if x<0x < 0used in neural networks

Each rule has a simple reason behind it. A constant never changes, so its slope is zero. The power rule says that to differentiate xnx^n you bring the power down in front and reduce the power by one, which is why x2x^2 becomes 2x2x, matching what we found numerically in the previous section. A constant multiplier simply comes along for the ride, and the derivative of a sum is the sum of the derivatives, because the two parts change independently. The exponential is special because its rate of growth is always equal to its current size, so exe^x is its own derivative. The logarithm grows more and more slowly as xx gets larger, and its slope is exactly 1/x1/x. The ReLU function used in neural networks is flat (slope 0) for negative inputs and a 45° line (slope 1) for positive inputs.

Let's combine several rules in one example. To differentiate f(x)=3x2+2x+5f(x) = 3x^2 + 2x + 5, we handle each term separately using the sum rule. The term 3x23x^2 gives 3×2x=6x3 \times 2x = 6x by the power and constant rules. The term 2x2x (which is 2x12x^1) gives 2×1=22 \times 1 = 2. The constant 5 gives 0. Adding these up,

f′(x)=6x+2. f'(x) = 6x + 2.

Now we can use this to find the lowest point of the curve. The bottom of a U-shaped curve is where the slope is zero, so we solve 6x+2=06x + 2 = 0, which gives x=−13x = -\tfrac13. This is the basic recipe for minimising a function by hand: differentiate, set the derivative equal to zero, and solve. Linear regression's normal equation is found in exactly this way, just with many variables at once.

Try it

Try it: watch the secant become the tangent

Shrink hh and watch the orange "rise over run" line approach the green tangent. Move xx to find where the slope is zero.

Interactive: this part runs in your browser.

Intermediate · Maths

The chain rule: the engine of deep learning

Machine learning models are built by feeding the output of one function into another. A neural network, for instance, passes its input through one layer, feeds that result into the next layer, and so on. To train such a model we need derivatives of these compositions of functions, and the tool for that is the chain rule.

xu = g(x)y = f(u)du/dxdy/dudy/dx = dy/du × du/dx (local slopes multiply along the chain)e.g. 2e.g. 2u
The chain rule: to differentiate a function of a function, multiply the local derivatives along the chain. Backpropagation does exactly this, layer by layer.

Suppose y=f(g(x))y = f(g(x)): first gg acts on xx, then ff acts on the result. It is convenient to give the intermediate value a name, u=g(x)u = g(x), so that y=f(u)y = f(u). The chain rule says

dydx=dydu⋅dudx,equivalentlyddxf(g(x))=f′(g(x))⋅g′(x). \frac{dy}{dx} = \frac{dy}{du}\cdot\frac{du}{dx}, \qquad\text{equivalently}\qquad \frac{d}{dx}f(g(x)) = f'(g(x))\cdot g'(x).

The intuition is that rates of change multiply. Imagine three gears: if gear A turns 3 times for every turn of gear B, and gear B turns 2 times for every turn of gear C, then gear A turns 3×2=63\times2 = 6 times for every turn of C. In the same way, if yy changes 3 times as fast as uu, and uu changes 2 times as fast as xx, then yy changes 6 times as fast as xx.

Example 1. Let's differentiate y=(2x+1)2y = (2x + 1)^2. The inner function is u=2x+1u = 2x + 1 and the outer function is y=u2y = u^2. Each piece is easy on its own: dydu=2u\frac{dy}{du} = 2u and dudx=2\frac{du}{dx} = 2. Multiplying them and then replacing uu by 2x+12x + 1 gives

dydx=2u⋅2=4(2x+1). \frac{dy}{dx} = 2u\cdot 2 = 4(2x + 1).

Example 2: the loss for one data point. Suppose a very simple model predicts wxwx, the true answer is yy, and we measure the error by L=(wx−y)2L = (wx - y)^2. We want dLdw\frac{dL}{dw}, because that tells gradient descent how to change ww. The inner function is the error u=wx−yu = wx - y, whose derivative with respect to ww is xx (remember that xx and yy are fixed data values here). The outer function is L=u2L = u^2, with derivative 2u2u. By the chain rule,

dLdw=2u⋅x=2(wx−y) x. \frac{dL}{dw} = 2u\cdot x = 2(wx - y)\,x.

Let's check that this makes sense with numbers. Take x=2x = 2, y=5y = 5 and a current weight w=1w = 1. The model predicts wx=2wx = 2, which is too small, since the answer is 5. The derivative is 2(2−5)(2)=−122(2 - 5)(2) = -12. It is negative, so gradient descent, which moves against the slope, will increase ww, pushing the prediction up towards 5. That is exactly the right thing to do.

Example 3: the slope of the sigmoid. The sigmoid σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}} can be written as u−1u^{-1} with inner function u=1+e−zu = 1 + e^{-z}. The outer derivative is −u−2-u^{-2} by the power rule. The inner derivative needs the chain rule a second time: e−ze^{-z} is the exponential of −z-z, so its derivative is e−z×(−1)=−e−ze^{-z}\times(-1) = -e^{-z}. Multiplying the outer and inner derivatives, the two minus signs cancel:

σ′(z)=−1(1+e−z)2⋅(−e−z)=e−z(1+e−z)2. \sigma'(z) = -\frac{1}{(1+e^{-z})^2}\cdot(-e^{-z}) = \frac{e^{-z}}{(1+e^{-z})^2}.

This can be rewritten in a much more useful form. Split the fraction into two factors, 11+e−z⋅e−z1+e−z\frac{1}{1+e^{-z}}\cdot\frac{e^{-z}}{1+e^{-z}}. The first factor is just σ(z)\sigma(z). For the second, notice that e−z1+e−z=(1+e−z)−11+e−z=1−11+e−z=1−σ(z)\frac{e^{-z}}{1+e^{-z}} = \frac{(1 + e^{-z}) - 1}{1+e^{-z}} = 1 - \frac{1}{1+e^{-z}} = 1 - \sigma(z). Therefore

σ′(z)=σ(z)(1−σ(z)). \sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big).

At z=0z = 0, where σ=0.5\sigma = 0.5, the slope is 0.5×0.5=0.250.5 \times 0.5 = 0.25, the steepest the sigmoid ever gets. Far from zero, σ\sigma is close to 0 or 1 and the slope is almost zero, a fact that will matter when we study why deep networks can be hard to train.

For longer chains the rule simply extends: if y=f3(f2(f1(x)))y = f_3(f_2(f_1(x))), then dydx\frac{dy}{dx} is the product of the three local derivatives, each evaluated at the right point. A neural network is a long chain of layers, and the algorithm that trains it, backpropagation, is nothing more than the chain rule applied systematically from the output back towards the input.

Code

Code: checking derivatives numerically

The definition of the derivative gives a free way to test any formula you derive. This exact trick is how deep-learning libraries test their gradients.

Python · NumPy
import numpy as np

def numerical_derivative(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)       # centred difference: more accurate

f = lambda x: x ** 2
for h in [1, 0.1, 0.01, 0.001]:
    print(f"h={h}: slope = {(f(3 + h) - f(3)) / h:.4f}")
print("rule says f'(3) = 2*3 =", 2 * 3)

# Chain rule check: squared-error loss for one example
x, y = 2.0, 5.0
L = lambda w: (w * x - y) ** 2
dL_dw = lambda w: 2 * (w * x - y) * x
print("analytic:", dL_dw(1.0), " numerical:", round(numerical_derivative(L, 1.0), 6))   # -12

# Sigmoid derivative: sigma * (1 - sigma)
sig = lambda z: 1 / (1 + np.exp(-z))
for z in [-2.0, 0.0, 3.0]:
    print(f"z={z}: formula {sig(z) * (1 - sig(z)):.5f}  numerical {numerical_derivative(sig, z):.5f}")
Open in playground ↗
Beginner

Practice: check yourself

Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.

Question 1.
What is the derivative of f(x)=3x2+2xf(x) = 3x^2 + 2x at x=4x = 4?
Question 2.
Using the chain rule, what is the derivative of f(x)=(2x+1)3f(x) = (2x + 1)^3 at x=1x = 1?
Question 3.
The sigmoid σ(z)\sigma(z) has derivative σ(z)(1−σ(z))\sigma(z)(1 - \sigma(z)). What is it at z=0z = 0?
Check yourself

Quiz

1. d/dx (4x³) =

2. If L′(w) is negative, gradient descent will…

3. d/dx (3x + 1)² =

4. σ′(z) equals…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready