Reading ML Maths: Notation, exp & log
The symbols, functions and log rules that every ML formula uses.
Why learn the notation first?
When people first open a machine learning paper, the formulas often look far harder than they really are. Most of that difficulty comes from notation: a compact shorthand that experienced readers decode without thinking. Once you know the shorthand, a formula such as reads as an ordinary sentence: "to make a prediction, multiply each feature by its weight, add all of those products together, and then add the number ." The aim of this section is to give you that shorthand, so that later topics feel like reading rather than code-breaking.
The first thing to notice is that letters come in three "sizes". A plain lower-case letter such as , or usually stands for a single number, which mathematicians call a scalar; for example, could be the size of a room in some unit. A vector is an ordered list of numbers. Some books print it in bold (), but in machine learning it is very common to write it as an ordinary and let the context make it clear. A house could be described by the vector , meaning 120 square metres, 3 bedrooms and 15 years old. Finally, a capital letter such as usually means a matrix, which is a table of numbers. In machine learning, is almost always the whole dataset, with one row for each example and one column for each feature.
Small subscripts tell you which number you are talking about. If is the house vector above, then is its second entry, the number of bedrooms. When we have many houses, we need a second index to say which house; you will see this written either as or as , so means "the fifth house in the dataset". The brackets in are there to show that the is a label, not a power. Two further counting letters appear constantly: is usually the number of examples (for instance houses) and is the number of features (here ).
Decorations on top of a letter change its meaning in a consistent way. A hat, as in (read "y-hat"), means a prediction or estimate: if the real price is and our model says 410, we write . A star, as in , means the true or best possible value, the value we would love to know. A bar, as in , means an average, so .
A few more symbols describe sets and relationships. The symbol stands for all real numbers, and stands for all lists of real numbers, so writing (read "x is in R-three") simply says that is a list of three numbers. The upside-down A, , means "for all": the statement says that every weight is zero or positive. Finally, means "approximately equal to", means "is defined to be", and means "is proportional to", so says that for some constant that we don't care about at the moment.
The table below collects all of these symbols in one place so that you can come back to it whenever a formula looks unfamiliar.
| Symbol | How to read it | Example |
|---|---|---|
| a single number (scalar) | ||
| or | a vector (list of numbers) | |
| a matrix; usually the dataset | one row per example | |
| entry of a vector | ||
| or | example number | = the fifth house |
| prediction, true/best value, average | ||
| number of examples, number of features | ||
| "is in", lists of numbers, "for all" | ||
| approximately, defined as, proportional to |
Sums, products, min and argmin
Three operations appear in almost every formula in this course: adding things up, multiplying things together, and asking "which value is best?". Each has its own symbol.
The capital Greek letter sigma, , means add up. The expression under it says where a counter starts, the number on top says where it stops, and the formula to its right says what to add at each step. For example,
tells us to let run through 1, 2, 3 and 4, square each value, and add the results. Writing it out in full gives . Nothing more mysterious than that is going on.
Here is a sum you will meet constantly. Suppose is the true value for example and is the model's prediction. Then
is read from the inside out. First, for each example, compute the error . Next, square it, which makes every error positive and makes large errors count much more than small ones. Then add up these squared errors over all examples, and finally divide by to get an average. This quantity is the mean squared error, the standard way to measure how wrong a regression model is.
The capital Greek letter pi, , works in exactly the same way but means multiply together: . It appears whenever we combine the probabilities of independent events, because the probability that several independent things all happen is the product of their individual probabilities.
The last idea, the difference between min and argmin, confuses many students, so let's take it slowly with an example. Consider the function . The squared term can never be negative, and it equals zero exactly when . So the smallest value can ever take is , and it takes that value at . We describe these two facts separately:
The min is the lowest output, while the argmin is the input that produces it ("arg" is short for argument, which is another word for input). When we train a model we write , meaning "the weights are whichever weights make the loss as small as possible". We almost always care about the argmin, because the weights are the model; the loss value itself is just a score.
Finally, vertical bars mean absolute value, so , and double bars around a vector, as in , mean the length of that vector, which the next topic explains.
Functions, exponentials and logarithms
A function is best thought of as a machine: you feed in an input and it hands back exactly one output. The function takes the input 3 and returns . A machine learning model is simply a function with some adjustable knobs, called parameters. For example, is a whole family of straight lines, and training means turning the knobs and until the line fits the data well.
Two particular functions appear everywhere in machine learning. The first is the exponential function, , where is a special constant. Three facts about it are worth remembering. It is always positive, whatever is. It equals 1 when . And it grows extremely fast as increases, while shrinking towards zero (without ever reaching it) as becomes more negative. Because it always produces positive numbers, we use it whenever we need to turn arbitrary scores into something that can behave like a probability.
The second is the natural logarithm, written (some books write ). The logarithm undoes the exponential: . In words, answers the question "to what power must I raise to get ?". Since is always positive, the logarithm is only defined for positive inputs. Because , we have , and numbers between 0 and 1 have negative logarithms. This last fact matters, because probabilities live between 0 and 1, so their logarithms are negative.
Logarithms obey three rules that we will use again and again. The first is that the log of a product is the sum of the logs:
For example, . The second rule says the log of a ratio is a difference, ; the quantity , called the log-odds, is built this way. The third rule says that a power comes down in front: , so .
Why does machine learning use logarithms so much? There are three reasons. First, they turn products into sums. The probability of a thousand independent observations is a product of a thousand numbers, which is awkward to work with, but its logarithm is a sum of a thousand terms, which is easy to differentiate and to compute. Second, they avoid numerical underflow. A computer cannot represent a number as small as , so it silently rounds it to zero; but its logarithm, , is a perfectly ordinary number. Third, the logarithm is an increasing function, which means that whatever input makes largest also makes largest. So we can maximise instead of and get exactly the same answer.
We can now build one of the most important functions in the course, the sigmoid:
Let's evaluate it at a few points to see what it does. At we have , so . At , , so . At , , so . As becomes very large, shrinks towards zero and approaches 1; as becomes very negative, becomes huge and approaches 0. So the sigmoid takes any real number and squashes it smoothly into the range between 0 and 1, which is exactly what we need to turn a model's raw score into a probability.
Try it: explore eˣ, log x and the sigmoid
Slide and switch between the three functions. Check that log undoes exp, that log is negative between 0 and 1 and undefined below 0, and that the sigmoid never leaves the range (0, 1).
Interactive: this part runs in your browser.
Code: the notation in NumPy
Every symbol above has a one-line NumPy equivalent. Run this and compare with the worked examples.
import numpy as np
i = np.arange(1, 5) # [1, 2, 3, 4]
print("sum of i^2:", np.sum(i ** 2)) # 30
print("product:", np.prod([0.5, 0.2, 0.9])) # 0.09
w = np.linspace(-2, 8, 1001)
f = (w - 3) ** 2 + 1
print("min f =", f.min(), " argmin w =", w[f.argmin()]) # 1.0 and 3.0
print("log(0.5*0.25) =", np.log(0.5 * 0.25), "=", np.log(0.5) + np.log(0.25))
print("0.01**200 =", 0.01 ** 200, "(underflow!) but 200*log(0.01) =", 200 * np.log(0.01))
sigmoid = lambda z: 1 / (1 + np.exp(-z))
print("sigmoid(0, 2, -2):", sigmoid(np.array([0, 2, -2])).round(3))
# Mean squared error written exactly like the formula (1/n) * sum (y - y_hat)^2
y, y_hat = np.array([3.0, 5.0, 2.0]), np.array([2.5, 5.5, 2.0])
print("MSE:", np.mean((y - y_hat) ** 2))Practice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. equals…
2. For f(w) = (w − 3)² + 1, argmin f is…
3. log(a·b) equals…
4. σ(0) equals…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Mathematics for Machine Learning (Deisenroth, Faisal, Ong) ↗ free textbook covering linear algebra, calculus, probability and optimisation for ML