Probability & Distributions
The language of uncertainty, noise and predictions.
Random variables, expectation and variance
Machine learning is full of uncertainty: noisy measurements, unpredictable labels, randomly chosen training batches. Probability is the language we use to reason about it. Its central idea is the random variable, a quantity whose value we don't know in advance, such as the result of rolling a die. A random variable comes with a distribution, which tells us how likely each possible value is. Probabilities always lie between 0 and 1, and the probabilities of all the possible values add up to exactly 1.
The first summary of a distribution is its expectation, or mean, written . It is the average value you would see over many repetitions, computed as a probability-weighted average of the possible values:
For a fair die, each face has probability , so . Notice that the expected value doesn't have to be a value you can actually roll; it is a long-run average.
The second summary is the variance, which measures spread: how far values typically land from the mean. It is defined as the average squared distance from the mean, , where . For the die, the distances from 3.5 are , so
Because the variance is in squared units, we often take its square root to get the standard deviation, , which is in the same units as the original quantity.
A handful of rules let us work with expectations and variances without recalculating from scratch. Expectation is linear: , and for any two random variables. Variance behaves differently. Adding a constant shifts every value by the same amount and doesn't change the spread, but multiplying by stretches every distance by , and therefore the squared distances by : . When two random variables are independent, meaning one tells you nothing about the other, their variances add: .
Putting those rules together gives one of the most important facts in the whole course. Take independent measurements , each with variance , and average them. The sum has variance (the variances add), and dividing by multiplies the variance by , so
Averaging reduces noise, and it does so in a predictable way. This is the mathematical reason why more data gives more reliable estimates, and why ensembles that average many models are more stable than any single model.
Finally, the simplest distribution of all is the Bernoulli distribution, which describes a biased coin: it takes the value 1 with probability and 0 with probability . Its mean is and its variance is . Every yes/no label in machine learning (spam or not, fraud or not) is modelled as a Bernoulli random variable.
The Gaussian (normal) distribution
The most important continuous distribution is the Gaussian, also called the normal distribution or the bell curve. It has two parameters: the mean , which sets where the centre of the bell is, and the standard deviation , which sets how wide it is. We write . Its density is
This formula is easier to understand if you read it from the inside out. The term is the squared distance of from the centre. Dividing by measures that distance relative to the spread of the distribution, so being 2 units from the centre counts as "far" when is small but "close" when is large. Taking of the negative of this quantity turns small distances into values near 1 and large distances into values near 0, which produces the bell shape. The fraction in front is just a scaling constant that makes the total area under the curve equal to 1, as every probability distribution must.
A very practical rule of thumb follows from the shape of the bell: about 68% of values fall within one standard deviation of the mean (between and ), about 95% fall within two, and about 99.7% fall within three. So a value more than three standard deviations from the mean is genuinely unusual.
Any Gaussian can be converted into the standard normal by subtracting its mean and dividing by its standard deviation, . The result, called a z-score, says how many standard deviations a value lies from the mean. This is exactly what scikit-learn's StandardScaler does to each feature.
Why is the Gaussian so common? The central limit theorem says that when a quantity is the sum or average of many small, independent random effects, its distribution is approximately Gaussian, whatever the individual effects look like. Measurement noise is usually the combined result of many small disturbances, so it is natural to model it as Gaussian. As you'll see in the likelihood topic, that single modelling assumption leads directly to the least-squares method of linear regression.
One subtle point is worth stating clearly. For a continuous variable, the height of the curve is a density, not a probability. The probability of any single exact value is zero; probabilities come from areas under the curve over a range of values. Because of this, a density can be larger than 1 (a very narrow Gaussian has a very tall peak) without breaking any rules.
Interactive: this part runs in your browser.
Try it: samples vs the bell curve
Draw samples and compare the histogram with the curve. With 20 samples the histogram is rough; with 1000 it matches the curve, and about 68% land within ±1σ.
Interactive: this part runs in your browser.
Conditional probability and Bayes' theorem
Often we want the probability of something given that we already know something else. The conditional probability of given is written and defined as
where is the probability that both happen. The idea is that once we know has happened, we restrict our attention to the situations where is true, and ask what fraction of those also have .
Rearranging the definition gives the product rule, . A second basic rule, the sum rule, says that we can recover the probability of alone by adding up the joint probabilities over every possible value of : . Two events are independent when , which is the same as saying that knowing doesn't change the probability of .
The product rule can be written two ways, because "A and B" is the same event as "B and A": . Dividing both sides by gives Bayes' theorem:
Bayes' theorem tells us how to update a belief when we see new evidence. is the prior, what we believed before seeing the evidence. is the likelihood, how probable the evidence is if is true. is the overall probability of the evidence, and is the posterior, our updated belief.
A classic example shows why this matters. Suppose a disease affects 1% of people. A test for it correctly detects the disease 90% of the time, but it also wrongly flags 5% of healthy people. You test positive. How likely is it that you actually have the disease? Many people guess about 90%, but let's calculate. First, we need the overall probability of a positive test. A positive can come from an ill person who is correctly flagged, or from a healthy person who is wrongly flagged:
Now Bayes' theorem gives the probability of being ill given a positive test:
The answer is only about 15%. The reason is that the disease is rare: the healthy group is so much larger than the ill group that even a small false-positive rate among the healthy produces far more positive tests than the disease itself does. This is exactly the same effect that makes precision low when a classifier looks for rare events. Bayes' theorem will reappear when we combine data with prior beliefs in MAP estimation.
Interactive: this part runs in your browser.
Try it: Bayes' theorem with 1000 people
Each dot is a person. Adjust how common the disease is and how good the test is, and count how many of the positive tests are real.
Interactive: this part runs in your browser.
Code: simulate to check the maths
Simulation is a great way to build intuition and to check a calculation.
import numpy as np
rng = np.random.default_rng(0)
rolls = rng.integers(1, 7, size=100_000)
print("die mean:", rolls.mean().round(3), "(theory 3.5) variance:", rolls.var().round(3), "(theory 2.917)")
# Averaging reduces variance: Var(mean of n) = sigma^2 / n
for n in [1, 10, 100]:
means = rng.integers(1, 7, size=(20_000, n)).mean(axis=1)
print(f"n={n:3d}: variance of the average = {means.var():.4f} theory {35/12/n:.4f}")
# Gaussian: 68-95-99.7 rule
z = rng.normal(0, 1, 100_000)
print("within 1, 2, 3 sigma:", [round(float(np.mean(np.abs(z) < k)), 3) for k in (1, 2, 3)])
# Bayes' theorem by simulation (medical test)
N = 1_000_000
ill = rng.random(N) < 0.01
positive = np.where(ill, rng.random(N) < 0.90, rng.random(N) < 0.05)
print("P(ill | positive) simulated:", round(ill[positive].mean(), 3), " formula: 0.154")Practice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press ✔ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. E[2X + 3] when E[X] = 5 is…
2. Var(3X) equals…
3. The variance of the average of n independent samples (each variance σ²) is…
4. In the medical-test example, why is P(ill | +) so low?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Khan Academy: Statistics and probability ↗ Free lessons and practice
- Mathematics for Machine Learning (Deisenroth, Faisal, Ong) ↗ free textbook covering linear algebra, calculus, probability and optimisation for ML