Maths for AI
Topic 14 ยท Probability & statistics ยท โฑ about 6 h

Statistics for Decisions: Intervals, Tests & A/B

Sampling variation, confidence intervals, hypothesis tests, p-values, power and sample size, and how to run an A/B test without fooling yourself.

Show:
In this topic
Beginner

Why ML engineers need statistics

Every number you report is computed from a sample: this month's patients, this test set, these 2,000 users in an experiment. A different sample would give a slightly different number. Statistics is the toolkit for answering "how much would this number move by chance?" and therefore "is this difference real?".

You need it whenever you compare models (is 0.83 really better than 0.82?), report a metric (accuracy 0.91, give or take how much?), or run an A/B test to check that a model actually changes behaviour in the real world. Interviewers for ML roles ask about p-values, confidence intervals and A/B tests often, and frequently find that candidates mix them up.

Beginner

Watch it: A/B tests: why peeking at the p-value fools you

A short narrated animation. Press โ–ถ Play, or step through it with โฎ โญ. Tick ๐Ÿ”Š narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

Sampling variation and the standard error

Suppose the true no-show rate is 14%. If you look at 100 appointments, you might see 10%, or 19%. With 10,000 appointments you will see something very close to 14%. The spread of an estimate across repeated samples is its standard error (SE). For a proportion p^\hat p from nn observations, and for a mean of values with standard deviation ss:

SE(p^)=p^(1โˆ’p^)n,SE(xห‰)=sn.\text{SE}(\hat p)=\sqrt{\frac{\hat p(1-\hat p)}{n}},\qquad \text{SE}(\bar x)=\frac{s}{\sqrt n}.

The n\sqrt n is the key fact: to halve the uncertainty you need four times as much data. By the central limit theorem, averages of many independent observations are approximately normally distributed, which is why the normal distribution appears in almost every formula below.

Beginner

Confidence intervals

A 95% confidence interval is estimate ยฑ 1.96 ร— SE. With 168 no-shows in 1,200 appointments, p^=0.14\hat p = 0.14, SE = 0.14ร—0.86/1200โ‰ˆ0.010\sqrt{0.14\times0.86/1200}\approx 0.010, so the interval is roughly 0.12 to 0.16.

The correct interpretation is about the method: if we repeated the sampling many times, 95% of the intervals built this way would contain the true value. In practice, read it as "the plausible range for the true value". An interval is much more informative than a single number: "accuracy 0.91 (95% CI 0.88โ€“0.94)" tells the reader how much to trust it.

When there is no simple formula (a median, an F1 score, an AUC), use the bootstrap: resample your data with replacement many times, recompute the statistic each time, and take the 2.5th and 97.5th percentiles of the results.

Beginner

Hypothesis tests and p-values

A hypothesis test starts from a sceptical assumption, the null hypothesis H0H_0: "there is no difference". The p-value is the probability of seeing a difference at least as large as the one observed if H0H_0 were true. A small p-value (conventionally below 0.05) means the data would be surprising under "no effect", so we reject H0H_0.

For two proportions (control x1/n1x_1/n_1, treatment x2/n2x_2/n_2) the standard z-test uses the pooled rate p^=(x1+x2)/(n1+n2)\hat p = (x_1+x_2)/(n_1+n_2):

z=p^2โˆ’p^1p^(1โˆ’p^)(1n1+1n2),p-value=2โ€‰(1โˆ’ฮฆ(โˆฃzโˆฃ)).z=\frac{\hat p_2-\hat p_1}{\sqrt{\hat p(1-\hat p)\left(\frac1{n_1}+\frac1{n_2}\right)}},\qquad p\text{-value}=2\,\big(1-\Phi(|z|)\big).

What a p-value is not: it is not the probability that the null hypothesis is true, not the probability the result is due to chance, and not a measure of how big or important the effect is. A tiny, useless effect can have p = 0.0001 with enough data; a large, important effect can have p = 0.2 with too little. Always report the effect size with its confidence interval.

Two error types: a false positive (type I: declaring an effect that isn't there; its rate is the significance level ฮฑ, usually 5%) and a false negative (type II: missing a real effect). Power is the probability of detecting a real effect of a given size; experiments are usually designed for 80% power.

Try it

Try it: run a thousand A/B tests

Set the true effect, the sample size and whether you "peek" at the results every day and stop as soon as p < 0.05. The simulator runs 1,000 experiments and shows how often each one declares a winner.

Interactive: this part runs in your browser.

Beginner

Sample size and A/B testing in practice

Decide the sample size before the experiment. For two proportions, a standard approximation for the number per group is

nโ‰ˆ(z1โˆ’ฮฑ/2+z1โˆ’ฮฒ)2โ€‰[p1(1โˆ’p1)+p2(1โˆ’p2)](p1โˆ’p2)2,n \approx \frac{(z_{1-\alpha/2}+z_{1-\beta})^2\,\big[p_1(1-p_1)+p_2(1-p_2)\big]}{(p_1-p_2)^2},

with z0.975=1.96z_{0.975}=1.96 for ฮฑ = 5% and z0.8=0.84z_{0.8}=0.84 for 80% power. Detecting a drop from 14% to 12% needs about 4,400 per group; from 14% to 10% about 1,000. Small effects need a lot of data.

Rules that keep A/B tests honest: randomise who gets which version (by patient, not by appointment, if one person has many appointments); choose one primary metric and the stopping rule in advance; don't peek and stop early (it inflates false positives, as the simulator shows); check guardrail metrics (complaints, workload) for harm; and beware many comparisons: testing 20 metrics at 5% will find one "significant" result by chance.

Try it

Code example: was the reminder call a real effect?

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Advanced

Comparing models statistically

When two models are evaluated on the same test set, their errors are paired, so use paired methods: McNemar's test for two classifiers (it looks only at cases where they disagree), or a paired bootstrap of the metric difference. Cross-validation scores from the same folds are also paired: compare fold-by-fold differences, and remember folds overlap, so standard errors from CV are optimistic.

For metrics with small numbers of positives (rare events), intervals are wide: a recall of 0.80 measured on 25 positives has a 95% interval of roughly 0.6 to 0.9. Report it. Many "improvements" in ML papers and dashboards are within noise.

Code

Code: tests and intervals in Python

The same calculations with SciPy and statsmodels, plus a bootstrap.

Python ยท scipy, statsmodels, numpy
import numpy as np
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

count, nobs = np.array([133, 168]), np.array([1180, 1200])        # treatment, control
z, p = proportions_ztest(count, nobs)
lo, hi = confint_proportions_2indep(133, 1180, 168, 1200)          # CI for p_treat - p_control
print(f"z={z:.2f} p={p:.4f} CI=({lo:.4f}, {hi:.4f})")

# sample size per group to detect 14% -> 12% with 80% power
h = proportion_effectsize(0.14, 0.12)
print(round(NormalIndPower().solve_power(h, alpha=0.05, power=0.8)))

# bootstrap CI for a difference in means (works for any statistic)
rng = np.random.default_rng(0)
a = rng.binomial(1, 0.14, 1200); b = rng.binomial(1, 0.11, 1180)
diffs = [rng.choice(b, len(b)).mean() - rng.choice(a, len(a)).mean() for _ in range(5000)]
print(np.percentile(diffs, [2.5, 97.5]))
Open in playground โ†—
Beginner

Practice: check yourself

Work each question out (on paper, in your head, or in Python), type the answer and press โœ” Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.

Question 1.
You flip a coin 10 times and get 7 heads. What is the maximum likelihood estimate of P(heads)?
Question 2.
The log-likelihood of 3 independent events with probabilities 0.5, 0.25, 0.8 is lnโก0.5+lnโก0.25+lnโก0.8\ln 0.5 + \ln 0.25 + \ln 0.8. Compute it to 3 decimal places.
Check yourself

Quiz

1. To halve the standard error of an estimate you needโ€ฆ

2. p = 0.03 in an A/B test meansโ€ฆ

3. Peeking every day and stopping when p < 0.05โ€ฆ

4. A 95% CI for a difference is (โˆ’0.01, +0.03). The result isโ€ฆ

5. Which statistic has no simple SE formula and is best given a bootstrap CI?

6. You test 20 metrics at ฮฑ = 0.05 with no real effects. You expect aboutโ€ฆ

Go deeper

๐Ÿ“š Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready