Statistics for Decisions: Intervals, Tests & A/B
Sampling variation, confidence intervals, hypothesis tests, p-values, power and sample size, and how to run an A/B test without fooling yourself.
Why ML engineers need statistics
Every number you report is computed from a sample: this month's patients, this test set, these 2,000 users in an experiment. A different sample would give a slightly different number. Statistics is the toolkit for answering "how much would this number move by chance?" and therefore "is this difference real?".
You need it whenever you compare models (is 0.83 really better than 0.82?), report a metric (accuracy 0.91, give or take how much?), or run an A/B test to check that a model actually changes behaviour in the real world. Interviewers for ML roles ask about p-values, confidence intervals and A/B tests often, and frequently find that candidates mix them up.
Watch it: A/B tests: why peeking at the p-value fools you
A short narrated animation. Press โถ Play, or step through it with โฎ โญ. Tick ๐ narrate to hear each step.
Interactive: this part runs in your browser.
Sampling variation and the standard error
Suppose the true no-show rate is 14%. If you look at 100 appointments, you might see 10%, or 19%. With 10,000 appointments you will see something very close to 14%. The spread of an estimate across repeated samples is its standard error (SE). For a proportion from observations, and for a mean of values with standard deviation :
The is the key fact: to halve the uncertainty you need four times as much data. By the central limit theorem, averages of many independent observations are approximately normally distributed, which is why the normal distribution appears in almost every formula below.
Confidence intervals
A 95% confidence interval is estimate ยฑ 1.96 ร SE. With 168 no-shows in 1,200 appointments, , SE = , so the interval is roughly 0.12 to 0.16.
The correct interpretation is about the method: if we repeated the sampling many times, 95% of the intervals built this way would contain the true value. In practice, read it as "the plausible range for the true value". An interval is much more informative than a single number: "accuracy 0.91 (95% CI 0.88โ0.94)" tells the reader how much to trust it.
When there is no simple formula (a median, an F1 score, an AUC), use the bootstrap: resample your data with replacement many times, recompute the statistic each time, and take the 2.5th and 97.5th percentiles of the results.
Hypothesis tests and p-values
A hypothesis test starts from a sceptical assumption, the null hypothesis : "there is no difference". The p-value is the probability of seeing a difference at least as large as the one observed if were true. A small p-value (conventionally below 0.05) means the data would be surprising under "no effect", so we reject .
For two proportions (control , treatment ) the standard z-test uses the pooled rate :
What a p-value is not: it is not the probability that the null hypothesis is true, not the probability the result is due to chance, and not a measure of how big or important the effect is. A tiny, useless effect can have p = 0.0001 with enough data; a large, important effect can have p = 0.2 with too little. Always report the effect size with its confidence interval.
Two error types: a false positive (type I: declaring an effect that isn't there; its rate is the significance level ฮฑ, usually 5%) and a false negative (type II: missing a real effect). Power is the probability of detecting a real effect of a given size; experiments are usually designed for 80% power.
Try it: run a thousand A/B tests
Set the true effect, the sample size and whether you "peek" at the results every day and stop as soon as p < 0.05. The simulator runs 1,000 experiments and shows how often each one declares a winner.
Interactive: this part runs in your browser.
Sample size and A/B testing in practice
Decide the sample size before the experiment. For two proportions, a standard approximation for the number per group is
with for ฮฑ = 5% and for 80% power. Detecting a drop from 14% to 12% needs about 4,400 per group; from 14% to 10% about 1,000. Small effects need a lot of data.
Rules that keep A/B tests honest: randomise who gets which version (by patient, not by appointment, if one person has many appointments); choose one primary metric and the stopping rule in advance; don't peek and stop early (it inflates false positives, as the simulator shows); check guardrail metrics (complaints, workload) for harm; and beware many comparisons: testing 20 metrics at 5% will find one "significant" result by chance.
Code example: was the reminder call a real effect?
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Comparing models statistically
When two models are evaluated on the same test set, their errors are paired, so use paired methods: McNemar's test for two classifiers (it looks only at cases where they disagree), or a paired bootstrap of the metric difference. Cross-validation scores from the same folds are also paired: compare fold-by-fold differences, and remember folds overlap, so standard errors from CV are optimistic.
For metrics with small numbers of positives (rare events), intervals are wide: a recall of 0.80 measured on 25 positives has a 95% interval of roughly 0.6 to 0.9. Report it. Many "improvements" in ML papers and dashboards are within noise.
Code: tests and intervals in Python
The same calculations with SciPy and statsmodels, plus a bootstrap.
import numpy as np
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
count, nobs = np.array([133, 168]), np.array([1180, 1200]) # treatment, control
z, p = proportions_ztest(count, nobs)
lo, hi = confint_proportions_2indep(133, 1180, 168, 1200) # CI for p_treat - p_control
print(f"z={z:.2f} p={p:.4f} CI=({lo:.4f}, {hi:.4f})")
# sample size per group to detect 14% -> 12% with 80% power
h = proportion_effectsize(0.14, 0.12)
print(round(NormalIndPower().solve_power(h, alpha=0.05, power=0.8)))
# bootstrap CI for a difference in means (works for any statistic)
rng = np.random.default_rng(0)
a = rng.binomial(1, 0.14, 1200); b = rng.binomial(1, 0.11, 1180)
diffs = [rng.choice(b, len(b)).mean() - rng.choice(a, len(a)).mean() for _ in range(5000)]
print(np.percentile(diffs, [2.5, 97.5]))Practice: check yourself
Work each question out (on paper, in your head, or in Python), type the answer and press โ Check. Fractions like 1/4 are fine. Hints and a worked answer are there when you need them.
Quiz
1. To halve the standard error of an estimate you needโฆ
2. p = 0.03 in an A/B test meansโฆ
3. Peeking every day and stopping when p < 0.05โฆ
4. A 95% CI for a difference is (โ0.01, +0.03). The result isโฆ
5. Which statistic has no simple SE formula and is best given a bootstrap CI?
6. You test 20 metrics at ฮฑ = 0.05 with no real effects. You expect aboutโฆ
๐ Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- OpenIntro Statistics (free book) โ Introductory statistics textbook
- Khan Academy: Statistics and probability โ Free lessons and practice