Becoming an AI Researcher
Topic 7 · Part 2 · Ideas & experiments · ⏱ about 5 h

Designing Experiments That Convince

Baselines that are strong and fairly tuned, ablations, controls, datasets and splits, choosing metrics, budgeting compute, and planning analyses before you see results.

Show:
In this topic
Beginner

Strong, fair baselines

A method is only as convincing as what it beats. Include: a trivial baseline (majority class, a simple heuristic), the standard approach practitioners actually use, and the strongest recent methods for the setting. Tune every baseline with the same budget (same number of hyperparameter trials, same validation data) as your method. Under-tuned baselines are one of the most common reasons published gains disappear when others try to reproduce them.

Use the original authors' code or numbers where possible, and say so. If you re-run a baseline and get a different number from the paper, report both and explain.

Beginner

Ablations and controls

An ablation removes or replaces one component at a time to show what each contributes: "full method", "without component A", "without B", "A replaced by a simpler alternative". If removing a component does not hurt, it should not be in the method, or your explanation of why the method works is wrong.

Controls rule out alternative explanations: if your method adds parameters, compare with a baseline given the same parameter count; if it trains longer, give the baseline the same compute; if it uses extra data, give the baseline access to it too. The goal is that the only difference between conditions is the thing you claim matters.

Beginner

Data, splits and metrics

  • Splits: train / validation / test, with the test set used once, at the end. Split by the unit that must generalise (patient, user, document source, time). Check for duplicates and near-duplicates across splits.
  • Contamination: for pretrained models, ask whether the test data could be in the pretraining corpus; prefer recent or private test sets, and say what you checked.
  • Several datasets: one dataset is an anecdote. Two or three diverse datasets make a general claim plausible.
  • Metrics: choose before running, aligned with the question (macro-F1 for imbalanced classes, calibration if probabilities matter, cost and latency for efficiency claims). Report the standard metrics of the area too, so readers can compare.
Beginner

Plan before you look: the analysis plan

Write a one-page experiment plan before the main runs: research question and hypothesis; datasets and splits; methods and baselines with their tuning budgets; metrics; number of seeds; the statistical comparison you will make; and what result would count as support or refutation. This protects you from the most common self-deception in ML: trying many variants and reporting the one that happened to look best (p-hacking or "garden of forking paths"). Exploratory experiments are fine, but label them as exploratory and confirm findings on fresh runs or data.

Beginner

Budget your compute

Estimate GPU hours before starting: runs per condition × seeds × conditions × hours per run, plus tuning. A plan of 4 methods × 3 datasets × 5 seeds × 2 hours is 120 GPU hours before tuning: feasible on a university cluster, painful on free Colab. Shrink the plan by using smaller models (often enough to answer the question), fewer but well-chosen conditions, and early small-scale pilots. Free and low-cost options include university HPC, Kaggle and Colab, cloud research credits programmes, and national research computing schemes.

Try it

Code example: ablation table and LaTeX

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Check yourself

Quiz

1. Your method has 10% more parameters than the baseline and wins by 1 point. A fair control is…

2. What does an ablation show?

3. How often should the test set be used?

4. Trying 40 variants and reporting the best one without saying so is…

5. 4 methods × 3 datasets × 5 seeds × 2 GPU-hours each is…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready