Becoming an AI Researcher
Topic 9 · Part 2 · Ideas & experiments · ⏱ about 6 h

Reproducibility, Research Engineering & Ethics

Experiment tracking, configs and seeds, versioning data and code, releasing code and models, documentation (model cards, datasheets), compute reporting, and research ethics and approvals.

Show:
In this topic
Beginner

Research code that you (and others) can trust

  • Version control: every experiment runs from a committed version of the code; record the commit hash with the results.
  • Configs, not edits: hyperparameters in config files (YAML, Hydra), never changed by hand inside scripts. One command reproduces any run.
  • Seeds and determinism: set seeds for Python, NumPy and PyTorch; note remaining non-determinism.
  • Experiment tracking (Weights & Biases, MLflow, TensorBoard, or a careful spreadsheet): config, data version, metrics, curves and outputs for every run, including failed ones.
  • Data versioning: checksums or DVC for datasets and splits; never overwrite raw data.
  • Tests for data processing and metrics: a bug in an evaluation script can invalidate a whole paper.
  • Environment: requirements file or container; record library versions and hardware.
Beginner

Releasing your work

Release code (GitHub with a clear README: setup, data download, one command per main result, expected numbers), and where appropriate trained models (Hugging Face Hub) and data. Choose licences deliberately (MIT/Apache for code; check data licences before redistributing anything). Many venues have a reproducibility checklist (NeurIPS's paper checklist, ACL's responsible NLP checklist): fill it in honestly, early. A released, working repository makes your paper more cited and your application more credible: supervisors do look at applicants' code.

Beginner

Documentation: model cards and datasheets

Model cards describe intended use, training data, evaluation (overall and by subgroup), limitations and ethical considerations. Datasheets for datasets describe motivation, collection process, consent, preprocessing, recommended uses and maintenance. Even for a small paper, a short version in the appendix answers most reviewer questions before they are asked.

Beginner

Research ethics and approvals

  • Human participants (surveys, user studies, annotation by people, interviews) usually need approval from your university's human research ethics committee before data collection. Health data has additional rules. Start early: approvals take weeks.
  • Data rights: check licences and terms of use; scraped data may not be redistributable; personal information needs a lawful basis and de-identification.
  • Annotators: fair pay, clear instructions, protection from harmful content.
  • Harms and dual use: could the work enable surveillance, discrimination or misuse? Many venues require a broader-impact or ethics statement.
  • Environmental cost: report compute used (GPU type and hours).
Try it

Try it: is this paper reproducible?

Score a (fictional) paper's reproducibility package against a checklist used by major venues.

Interactive: this part runs in your browser.

Try it

Code example: find leakage before reviewers do

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Check yourself

Quiz

1. Which practice lets anyone reproduce a specific result months later?

2. You plan to ask 30 nurses to rate model outputs. Before starting you need…

3. A model card should include…

4. Why test your evaluation script?

5. Scraped web data in your experiments can be redistributed…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready