Reproducibility, Research Engineering & Ethics
Experiment tracking, configs and seeds, versioning data and code, releasing code and models, documentation (model cards, datasheets), compute reporting, and research ethics and approvals.
Research code that you (and others) can trust
- Version control: every experiment runs from a committed version of the code; record the commit hash with the results.
- Configs, not edits: hyperparameters in config files (YAML, Hydra), never changed by hand inside scripts. One command reproduces any run.
- Seeds and determinism: set seeds for Python, NumPy and PyTorch; note remaining non-determinism.
- Experiment tracking (Weights & Biases, MLflow, TensorBoard, or a careful spreadsheet): config, data version, metrics, curves and outputs for every run, including failed ones.
- Data versioning: checksums or DVC for datasets and splits; never overwrite raw data.
- Tests for data processing and metrics: a bug in an evaluation script can invalidate a whole paper.
- Environment: requirements file or container; record library versions and hardware.
Releasing your work
Release code (GitHub with a clear README: setup, data download, one command per main result, expected numbers), and where appropriate trained models (Hugging Face Hub) and data. Choose licences deliberately (MIT/Apache for code; check data licences before redistributing anything). Many venues have a reproducibility checklist (NeurIPS's paper checklist, ACL's responsible NLP checklist): fill it in honestly, early. A released, working repository makes your paper more cited and your application more credible: supervisors do look at applicants' code.
Documentation: model cards and datasheets
Model cards describe intended use, training data, evaluation (overall and by subgroup), limitations and ethical considerations. Datasheets for datasets describe motivation, collection process, consent, preprocessing, recommended uses and maintenance. Even for a small paper, a short version in the appendix answers most reviewer questions before they are asked.
Research ethics and approvals
- Human participants (surveys, user studies, annotation by people, interviews) usually need approval from your university's human research ethics committee before data collection. Health data has additional rules. Start early: approvals take weeks.
- Data rights: check licences and terms of use; scraped data may not be redistributable; personal information needs a lawful basis and de-identification.
- Annotators: fair pay, clear instructions, protection from harmful content.
- Harms and dual use: could the work enable surveillance, discrimination or misuse? Many venues require a broader-impact or ethics statement.
- Environmental cost: report compute used (GPU type and hours).
Try it: is this paper reproducible?
Score a (fictional) paper's reproducibility package against a checklist used by major venues.
Interactive: this part runs in your browser.
Code example: find leakage before reviewers do
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Quiz
1. Which practice lets anyone reproduce a specific result months later?
2. You plan to ask 30 nurses to rate model outputs. Before starting you need…
3. A model card should include…
4. Why test your evaluation script?
5. Scraped web data in your experiments can be redistributed…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Kapoor & Narayanan, Leakage and the Reproducibility Crisis in ML-based Science ↗ a taxonomy of leakage
- Pineau, Machine Learning Reproducibility Checklist ↗ the checklist many venues adopted
- NeurIPS Paper Checklist ↗ what reviewers expect you to report