experiment-reproducibility git:20260921.c8f5dc3A

Immutable. This exact content is served forever at /api/v1/blob/19f966538c221eb9.

---
name: experiment-reproducibility
description: "Make a thesis's computational experiments reproducible by the examiner and by the student three months later: pinned environments, seeded randomness, a single config per run, a run manifest that records code version, data version, parameters, hardware, and timings, results written to versioned files that figures and tables are generated from, and a one-command rerun. Use when setting up experiments, before the first real run, or when a result cannot be regenerated."
argument-hint: "(optional) the experiment directory, the framework, and where results should go"
---

A result that cannot be regenerated from a command is an anecdote. Reproducibility is cheap at the start and impossible at the end, so this is set up before the first run that will appear in the thesis.

## Layout

```
experiments/
  env/          environment.yml or requirements.txt with exact pins; Dockerfile if system deps matter
  configs/      one YAML per experiment: exp01-baseline.yaml, exp02-ours.yaml
  data/         raw/ (never modified), processed/ (generated by scripts), DATA.md (source, licence, version, checksum)
  src/          the code; run.py takes --config and --seed
  runs/         one directory per run: <exp>-<seed>-<timestamp>/ with manifest.json, metrics.json, log.txt
  results/      aggregated tables (CSV) generated from runs/ by aggregate.py
  figures/      generated from results/ by figure scripts (research-figures)
  Makefile      or justfile: `make exp01` runs all seeds; `make results` aggregates; `make figures`
```

Everything under `runs/`, `results/`, and `figures/` is generated; regenerate rather than edit. Commit `results/` and `figures/` (they are small) so the thesis builds from a clean clone; do not commit `runs/` unless small (use DVC or a release asset otherwise).

## Environment

- Python: `conda env export --no-builds` or `pip freeze` into `env/`, with the Python version. Better: `pyproject.toml` with a lockfile (`uv lock`, `poetry lock`).
- Record the CUDA/driver version and the exact framework build when GPUs are involved; results differ across them.
- A `Dockerfile` when anything outside the language environment matters (system libraries, a database).
- The manifest records `python --version`, `pip freeze` hash, `nvidia-smi` output, and `uname -a` at run time regardless.

## Determinism

- One `--seed` argument that seeds every source: `random`, `numpy`, `torch` (and `torch.cuda.manual_seed_all`), `tf`, data shuffling, and worker init. Use a helper `seed_everything(seed)` called first.
- Set deterministic flags where the framework offers them (`torch.use_deterministic_algorithms(True)`, `CUBLAS_WORKSPACE_CONFIG`) and record whether they were on; some ops stay nondeterministic on GPU and the thesis should say so.
- Run every experiment with at least three seeds (five or ten when cheap); report mean and SD or CI (`statistics-advisor`). A single-seed number is not a result.

## Configuration

- All parameters in the config file; nothing hard-coded, nothing from the command line except `--config` and `--seed` (and `--out`). The config is copied into the run directory unchanged.
- Named experiments, not edited-in-place configs: `exp03-ours-lr0.001.yaml` rather than changing `exp02` and losing what produced the earlier result.

## The run manifest

`runs/<run>/manifest.json`, written at start and completed at end:

```json
{
  "experiment": "exp02-ours", "seed": 1, "started": "2026-09-21T10:00:00Z", "finished": "...",
  "git": { "commit": "a1b2c3d", "dirty": false, "branch": "main" },
  "config_sha256": "...", "data_version": "v3 (sha256 ...)",
  "env": { "python": "3.12.4", "packages_sha256": "...", "cuda": "12.4", "gpu": "A100 40GB", "host": "..." },
  "duration_s": 1832.4, "status": "ok"
}
```

Refuse to start a run that will appear in the thesis from a dirty working tree (`git status --porcelain` non-empty): commit first, so the commit hash means something. Log the refusal.

## Data

`data/DATA.md`: source URL, download date, licence, version, checksum of the raw archive, the preprocessing script and its parameters, the resulting row counts and split sizes. Splits are made once with a seed and saved as index files; every run reads the same splits.

## Aggregation and figures

`aggregate.py` reads every `runs/*/metrics.json`, joins with the manifests, and writes `results/<table>.csv` with one row per (experiment, seed) and a summary with mean, SD, n. Figure scripts read only `results/`. The thesis `\input`s tables generated from `results/` (`benchmark-reporting`), so a rerun updates the document.

## One command

`make all` (or `just all`): environment check, all experiments for all seeds, aggregate, figures, tables. Document it in `experiments/README.md` with the expected total runtime and hardware. The examiner, or a future student, runs it and gets the thesis's numbers.

## Checklist before a result enters the thesis

- Run from a clean commit, manifest present, seeds ≥ 3.
- Config committed, data version recorded, environment pinned.
- Table and figure generated from `results/`, not typed.
- The commit hash cited in the thesis appendix.