experiment-tracking-review · diff
git:20260602.1457735 to git:20260605.9de4340
38 added, 8 removed. Audit A to A.
---
name: experiment-tracking-review
- description: "Verify that all hyperparameters, metrics, and data references are properly logged."
+ description: Use when reviewing ML training code to confirm a run could be reconstructed later -- hyperparameters, metrics, data references, and artifacts logged, not just printed to stdout.
---
- # Experiment Tracking Review
+ ## Purpose
- An ML experiment is useless if you cannot reconstruct exactly how it was run and what data it used.
+ An ML experiment is useless if you cannot reconstruct exactly how it was run and what data it used. This review checks that everything needed for reconstruction is logged.
- ## Review Protocol
+ ## When to Use
- 1. **Hyperparameter Logging**: Ensure the script logs *every* hyperparameter (learning rate, batch size, architecture details). Hardcoded magic numbers in the script must be extracted to a config and logged.
- 2. **Metric Logging**: Verify that training and validation metrics are logged at each epoch or step, not just at the end.
- 3. **Artifact Saving**: Ensure the final model weights, preprocessing scalers/encoders, and the exact configuration file are saved together in a versioned directory or tracking system.
- 4. **Action**: Do not allow training scripts to print metrics to stdout only. Enforce structured logging (JSON, MLflow, wandb).
+ - Reviewing a training script before serious experiment cycles begin
+ - Results exist but nobody can say which configuration produced them
+ - Metrics are printed to stdout only
+
+ ## Inputs
+
+ - The training script(s) and any tracking/config setup (MLflow, wandb, config files)
+
+ ## Workflow
+
+ 1. **Hyperparameter logging**: the script logs *every* hyperparameter (learning rate, batch size, architecture details). Hardcoded magic numbers must be extracted to a config and logged.
+ 2. **Metric logging**: training and validation metrics are logged at each epoch or step, not just at the end.
+ 3. **Artifact saving**: final model weights, preprocessing scalers/encoders, and the exact configuration file are saved together in a versioned directory or tracking system.
+ 4. **Data reference**: the run records which data it used (path, version, or hash).
+ 5. **Enforce structured logging** (JSON, MLflow, wandb) — stdout-only metric reporting is a blocking finding.
+
+ ## Output
+
+ - A finding list of what is not logged or saved, with the specific fix for each gap
+
+ ## Verification
+
+ - [ ] Every hyperparameter logged (no unlogged magic numbers remain)
+ - [ ] Per-epoch/step train and validation metrics logged
+ - [ ] Weights + preprocessors + config saved together, versioned
+ - [ ] Data reference (path/version/hash) recorded with the run
+ - [ ] No stdout-only metric reporting remains
+
+ ## Failure Modes
+
+ - **End-only metrics** — a final score without curves hides divergence and the onset of overfitting.
+ - **Orphaned artifacts** — weights saved without the matching scaler/encoder and config cannot be served correctly later.
+ - **Config drift** — the logged config differs from what the script actually used; log the resolved config at runtime.
+