training-pipeline-debugging · diff
git:20260502.647df10 to git:20260605.2671f86
41 added, 10 removed. Audit A to A.
---
name: training-pipeline-debugging
- description: "Diagnose NaN losses, out-of-memory errors, and shape mismatches in deep learning or ML pipelines."
+ description: Use when an ML training run fails or misbehaves -- NaN losses, out-of-memory errors, shape mismatches, or a model that cannot fit even a single batch.
---
- # Training Pipeline Debugging
+ ## Purpose
- ML training bugs are often silent mathematical errors rather than explicit code crashes.
+ ML training bugs are often silent mathematical errors rather than explicit code crashes. This protocol localizes the standard failure classes quickly.
- ## Debugging Protocol
+ ## When to Use
- 1. **NaN Losses**: If loss goes to NaN, check:
+ - Loss becomes NaN or diverges
+ - Training crashes with out-of-memory errors
+ - Shape/dimension errors at matrix multiplications or loss computation
+ - The pipeline runs but the model never learns
+
+ ## Inputs
+
+ - The failing training script and its error output or loss curve
+
+ ## Workflow
+
+ 1. **Run the overfit test first**: fit a single batch of ~10 examples. If the model cannot achieve near-zero loss on a single batch, the pipeline is fundamentally broken — do not debug full runs until the single-batch test passes.
+ 2. **NaN losses** — check:
- Learning rate too high?
- Missing data (NaNs in input)?
- - Log/Exp/Divide by zero in custom loss functions?
+ - Log/Exp/Divide-by-zero in custom loss functions?
- Exploding gradients (clip gradients)?
- 2. **OOM (Out of Memory)**:
+ 3. **OOM (out of memory)**:
- Reduce batch size.
- Check for memory leaks in the training loop (e.g., accumulating history across epochs without `.detach()`).
- 3. **Shape Mismatches**:
- - Add temporary print statements or assertions asserting `tensor.shape` before matrix multiplications or loss calculations.
- 4. **The Overfit Test**: The ultimate test of a pipeline is fitting a single batch. If the model cannot achieve near 0 loss on a single batch of 10 examples, the pipeline is fundamentally broken. Do not debug full runs until the single-batch test passes.
+ 4. **Shape mismatches**:
+ - Add temporary print statements or assertions on `tensor.shape` before matrix multiplications and loss calculations.
+
+ ## Output
+
+ - The identified failure class and its specific fix, or a single-batch reproduction showing exactly where the pipeline breaks
+
+ ## Verification
+
+ - [ ] Single-batch overfit test run before any full-run debugging
+ - [ ] For NaN: learning rate, input NaNs, unsafe math, and gradient explosion all checked
+ - [ ] For OOM: batch size and training-loop accumulation both checked
+ - [ ] For shapes: assertions placed before matmul/loss sites
+ - [ ] Fix validated by re-running the previously failing case
+
+ ## Failure Modes
+
+ - **Debugging full runs first** — hours per iteration; the single-batch test gives answers in minutes.
+ - **Treating NaN as random** — NaN losses have a small set of causes; check all four systematically instead of restarting with a lower LR.
+ - **Batch-size-only OOM fixes** — shrinking the batch hides a loop leak that will OOM again later; check accumulation too.
+ - **Leftover instrumentation** — remove temporary shape prints/assertions once the fix is validated.
+