Investigate training failures, NaNs, missing gradients, misleading losses, and non-reproducible runs in PyTorch, Lightning, or TensorFlow/Keras. Use for concrete training bugs, not open-ended architecture or hyperparameter search.
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
# Debug a training run
Turn the reported symptom into an observable failure on a bounded input. Preserve the user's framework, backend, and intended training semantics. A CPU or eager-mode reproduction can isolate a bug, but does not establish what happens on the original accelerator, compiled graph, or distributed setup.
## Establish the failing boundary
Record the command, revision, Python/framework versions, imported package paths, device, dtype, precision, and execution mode. For resume or reproducibility issues, include data order, seed handling, checkpoint, optimizer, scheduler, and scaler state. Classify dependency failures separately from training bugs.
Reduce to a fixed batch while retaining shapes, label semantics, masks, and value ranges. Inspect tensors entering the loss: class indices, probability targets, and scalar targets have distinct contracts. Avoid blanket reshapes or squeezes that mask errors.
Example: scalar predictions of shape `[N, 1]` minus labels `[N]` produce `[N, N]` residuals via broadcasting. Aligning scalar labels to `[N, 1]` fixes the residual calculation.
…
The line at the top of this page pins the label git:20261016.4de5604, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
GET https://markdownregistry.com/api/v1/artifacts/art_gi7rm7lzwbrquzm7
GET https://markdownregistry.com/api/v1/resolve?ref=00200200/maintainer-skills-lab/providers/cursor/.cursor/rules/mkl-debug-ml-training.mdc
GET https://markdownregistry.com/api/v1/blob/940d943772e212e5996956981ef67b2faf42cfd0c081e795a344822ed8d20181
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
00200200/maintainer-skills-lab · providers/claude/.claude/skills/mkl-architect-editor/SKILL.md · Two-model pipeline routing high-level design to an Architect model, and mechanical search/replace edits to an Editor…
00200200/maintainer-skills-lab · providers/claude/.claude/skills/mkl-bisect-regression/SKILL.md · Pinpoint regressions using automated git bisect test scripts with zero prompt token waste.
00200200/maintainer-skills-lab · providers/claude/.claude/skills/mkl-bump-version/SKILL.md · Evaluate conventional commits since latest tag, determine next SemVer, and surgically update version files.
00200200/maintainer-skills-lab · providers/claude/.claude/skills/mkl-humanize/SKILL.md · Edit stiff or AI-sounding prose into natural writing while preserving meaning, facts, citations, and the author's point…
