training-check · diff
git:20260325.52ac4ec to git:20260427.0aec5ea
55 added, 100 removed. Audit A to A.
---
name: training-check
- description: "Periodically check WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs. Use when training is running and you want automated health checks."
- allowed-tools: Bash(*), Read, Grep, Glob, Write, Edit, Agent
+ description: "Interactively monitor training metrics from the current Codex session, periodically checking WandB or fallback logs for NaN, divergence, plateaus, and broken runs."
+ argument-hint: [wandb-run-or-monitoring-brief]
+ allowed-tools: Bash(*), Read, Write, Edit, Grep, Glob
---
# Training Check
- Periodically read WandB metrics during training to catch problems early. Do not wait until training finishes to discover it was a waste of GPU time.
-
- ## Context: $ARGUMENTS
-
- ## Constants
-
- - **WANDB_RUN** - Read from project notes or pass as `entity/project/run_id`.
- - **CHECK_INTERVAL** - Starts at 10 minutes, then gradually increases if consistently healthy: 10 min -> 20 min -> 30 min -> 60 min (cap).
- - **REVIEWER_MODEL = `gpt-5.4`** - Used via a secondary Codex agent for ambiguous cases only.
-
- ## When to Use
-
- - After training is confirmed running (session alive, loss decreasing for the first few steps)
- - When the user wants recurring health checks during training
- - **This skill checks training QUALITY, not process HEALTH.** Process health (session alive, GPU utilization) belongs to watchdog-style monitoring.
-
- ## Workflow
+ You are now in **interactive watch** / 交互式训练监控模式.
- ### Step 1: Read WandB Metrics
+ Keep the current session open and report directly in the current terminal. The user is watching this terminal for updates. By default, run a training health check every 30 minutes, output a concise but complete analysis report after each check, state the next check time, then continue monitoring.
- ```python
- import wandb
- api = wandb.Api()
- run = api.run("<entity>/<project>/<run_id>")
- history = run.history()
- ```
+ This skill checks training **quality**, not basic process health. Process health checks such as whether a tmux session exists or whether the GPU is idle can be handled by watchdog-style tooling; this skill focuses on whether the run is still worth continuing.
- If WandB is unreachable (API error, network issue), fall back to reading the log file directly via SSH:
+ ## Inputs To Establish First
- ```bash
- ssh server "tail -100 /path/to/training.log"
- ```
+ Before the first check, identify or ask for the minimum monitoring context:
- Check these signals:
+ - WandB run path or URL, if available.
+ - Fallback log path, SSH command, or local command for reading recent training logs.
+ - Training target, expected baseline, and key metrics that define success.
+ - How the training was launched, so it can be stopped if needed.
+ - Project notes path for recording decisions and evidence.
- - **Loss trend** - Is training loss decreasing over the last N steps?
- - **Eval metrics** - Are evaluation metrics improving (or at least not degrading)?
- - **NaN / Inf** - Any NaN or Inf values in loss or gradients?
- - **Spikes** - Sudden large jumps in loss (>10x normal variance)?
- - **Learning rate** - Is the schedule behaving as expected?
- - **Gradient norm** - Exploding or vanishing?
+ If a source is unavailable, say so clearly and continue with the available source. If both WandB and fallback logs are unreachable, report the connectivity issue, classify the round as `WAIT`, and check again later. Do not infer that training is bad only because data is unreachable.
- ### Step 2: Judgment
+ ## Per-Round Check
- | Signal | Judgment | Action |
- |--------|----------|--------|
- | NaN/Inf in loss | **Clearly bad** | Stop training, investigate |
- | Loss diverging (increasing for >N steps) | **Clearly bad** | Stop training, investigate |
- | Eval metrics significantly worse than baseline | **Clearly bad** | Stop training, investigate |
- | Loss decreasing, metrics improving | **Clearly fine** | Continue, increase check interval |
- | Loss flat but not diverging | **Unsure** | -> Step 3 (secondary review) |
- | Metrics noisy, can't tell trend | **Unsure** | -> Step 3 (secondary review) |
- | Slightly worse than baseline but still early | **Unsure** | -> Step 3 (secondary review) |
+ Every round, read WandB first when configured. If WandB is unreachable, read the fallback logs. Inspect at least:
- ### Step 3: Secondary Codex Judgment (only when unsure)
+ - Training loss trend over recent checkpoints or steps.
+ - Eval metrics and whether they improve, flatten, or degrade against baseline.
+ - NaN or Inf in loss, gradients, activations, or logged metrics.
+ - Sudden loss spikes, divergence, or repeated failed evaluations.
+ - Learning rate schedule behavior.
+ - Gradient norm, if logged.
+ - Plateau patterns that suggest the run is no longer useful.
- Only escalate when the signal is ambiguous. For clearly good or clearly bad signals, act directly.
+ Output one report in the current terminal with this structure:
```text
- spawn_agent:
- model: REVIEWER_MODEL
- reasoning_effort: high
- message: |
- TRAINING HEALTH CHECK - need your judgment on ambiguous metrics.
-
- Run: <entity>/<project>/<run_id>
- Current epoch/step: X / Y total
- Training loss (last 10 checkpoints): [values]
- Eval metrics (last 3 evals): [values]
- Baseline reference: [numbers from paper/reproduction]
-
- What I'm unsure about: [specific concern]
+ ## Training Check - <local timestamp>
- Please respond with exactly one of:
- - STOP: clearly problematic, should kill training
- - CONTINUE: looks fine, check again next interval
- - WAIT: not enough data to judge, check again sooner
+ - Data source: wandb_ok | log_fallback | unreachable
+ - Run: <wandb run or training identifier>
+ - Recent metrics: <loss/eval/lr/grad summary>
+ - Anomalies: <NaN/Inf/spike/divergence/plateau findings>
+ - Evidence: <WandB URL, log lines, metric values, or files inspected>
+ - Decision: CONTINUE | WAIT | STOP
+ - Reason: <why this decision is justified>
+ - Next check: <local timestamp, normally 30 minutes later unless ending>
```
- If delegation is unavailable, make a local judgment using the same rubric and mark the decision `[pending external review]`. In ambiguous cases with no hard failure, prefer `WAIT` over `STOP`.
-
- ### Step 4: Act
-
- | Decision | Action |
- |----------|--------|
- | **Stop** | Kill the training session. Save the WandB run URL, key metrics, and reason for stopping. Log to project notes for debugging. |
- | **Continue** | Do nothing. Re-run at the next interval (increase interval if consistently healthy). |
- | **Wait** | Do nothing but keep the current short interval (do not increase). |
-
- ## Integration with Watchdog
-
- `training-check` and watchdog-style monitoring operate at different levels:
-
- | Layer | Tool | What it checks | Frequency |
- |-------|------|----------------|-----------|
- | Process health | watchdog | Session alive? GPU active? | Every 60s (continuous) |
- | Training quality | training-check | Loss trend? Metrics improving? | Every 10-60 min (periodic) |
+ Use the decisions as follows:
- Use both together:
+ | Decision | Meaning | Action |
+ |----------|---------|--------|
+ | `CONTINUE` | Run looks healthy enough to keep training. | Keep monitoring and check again in 30 minutes. |
+ | `WAIT` | Evidence is inconclusive, noisy, too early, or temporarily unreachable. | Do not stop training; keep monitoring and check again later. |
+ | `STOP` | Training is clearly problematic or no longer worth continuing. | Stop the training task, save evidence, write notes, output final summary, and end monitoring. |
- - Watchdog catches crashes and idle GPUs immediately
- - `training-check` catches subtle quality issues (loss plateau, metric degradation)
+ ## Stop Behavior
- ## Rules
+ When the decision is `STOP`:
- - Do not stop training on the first sign of noise - some loss spikes are normal. Look at **trends over multiple checkpoints**.
- - When stopping training, always save the WandB run URL and key metrics as evidence.
- - If both WandB and log files are unreachable, report the connectivity issue and try again next interval. Do not assume training is broken.
- - Gradually increase check interval when healthy (10 -> 20 -> 30 -> 60 min). Reset to 10 min after any anomaly.
- - This skill is meant to be automated via a recurring scheduler. If the user wants ongoing monitoring, set up the best local mechanism available instead of waiting for manual reruns.
+ - Stop the training task.
+ - If the context contains `stop_command`, run `stop_command` first.
+ - If no `stop_command` is available, choose the appropriate stop action from how the training was launched, such as stopping the relevant tmux session, local process, remote process, scheduler job, or notebook job.
+ - Save evidence: WandB URL, key metrics, relevant log snippets, files inspected, and the reason for stopping.
+ - Append a project note for debugging and future analysis.
+ - Output `FINAL_SUMMARY` in the terminal.
+ - End the interactive monitoring loop.
- ## Recurring Setup Example
+ Never stop on the first sign of ordinary metric noise. Look for sustained trends, hard failures, or clear divergence. Always preserve enough evidence for a later agent or human to understand why the run was stopped.
- ```text
- After training is confirmed stable:
- Create a recurring job (cron, task scheduler, tmux loop, etc.)
- that runs `/training-check <entity>/<project>/<run_id>` every 10 minutes.
- ```
+ ## Interactive Loop Guidance
- As the check interval increases, update the old recurring job to match the new interval.
+ - The normal interval is 30 minutes.
+ - If a round is `CONTINUE`, announce the next check time and wait until then.
+ - If a round is `WAIT`, explain what evidence is missing or noisy and check again later. Use a shorter interval only when the run looks suspicious but not yet stop-worthy.
+ - If an anomaly recovers, say so explicitly and continue monitoring.
+ - Keep the user-facing report short enough to read in a terminal, but include concrete metric values and evidence paths.