job-babysitting git:20261107.9f2365fA

Immutable. This exact content never changes and is served at /api/v1/blob/fb71afc4b6618c3b.

---
name: job-babysitting
description: "Directly monitor long-running jobs or pipelines, classify them as healthy, waiting, stalled, failed, completed, or unknown, and choose the next action. Use when the active agent needs to babysit training runs, batch jobs, services, experiments, or scheduled work over time."
---

# Job Babysitting

## Treat Babysitting As State Classification

- Babysitting means maintaining an evidence-backed view of whether work is progressing and intervening only when the evidence supports it.
- Compose this skill with scheduler, experiment, or domain skills. Use those skills for platform-specific commands and this skill for monitoring logic.
- Reuse the project's existing monitoring signals before inventing a new dashboard or parser.
- The active agent owns the monitoring loop end to end. Do not hand the work to a separate service, timer, watcher, or one-off agent.

## Run The Monitoring Loop Directly

1. Define the requested terminal or decision condition.
2. Define progress signals and their expected cadence before polling.
3. Inspect scheduler or process state, then verify logs and expected artifacts.
4. Classify the state and take only the proportionate next action.
5. Report the evidence, wait for the next meaningful cadence, and repeat.

- Continue until the terminal condition is met, the user stops monitoring, or a genuine blocker requires new authority or information.
- Use native scheduler, process, log, and filesystem tools from the current session. Do not install or invoke a separate monitoring subsystem.
- Keep healthy polls cheap. Read only new log output and changed artifacts instead of repeatedly rescanning complete run trees.
- Preserve interventions and decision-changing results in the run README or routed lab notebook, not in a parallel monitoring ledger.
- Do not spawn a persistent monitoring subagent unless the user explicitly requests delegated ownership.

## Use This Output Contract

When using this skill, report:

1. State: `healthy`, `waiting`, `stalled`, `failed`, `completed`, or `unknown`.
2. Evidence: the strongest signals supporting that state.
3. Last progress point: timestamp, counter, checkpoint, or artifact.
4. Blocker or failure cause: if known.
5. Next action: wait, recheck, inspect, cancel, restart, resubmit, or escalate.

## Define Progress First

- Identify what counts as real progress for the job, such as queue state changes, fresh log lines, checkpoints, metrics, output files, heartbeats, or completed subtasks.
- Identify the expected cadence from healthy prior runs, sibling jobs, or the launcher's documented behavior.
- Do not call a job stalled until you know what normal progress looks like.

## Gather Evidence In Layers

- Check orchestration state, such as scheduler, process, or service status.
- Check progress artifacts, such as log freshness, step counters, metrics, checkpoints, or output directories.
- If outputs exist, inspect one representative artifact or plot when visual corruption or qualitative failure is part of the success criterion.
- Check failure signals, such as tracebacks, repeated retries, OOMs, timeouts, dead heartbeats, or frozen timestamps.
- Check external blockers, such as dependency waits, missing inputs, storage issues, or exhausted resources.

## Apply These Rules

- Do not label a job `stalled` without at least two stale or frozen signals when a second signal is available.
- Do not relaunch a failed job before deciding whether the failure was scheduler-only or a real runtime defect.
- Do not declare success until the expected output artifacts exist.
- Do not mistake process existence, GPU allocation, or `RUNNING` state for proof of health.

## Classify The Current State

- `healthy`: evidence of forward progress exists at the expected cadence.
- `waiting`: blocked by scheduler, dependency, quota, reservation, or another external gate, without evidence of code failure.
- `stalled`: the job is nominally active but progress signals stopped beyond a reasonable threshold.
- `failed`: the job reached a terminal state or logs show a real runtime error.
- `completed`: the job ended and expected outputs exist.
- `unknown`: evidence is incomplete or contradictory.

## Act Proportionally

- For `healthy`, keep monitoring and report the strongest progress evidence.
- For `waiting`, report the blocker and the earliest meaningful recheck.
- For `stalled`, verify with at least two independent stale signals before cancelling or restarting when possible.
- For `failed`, inspect the root cause before relaunching. If the failure is real, fix or escalate instead of blindly retrying.
- For `completed`, verify the expected artifacts before declaring success.
- For `completed`, inspect a representative visual artifact as well when a job can finish while still producing qualitatively bad output.
- For `unknown`, run the fastest check that can reclassify the state.

## A Failed Login Halts Everything

A refused login to an external service (a file share, a web portal, a database, a
remote host) stops all work that uses that credential, immediately. Do not retry,
resubmit, rerun, or test the login, by hand or by script, and do not start another
job that logs in with it. Most accounts lock after a few failed attempts, and a
locked account usually needs a favour from someone outside the lab to undo. When a
password changes, every client still holding the old one fails, and those failures
together are what lock the account: one password change once locked a shared
hospital account while two agents' transfers kept logging in.

- Treat any authentication refusal as this case, whatever the error code:
  `NT_STATUS_LOGON_FAILURE`, `NT_STATUS_ACCESS_DENIED` at login,
  `NT_STATUS_ACCOUNT_LOCKED_OUT`, HTTP 401, a returned login page, `Permission
  denied (publickey)`. A first refusal is enough. No allowance of one more try.
- Stop every client you own that uses the credential: cancel or hold its jobs,
  timers, cron entries and retry loops, and confirm none is still running, including
  processes left on a data-transfer node after the scheduler job ended.
- Tell the user at once: which service refused, when, the error code, and how many
  attempts your clients made. Tell every other agent that uses the same account so
  they stop too. Never print the credential.
- Resume only after the user says the credential works. Then make a single attempt
  that halts on its first refusal.

## Report Like An Operator

- Include exact evidence paths, timestamps, counters, and job or run IDs.
- Distinguish "no recent evidence of progress" from "confirmed failure".
- Make the next action explicit: wait, recheck, inspect logs, cancel, restart, resubmit, or escalate.
- When handing off, say what you checked and what would trigger the next intervention.


## Monitor By Exclusion, Never By An Allowlist

A selector that lists the jobs, runs, or artifacts to watch stops covering work that
is later renamed, and the resulting silence is indistinguishable from health.

- Watch everything in scope and subtract only known-benign long-running owners, so
  new work is covered by default.
- Report every terminal failure state, rather than the absence of a success marker.
- Seed already-finished work silently at startup, so history is not replayed as
  events.
- Keep the failure-detection period shorter than any general review pass.
- When one account serves several projects, scope by submission directory rather
  than by job name, and subtract only directories named explicitly, so an
  unrecognised one is reported once rather than ignored forever.
- A lane submitting from the same checkout can be separated only by name. Treat that
  as the last resort it is: keep the pattern too narrow to match your own work, and
  check that a rename there would surface as noise rather than silence. Noise you
  route by hand costs less than a filter broad enough to swallow something real.

## The Wait Condition Is Where The Defect Hides

A monitor's wait condition is subject to the same known-positive requirement as its
output filter, and it is the easier of the two to get wrong: the filter is what you
think about, so the condition is where the defect hides.

A `pgrep -f` on the command being watched matches the watching shell's own
arguments, so the loop never exits and the watch expires silently no matter what the
work did. Key the condition on a pid, an exit marker, or a file the work writes, and
ask what would make the loop terminate before trusting that it will.

## Prove The Query Fires On A Real Positive

Before treating a monitor's clean result as evidence, verify it returns a known
positive. A query that cannot be shown to fire on a real failure reports the same
silence whether the system is healthy or the query is wrong. Prefer a checked owner
that fails loudly over an ad-hoc command whose silence is unfalsifiable.

A standing notification channel must be a committed script rather than an inline
command, and this holds even when the inline query is correct. An inline loop keeps
its seen-set in a temporary file, so every restart replays days of handled history
as new events; it cannot be reviewed, diffed, or fixed once, and editing it while it
runs shifts byte offsets under a live shell. Give the channel a durable ledger and
prove its query against a real in-scope positive at startup, saying so plainly when
the window holds none.

## Make A Long Run Fail Early And Fail Once

A run that can refuse individual units (an exam, a file, a receipt) and stops the
whole batch at the first refusal turns every bad unit into a full relaunch. When the
check sits many hours in, each relaunch costs those hours again. One cohort intake
failed twice this way at hours 22 and 23: once on a delivery it was comparing against
itself, then on a single non-image file in an accepted series. Each failure was found,
fixed and relaunched before the next one could be seen.

- Order the checks by cost. A check that needs only metadata the run already has, such
  as receipts, ledgers, manifests or header tables, runs before the step that hashes or
  decodes the data, so a bad unit fails in minutes.
- A check that refuses one unit for its own reasons collects every refusal in one
  pass and stops the run at the end of that pass, reporting all of them. A refusal that
  means the run's own inputs or code are corrupt still stops at once.
- Before relaunching after one such failure, look for the others. Run the failing
  check over every unit in a short diagnostic job, and fix the whole class instead of
  the one instance.
- Know before launch whether a relaunch can reuse finished work. If the intermediates
  are keyed to the code version, every fix costs the whole run, and that cost belongs
  in the decision to launch an untried path at cohort scale.

## A Completed Run Is Not A Healthy One

Monitor learning, not only liveness. Terminal-state sweeps and gates asserting that
a loss is finite and a gradient nonzero all pass for a run that trained to a
constant: nonzero is not decreasing and finite is not improving. On each pass over a
training run, read the scale-free loss against its no-skill reference and the
latent-field spread, and report a run that is not learning as a failure even though
nothing crashed. Detecting this is the agent's job, not the user's.

## References

- Read `references/health-signals.md` for monitoring cues, stall heuristics, and next-action patterns.