git:20260810.892e567 to git:20260915.e44ada8

- description: "Use when running, debugging, verifying, or delivering a deep-learning experiment on an owned or rented GPU, especially AutoDL or a remote SSH host; also use for Windows + Clash/Mihomo high-port SSH banner timeouts, fake-IP, or TUN routing interference. Covers launch, checkpoint/resume, detached monitoring, OOM/NaN/convergence/data-loader failures, multi-GPU hangs, ablations, result verification, pull/teardown safety, and canonical export closure. Routes durable replicas to mirror-research-artifacts. Triggers: owned/rented GPU, AutoDL, SSH, Windows Clash/Mihomo, banner timeout, fake-IP, TUN, train/debug/verify/pull/export, 远程GPU训练/租卡, 断点续训, 消融复现, checkpoint 拉回."

- last-model-review: "2026-08-10 AutoDL canonical export and generic mirror handoff; preserves 2026-07 lifecycle review findings (day completed from aca1c467, which rewrote the description and added the Windows/Clash SSH gate — the bare 2026-08 form is unparseable to the staleness hook)"

- # remote-gpu-trainer — the DL Experiment Lifecycle

-

- ## Overview

-

- One skill for the whole arc of a DL experiment: **RUN → VERIFY → DELIVER.**

- - **RUN** — get a long GPU job to start, survive, and finish, then get the result off the box. On a

- machine **you own** there is no meter; on a **rented** box the core insight is that **you are a

- short-term tenant on someone else's machine** — so the job is to *detach the work, make the result

- outlive the instance, and stop the meter safely*, not to provision a cluster. Platform-specific at the

- edges (one `profiles/<platform>.md` owns every path, proxy, billing verb, and spot rule), invariant at

- the core.

- - **Remote ownership boundary** — this skill is the **compute/control layer**: it binds inputs, runs and

- verifies compute, and closes one run into `export/<run-id>`. Long-term project organization belongs to

- `research-artifact-hygiene`; any durable/local/cloud mirror begins only at a validated export and belongs

- to the generic `mirror-research-artifacts` skill. Never mirror a mutable `active/` tree.

- - **VERIFY** — *is this number a bug, a real effect, or noise?* A surprising result is a hypothesis, not

- a fact to report. Platform-agnostic.

- - **DELIVER** — organize the result so every shipped number/figure/table is a *deterministic function of

- one immutable evidence layer*; provenance and cross-document consistency are locked by mechanism, not

- by a human remembering to update three documents. Platform-agnostic.

- Two stances run through VERIFY and DELIVER: **user sovereignty** (the science — seed count, which samples,

- whether an `aux` channel exists — is the user's call; the skill organizes and discloses a tradeoff *once*,

- then stops nagging) and **audit → disclose, not enforce** (the skill is an honest auditor, not a gate

- guard — an integrity issue must surface *with the conclusion it affects*, but the skill never blocks the

- user from shipping). Mantra: **"disclose it, or don't claim it."**

- ## Route first

- 1. **RUN — own the box or rent it?**

- - **Local** (a workstation/laptop you own, no meter) → `references/run-local/` and `profiles/local.md`.

- - **Rented / remote** (any metered or shared box you don't own) → `references/run-remote/`, and pick

- your **`profiles/<platform>.md`** FIRST (it owns every path/verb/proxy the phases delegate to).

- 2. **Then ALWAYS** → **VERIFY** the result (`references/verifying/`). A green run is not a real number.

- 3. When publication synthesis is requested, optionally consult `references/delivering/` for

- **legacy/non-canonical publication synthesis guidance**. It never defines the artifact layout: canonical

- runs, hardware evidence, trust records, figures and tables belong to `research-artifact-hygiene`.

- > Already debugging a model that won't converge / OOMs / hangs / NaNs, regardless of where it runs? Jump

- > straight to **`references/training/`** (the 8-file debug layer), then come back to VERIFY before you report.

- ## Operating principles (the spine)

- The load-bearing invariants. One line each; the full cross-platform set (10 invariants for the remote

- lifecycle) is in **`references/run-remote/principles.md`** — read it before Phase 0 of a remote run.

- - **Checkpoint-to-durable + idempotent resume is the universal spine.** File-checkpoint to the durable

- location + unconditional load-latest-on-startup is the *one* mechanism that survives an SSH drop, a

- Slurm walltime kill, a K8s reschedule, a spot preemption, a Colab disconnect. The detach primitive

- (tmux / sbatch / Job) is the swappable plug; this is the invariant.

- - **Trust the artifact you loaded, not a log line that claims success.** "synced / saved / done" lies

- under a silently-failed write; a watcher's own state is also a claim — reconcile it against the real

- process / artifact / pixels / bytes.

- - **Cheap checks before expensive compute.** A 1–2 batch CPU smoke (logger off) kills import/config/

- shape/scale bugs for ~free, before they bill GPU-hours.

- - **Cost and destructive actions are the user's call.** Never auto-release/terminate, never delete durable

- files without confirmation; if cleanup can't free space, ask to expand the disk, don't silently shrink

- the experiment.

- - **Execution permission is not task authority.** Sandbox / Full Access only controls whether a command

- can run. Keep operational authority and scientific promotion separate: once a bounded, non-overwriting

- delivery objective is authorized, same-scope diagnostics, verifier/schema version bumps, tests, hashes,

- and control-plane repairs do **not** require a fresh confirmation merely because they mint a new immutable

- ID. Re-ask for new billable compute, destructive/irreversible actions, science-protocol changes, metric

- promotion, publication, or any other material scope expansion. A standing unattended contract applies

- only after its own activation rule is satisfied.

- - **Make the control plane cheap and the data plane rare.** Validate small schemas, identities, paths, and

- contract hashes before rereading multi-GB bundles. A synthetic fixture may test rejection behavior but

- may never invent the producer's positive schema; freeze a redacted real-shape fixture and prove its test

- is live. Recompute every large payload once per trust boundary—producer, independent remote acceptance,

- and local pull—not once per wrapper or verifier revision.

- - **One-way run closure.** Mutable work stays under `active/<run-id>`; only a validated capsule may move

- from `export/.partial/<run-id>` to `export/<run-id>`, and failed closeouts move to `quarantine/`.

- AutoDL binds this exactly as

- `/root/autodl-tmp/<project>/{cache,active,export/.partial,export/<run-id>,quarantine}`. A closed export is

- fully isomorphic to canonical local `runs/<run-id>`: `run.json`, `config.yaml`, `train.csv`, `best.pth`,

- optional frozen `last.pth`, and `test/<test-id>/{metrics.json,results.parquet,vis/<condition-id>/<task-native-role>/<sample-id>.png}`.

- Every declared software test must include visualization coverage for all declared conditions × task-native

- roles × K fixed selected samples. Real capture/hardware results never enter this software capsule; close

- them separately as `export/hardware/<hardware-run-id>` (capture/decode/model-run bindings, no copied

- weights) or hand them to `research-artifact-hygiene`. A hardware test without machine-readable ground truth

- must declare that status and metric non-applicability; it keeps finite-forward rows and prediction/overlay

- visuals but never invents ground truth or GT-derived metrics. Caches and whole active trees never cross a

- mirror boundary. Layout and gates → `references/run-remote/artifact-layout.md`.

- - **Before teardown, prove the evidence outlives the host.** Teardown is irreversible and *"I scp'd it

- back"* is just another log line. The gate is not "files copied" but **"every number I reported

- re-reads from the local copy"** — diff each claim against the pulled artifact, then write a

- provenance note *next to the data* (protocol, reference frame, caveats) so a later reader can

- retrace it without the chat log. Two traps: (1) what you did **not** pull is a decision, not an

- oversight — say which (checkpoints are usually re-derivable from config+seed *if* determinism is

- established; results are not); (2) **re-read the inventory, don't trust its prose** — a note saying

- "none of these are local" may mean *none was trained here*, not *none is stored here*; the two differ

- by everything when you are about to press destroy.

- - **Audit → disclose, not enforce.** What is mandatory is *disclosure*, not the *fix*. An integrity

- finding (no disjoint val, leakage, test touched during selection, a number you can't re-derive) must

- ride *with* the conclusion — but the skill discloses, it does not block.

- ## RUN — local (a box you own)

- No meter, no teardown clock — the risks move from *money* to *resource contention and your machine's

- stability*. The discipline that does **not** relax: env hygiene, resource awareness, artifact/checkpoint

- care, and "state the seed." Start at `profiles/local.md`, then the matching doc:

- - **Env hygiene** — never train/install in conda `base` on a persistent box; the 4-step gate (enumerate →

- pick the project env → confirm `sys.executable` → run) → `references/run-local/env-hygiene.md`.

- - **Launch & detach** — nohup/tmux, log + alive probe, don't foreground-block → `references/run-local/launch.md`.

- - **Single-node multi-GPU** — `torchrun`/`accelerate` DDP env contract, the first-run rank/hang basics →

- `references/run-local/multi-gpu.md` (multi-*node* → `references/run-remote/multinode.md`).

- - **Local OOM** — the fit-it ladder on hardware you can't rent bigger → `references/run-local/local-oom.md`.

- ## RUN — remote (a box you rent)

- **Pick your profile FIRST** — it binds every concrete path/proxy/credential/billing verb/spot rule the

- phases delegate to. Mental verb model (one API across platforms; the profile binds each verb to real

- commands): `up` (rent+reach) → `push` (code/data on) → `run` (detached + checkpointing) → `watch`

- (durable monitor) → `pull` (results off + verify) → `down` (stop the meter).

- **Windows + Clash/Mihomo high-port SSH gate.** Use **OpenSSH direct first** with strict host-key

- checking. A **Paramiko fallback** is allowed only after recorded `banner_timeout`, `fake_ip`, or

- `tun_interference` evidence; it must use a DoH-selected address and Windows `IP_UNICAST_IF` on the

- single socket handed to the SSH transport. Never treat authentication failure, host-key mismatch,

- connection refusal, or an unexplained error as proxy evidence. Never mutate system routes, DNS,

- proxy settings, or Clash/Mihomo configuration. Once fallback is authorized, you **must not report**

- the host unreachable or a live refresh blocked before a bounded Paramiko single-socket attempt completes

- or host identity fails closed. A **transport failure proves only transport unavailability**; it never

- proves that a remote run is completed, live, failed or stalled. Full parameterized decision ladder and

- offline planner → `references/run-remote/ssh_transport.md` §4A.

- | You're on… | Profile | Meter-stop verb (the trap) |

- |---|---|---|

- | AutoDL (deepest, battle-tested) | `profiles/autodl.md` | 关机 stops meter, **keeps disk** (the AutoDL exception) |

- | RunPod | `profiles/runpod.md` | **terminate** (stop still bills 2×; destroys volume disk) |

- | vast.ai | `profiles/vastai.md` | **destroy** (stop bills disk forever) |

- | Lambda | `profiles/lambda.md` | **terminate** (no stop state) |

- | Paperspace | `profiles/paperspace.md` | **destroy + release IP + delete storage** |

- | 恒源云 / 矩池云 / Featurize / 揽睿星舟 | `profiles/china.md` | per-platform (data disk often bills while stopped) |

- | Bare SSH / Slurm / K8s / Colab | `profiles/generic-ssh.md` | **manual** (a forgotten box bills 24/7) |

- **The 6-phase lifecycle** (full per-platform checklist → `references/run-remote/lifecycle_checklist.md`):

- **0** env + storage-layout audit (`df -i` not just `df -h`, cgroup `memory.max`, checkpoint/inode budget) · **1** SSH +

- credentials (the prebuilt image **is** the env — don't `conda create` on a rental; secrets via stdin) ·

- **2** identity-bound inputs + isolated active run + **CPU-smoke gate before renting** · **3** detached launch (probe, then hand back — never

- a blocking `sleep`) · **4** durable monitoring (the four-layer architecture →

- `references/run-remote/monitoring_patterns.md`; a session-bound watcher dies with the session) · **5**

- close `active → export`, verify/pull or hand the closed export to the generic mirror skill, then teardown.

- > **Iron Law — teardown gate:** NO `release` / `terminate` / `destroy` / file-delete until the remote

- > durable result root has an external immutable `PULL_MANIFEST.json` built by

- > `scripts/aggregate_to_fs.sh` + `scripts/build_pull_manifest.py` from an explicit expected roster

- > (never embedded in canonical `run.json`; the mirror workflow's custody manifest is an additional

- > layer, not a replacement for this one),

- > the pull matches that exact roster + every byte size + SHA-256, every checkpoint loads, and

- > `scripts/verify_local.py` writes local `PULL_VERIFIED.json`; then the user must still explicitly

- > approve the cost-affecting action. A directory count, a size heuristic, an old loadable checkpoint,

- > or "it looked done in the log" is not evidence. On most platforms the

- > meter-stopping action is **irreversible** (deletes the disk) — confirmation matters more, not less.

- Other remote references: `ssh_transport.md` (rsync/scp resumable, secrets-via-stdin, CRLF) ·

- `artifact-layout.md` (active/export/quarantine boundary, canonical capsule and atomic-close contract) ·

- `spot-resilience.md` (preemption grace, Young/Daly cadence, atomic-write resume) · `china-network.md`

- (mirrors + `HF_ENDPOINT` + the `no_proxy` trap) · `parallel_ablation.md` (fan-out independence +

- reconciliation) · `multinode.md` (NCCL/fabric, advanced) · `production-matrix-acceptance.md`

- (shared-disk multi-node patrol, producer→independent acceptance, control/science failure separation) ·

- `gotchas_universal.md` (the full U1–U44 catalog with a grep index).

- ## When training itself breaks (the model, not the platform)

- Once the box runs, training breaks in its own ways — **local or remote, the same debug layer**

- (`references/training/`, 8 files; each entry symptom → root cause → fix with cited docs). Route by symptom:

- - **OOM / won't fit** (CUDA-VRAM or host-RAM, OOM-at-a-step, the fit-it ladder) → `oom-memory.md`.

- - **Multi-GPU launch / HANGS** (`torchrun`/`accelerate`/`deepspeed` env contract, DDP/FSDP/ZeRO) → `distributed-launch.md`.

- - **NaN / Inf / loss spikes** (fp16/bf16/tf32, AMP/GradScaler, LLM divergence) → `precision-stability.md`.

- - **Too slow** (GPU- vs data- vs comms-bound, dataloader knobs, `torch.compile` traps) → `throughput-profiling.md`.

- - **Resume bugs** (full-state + sharded save/resume; epoch restart, reshuffle, scaler/EMA dropped) → `checkpoint-resume.md`.

- - **Per-domain gotchas** (LLM, vision det/seg, diffusion, RL, multimodal/VLM) → `by-domain.md`.

- - **Runs but won't learn** (overfit-one-batch, params-not-updating, LR/schedule, loss-function footguns, freezing) → `convergence-debugging.md`.

- - **Dataloader correctness** (worker-RNG aug duplication, IterableDataset sharding, RGB-vs-BGR / ÷255 / `set_epoch`) → `data-pipeline.md`.

- ## VERIFY — is the number real?

- Before you trust or report **any** metric, ablation delta, or "it works now": classify it **bug / effect /

- noise**, hold a comparison to **exactly one** changed variable, and probe leakage / fair-comparison /

- variance / metric-direction. A number you can't re-derive from the saved artifact is not a result yet.

- Stance: **audit → disclose** — surface an integrity issue with the conclusion, never silently pass or hard-block.

- - Full methodology (the 14-section probe ladder + the 6 invariants) → `references/verifying/methodology.md`.

- - Constant / degenerate output, `real == shuffle`, model-ignores-input → `references/verifying/representation-collapse.md`.

- - A green smoke that hides undertraining vs a real bug; loss-low-but-samples-bad → `references/verifying/smoke-hidden-failures.md`.

- - For an atomic producer bundle, run `scripts/verify_artifact_bundle.py` on a different node; compare two

- independently produced acceptances with `scripts/compare_acceptance.py`. A hash match proves identity,

- a safe load proves checkpoint structure, and a fresh evaluator proves the metric—none substitutes for

- the others.

- > **State the metric's direction when comparing** (PSNR/SSIM/mAP ↑ better; LPIPS/NMSE/loss ↓ better) —

- > never assume. Tracker forensics / pruning duplicate runs → `scripts/wandb_forensics.py`.

- ## DELIVER — legacy/non-canonical publication synthesis guidance

- On a remote rental, delivery ends at the validated closed `export/<run-id>` capsule. Do not turn the compute

- host into the long-term project archive, figure workshop or mirror manager; hand canonical organization to

- `research-artifact-hygiene` and replicas to `mirror-research-artifacts`.

- The `references/delivering/` group is retained only for publication-synthesis principles such as

- generated-not-transcribed reporting, claim reconciliation, disclosure and pixel re-open QA. **It is not an

- artifact-layout authority.** Any directory/manifest example there that differs from

- `research-artifact-hygiene` is legacy and non-canonical.

- For synthesis, read evidence from canonical `runs/<run-id>/test/<test-id>/{metrics.json,results.parquet,vis/}`

- or `hardware/runs/<hardware-run-id>/test/<test-id>/...`; never create a parallel `results/<exp-id>/runs/`

- tree, checkpoint subdirectory, qualitative tree or selected symlink. Figures/tables use the canonical flat

- workshops (`figure.json`/`table.json`, root-level source, `build.py`, `final.*`) rather than README portals,

- nested output directories or provenance sidecars. The fixed selection roster remains the canonical

- `run.json`-bound `_trust/selections` manifest; a publication montage may cite existing atomic PNGs but may

- not redefine the run's visualization roster.

- This skill owns one execution attempt, not project-wide closeout state. Route authorization gates, node

- queues/heartbeats, legacy checkpoint acceptance, and the separate training/evaluation/pull/paper axes to

- `supervise-research-closeout`; that controller must call this skill rather than duplicate its SSH or trainer logic.

- Legacy synthesis navigation: `references/delivering/data-architecture.md` states the boundary and current

- flat-workshop mapping; `principles.md`, `figures.md`, `delivery-gate.md` and

- `completeness-reconciliation.md` provide advisory synthesis/QA checks only. None may override canonical

- paths, manifests or retention rules.

- ## Companion skills (all OPTIONAL — this skill is standalone)

- Recommended separate installs that deepen RUN / VERIFY / DELIVER; **the skill needs none of them** and works

- fully standalone. One-line-each list, what each adds, and the no-companion fallback →

- **`references/companions.md`**. In short: figure drawing (nature-figure),

- data availability (`nature-data`), experiment verification (the `experiment-verifier` agent), parallel ablation

- (`superpowers:dispatching-parallel-agents`), durable artifact mirroring

- (`mirror-research-artifacts`), and — one layer above — an idea→conclusion

- orchestrator (`auto-research-pipeline`: human gates + stage wiring; this skill executes, that one decides

- when each stage fires and what a human signs).

- ## Getting better over time

- The skill is static, but every run can teach it a gotcha — without corrupting it. Protocol →

- `references/self-improvement.md`: only sediment a **root-caused, reproduced, generalizable** gotcha (a

- one-off flake is a hypothesis, not a gotcha); route user/project-specific facts to the host's memory and

- generalizable ones to a proposed catalog edit; **never silently rewrite a skill file** — draft the

- `symptom → root cause → fix` and let the user approve. Platform facts carry a `verified <month>` stamp —

- re-verify any teardown/billing fact against current docs before betting money or data

- (`scripts/check_staleness.py`).

- ## Bundled resources

- Load only what the current phase needs (the body sections above name the individual files).

- - `references/run-local/` — **own-a-box**: env-hygiene · launch · multi-gpu · local-oom.

- - `references/run-remote/` — **rented-box**: principles · lifecycle_checklist · artifact-layout · monitoring_patterns · ssh_transport · spot-resilience · china-network · parallel_ablation · multinode · production-matrix-acceptance · gotchas_universal (U1–U44).

- - `references/training/` — the **DL-training debug layer** (8 files; local/remote-agnostic) — routed above.

- - `references/verifying/` — **is-the-number-real**: methodology · representation-collapse · smoke-hidden-failures.

- - `references/delivering/` — **legacy/non-canonical publication synthesis guidance only**: advisory

- principles · canonical-boundary data-architecture · historical manifest/figure notes · delivery gate ·

- completeness reconciliation. `research-artifact-hygiene` owns every canonical path and schema.

- - `references/companions.md` (optional skills + fallbacks) · `references/self-improvement.md` (capture-a-gotcha loop).

- - `profiles/<platform>.md` — per-platform substrate (7 rental profiles + `local.md`; `_schema.md` = the fields).

- - `scripts/` — wrappers (`run_one`/`run_queue`), monitors (`mem_monitor`, `gpu_health`, `health_patrol.sh.template`, `reap_vram_zombies.sh`),

- transfer (`download_loop`, `aggregate_to_fs`, `build_pull_manifest.py`, `setup-china-mirrors`), `verify_local.py`, delivering

- (`manifest_scaffold.py`, `reconcile.py`, `repro.sh.template`), atomic acceptance

- (`verify_artifact_bundle.py`, `compare_acceptance.py`), `wandb_forensics.py`, `check_staleness.py`.

- - `examples/autodl_sweep/` — one runnable worked case · `evals/` — the regression harness.