--- name: rch user-invocable: false skill_api_version: 1 hexagonal_role: supporting metadata: tier: execution description: 'Use when offloading slow builds to remote workers or recovering RCH worker, hook, SSH, sync, or disk issues. Triggers: "rch", "use when offloading slow builds", "rch skill".' practices: - pragmatic-programmer --- # RCH — Remote Compilation Helper `rch` transparently offloads compilation commands to remote workers via a Claude Code PreToolUse hook. The daemon picks the fastest healthy worker, rsync's the workspace, runs the build, syncs artifacts back, and exits with the worker's exit code. This skill is the operational layer agents use when something about that pipeline isn't working — and the much more common case where it *thinks* it's working but is silently falling back to local execution. The skill is built around a single principle: **self-resolve before asking the human.** Every recovery path here is one the agent can run on its own. ## Constraints - **Observe before mutation.** Capture the `[RCH]` summary, `rch doctor --json`, and the first failing triage stage before changing daemon, hook, worker, or disk state because an unobserved repair destroys the evidence needed to prove causality. - **Fail-open is not success.** Treat every `[RCH] local ()` result as unresolved until the named reason is repaired or deliberately accepted because a green local build does not prove remote offload worked. - **Bound autonomous repair.** Apply only the documented idempotent or precondition-guarded fixes in this skill, and preview destructive cleanup because remote workers may contain active builds or another agent's state. - **Consult the pawl before raising the andon.** WARN, FAIL, or REFUTED contract results repair and rerun automatically because ordinary rejection is diagnostic evidence; only a breaker may enter HOLD or consume the single helper lane. ## Breaker State Machine - **Ordinary rejection — `WARN|FAIL|REFUTED -> AUTO-REDO`:** repair the named defect, recapture evidence, and rerun the failed check; plain rejection never enters HOLD and never consumes the helper lane. - **Breaker — `BREAKER -> HOLD -> ONE-HELPER`:** when recovery cannot proceed safely with available permissions or observability, freeze mutations and route exactly one bounded helper consultation with the evidence packet. - **Recovered — `HELPER-UNSTUCK -> AUTO-REDO`:** leave HOLD, apply the bounded recovery, and re-earn the deterministic check plus pawl verdict before declaring the pipeline healthy. - **Helper escalation — `HELPER-ESCALATE -> HUMAN`:** stop automation and send the helper-provided concrete escalation packet to the operator. - **Direct human lane — `REFUSAL-LANE|EXPLICIT-JUDGMENT|EXHAUSTED-BUDGET -> HUMAN`:** skip the helper and route directly to the operator; these are the only direct-human states. **Don't re-learn the command surface here.** `rch --help`, `rch doctor`, and the machine surfaces (`--json`, `--schema`, `--help-json`, `--capabilities` — see [MACHINE_INTROSPECTION.md](references/MACHINE_INTROSPECTION.md)) self-describe every subcommand, flag, and env var. Env-var knobs and config precedence: [CONFIGURATION.md](references/CONFIGURATION.md). This skill carries the triage doctrine and recovery playbook routing. Tested against rch v1.0.18; concepts apply to v1.0.16+. --- ## Read This First When a build feels slow, run **one** thing: ```bash RCH_VISIBILITY=verbose 2>&1 | grep -E '^\[RCH\]' ``` The summary line is a contract: | Pattern | What to do | |---|---| | `[RCH] remote (...)` | Healthy. Done. | | `[RCH] remote failed [RCH-Exxx] ...` | Real build/env failure. See [ERROR_CODES.md](references/ERROR_CODES.md). | | `[RCH] local ()` | **Fail-open.** See [FAIL_OPEN.md](references/FAIL_OPEN.md) and look up the reason verbatim. | | *no `[RCH]` line at all* | Hook didn't fire. Run `scripts/protocol_test.sh ""`. | If you can't see why offload isn't happening, **prove the path works in isolation** before doing anything else: ```bash rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_$(basename "$PWD")" cargo check --workspace --all-targets ``` If that prints `[RCH] remote (...)`, the offload pipeline is healthy. The problem is upstream of `rch exec` — usually the hook classifier or the agent's invocation form. If it also fails, follow [RECOVERY_PLAYBOOKS.md](references/RECOVERY_PLAYBOOKS.md). --- ## Fast Triage Order Run in this order and stop at the first failing stage: 1. **Availability** — `rch check`, `rch status --workers --jobs`, `rch workers probe --all`, `rch queue` 2. **Config + socket consistency** — `rch config show --sources`, `rch --json config get general.socket_path`, `rch --json daemon status` 3. **Hook integration** — `rch hook status`, `rch agents status`, `rch hook install` (idempotent) 4. **Command classification + path closure** — `rch diagnose --dry-run ""` 5. **Remote compile proof** — the `rch exec` probe above 6. **If sync fails or storage looks bad, inspect the worker directly:** ```bash ssh ubuntu@ 'df -h / /tmp && free -h && cat /proc/pressure/memory && cat /proc/pressure/io' ssh ubuntu@ 'du -sh /tmp/rch-* /tmp/rch_target_* 2>/dev/null | sort -h' ``` Always check both `/` and `/tmp` on the worker before deciding what to fix. End-to-end verify: `rch self-test --all`; comprehensive checks: `rch doctor` (`--fix --dry-run` previews auto-fixes). **Checkpoint:** before the first mutation, record the failing stage, exact command, exit code, and relevant `[RCH]` summary in the run evidence; after repair, rerun that same probe so the before/after claim is falsifiable. --- ## Quick Fixes | Symptom | Command | |---------|---------| | Hook not installed | `rch hook install && rch hook status` | | Daemon not running | `rch daemon start` | | Daemon version drift / stale socket state | `rch daemon restart -y` (drains gracefully — safe by default) | | No workers configured | `rch workers discover --add --yes && rch workers setup --all` | | Workers unreachable | `rch workers probe --all`, fix SSH key/host — or [SSH_KEY_RECOVERY.md](references/SSH_KEY_RECOVERY.md) | | All workers busy + fail-open | Queueing is default-on; bump `RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120` or raise `total_slots` | | Transfer churn under target dirs | Add excludes in `~/.config/rch/config.toml`, then `rch daemon reload` | | Path dependency missing remotely | [PATH_DEPENDENCIES.md](references/PATH_DEPENDENCIES.md) (configurable via `[path_topology]`) | | Sync fails `Permission denied` in `/data/projects/` | `ssh ubuntu@ 'sudo chown -R ubuntu:ubuntu /data/projects/ && sudo chmod 775 /data/projects/'` | | Worker disk pressure (RCH-E210/211/...) | [DISK_AND_PRESSURE.md](references/DISK_AND_PRESSURE.md) — hand off to the `sbh` skill | | Telemetry / SpeedScore broken | [TELEMETRY_RECOVERY.md](references/TELEMETRY_RECOVERY.md) — move db aside, restart | | Hook says installed but isn't intercepting | `scripts/protocol_test.sh ""` | | Multiple agents racing on fleet ops | Wrap with `scripts/multi_agent_safety.sh ` and use Agent Mail file reservations | | Need full environment diagnosis | `rch doctor --json` and `rch config doctor` | Debugging fail-opens: `RCH_VISIBILITY=verbose` shows the summary line; `RCH_LOG_LEVEL=debug` surfaces which fail-open path was taken. All other env knobs (priority, env allowlist, SSH keepalives, compression, profiles): `rch --help` + [CONFIGURATION.md](references/CONFIGURATION.md) + [SSH_TUNING.md](references/SSH_TUNING.md). --- ## Anti-Asking Rules These are the questions agents historically ask the human that they should *just answer themselves*. The answer is in this skill or trivially derivable. **Do not ask. Do.** - "Can I restart the daemon?" — Yes. `rch daemon restart -y` drains in-flight builds gracefully. It's the documented upgrade path. - "Can I clean up `/tmp/rch_target_*`?" — If `sudo lsof +D ` is empty, yes. See [DISK_AND_PRESSURE.md](references/DISK_AND_PRESSURE.md). If non-empty, never. - "Should I fix the chown on the worker?" — If the symptom matches the Permission denied recipe, yes. It's documented. - "Should I disable an unreachable worker and continue?" — Yes. `rch workers disable --reason "..." --drain -y`, then proceed with what's healthy. - "Should I reinstall the hook?" — If `rch hook status` says missing, yes. `rch hook install` is idempotent. - "Should I sync the toolchain to the workers?" — If `RCH-E205` or "toolchain missing on X" appears, yes. `rch workers sync-toolchain --all`. - "The cooldown is blocking my retry — should I delete it?" — No. Wait `auto_start_cooldown_secs`. If you really need to bypass, use `rch daemon start` directly (it's not gated by the hook autostart cooldown). - "Can I drop the corrupt telemetry db?" — Yes. [TELEMETRY_RECOVERY.md](references/TELEMETRY_RECOVERY.md). Telemetry is derived data. - "Should I recover SSH keys from a sibling host?" — If the keys are missing on this host but reachable on another, yes. [SSH_KEY_RECOVERY.md](references/SSH_KEY_RECOVERY.md) Step 3. When in genuine doubt, capture the escalation packet (Playbook end of [RECOVERY_PLAYBOOKS.md](references/RECOVERY_PLAYBOOKS.md)) and surface that — not a wall of text — to the human. --- ## Output Specification - **Path:** `.agents/evidence/remote-compilation//evidence.json` in the active repository, with raw command output stored beside it when needed. - **Filename convention:** the machine handoff is always `evidence.json`; `` is a filesystem-safe timestamp or task identifier unique to the recovery attempt. - **Serialization/schema format:** JSON object `rch-evidence.v1` with nonempty `run_id`, `summary_line`, and `next_action`; enum `status` (`healthy|recovered|breaker`); enum `stage` (`availability|config|hook|classification|remote-compile|worker-pressure|complete`); nullable string `worker`; and a nonempty `commands` array of `{command:string,exit_code:number}` objects. - **Validator command:** set `OUT=".agents/evidence/remote-compilation//evidence.json"`, then run `jq -e '. as $in | .schema_version=="rch-evidence.v1" and (.run_id|type=="string" and length>0) and (["healthy","recovered","breaker"]|index($in.status))!=null and (["availability","config","hook","classification","remote-compile","worker-pressure","complete"]|index($in.stage))!=null and ($in.summary_line|type=="string" and length>0) and (($in.worker==null) or ($in.worker|type=="string")) and ($in.next_action|type=="string" and length>0) and ($in.commands|type=="array" and length>0) and all($in.commands[]; (.command|type=="string" and length>0) and (.exit_code|type=="number"))' "$OUT"`. - **Downstream handoff:** a `healthy|recovered` packet returns the verified worker and probe to the build lane; a `breaker` packet enters HOLD and accompanies the single helper, and only the explicit human states above reach the operator. ## Quality Checklist - The evidence names the first failing stage and preserves the exact pre-repair command, exit code, and `[RCH]` summary instead of inferring success from build completion. - Every mutation is documented, scoped to the diagnosed failure, and either idempotent or guarded by the playbook's safety precondition. - The same probe is rerun after repair, and `rch self-test --all` or an isolated `rch exec` demonstrates remote execution rather than local fallback. - Ordinary rejection remains in AUTO-REDO; HOLD has exactly one helper, and operator escalation is limited to the declared human states. --- ## Reference Index Everything below ships in the skill. Read whichever is relevant. **Recognising what's wrong:** - [FAIL_OPEN.md](references/FAIL_OPEN.md) — every `[RCH] local (...)` reason mapped to a self-fix - [ERROR_CODES.md](references/ERROR_CODES.md) — full RCH-Exxx catalog with skill-doc cross-refs - [TROUBLESHOOTING.md](references/TROUBLESHOOTING.md) — diagnostic flow + common errors **Solving specific failure classes:** - [RECOVERY_PLAYBOOKS.md](references/RECOVERY_PLAYBOOKS.md) — symptom → fix in ≤90s, organized as 12 lettered playbooks - [SSH_KEY_RECOVERY.md](references/SSH_KEY_RECOVERY.md) — when workers.toml references keys this host doesn't have - [PATH_DEPENDENCIES.md](references/PATH_DEPENDENCIES.md) — multi-repo workspaces, closure planner, `[path_topology]` - [DISK_AND_PRESSURE.md](references/DISK_AND_PRESSURE.md) — RCH-E210..217 + the `sbh` handoff - [TELEMETRY_RECOVERY.md](references/TELEMETRY_RECOVERY.md) — corrupt `~/.local/share/rch/telemetry/telemetry.db` - [SELF_HEALING.md](references/SELF_HEALING.md) — autostart cooldown, daemon supervision, `[self_healing]` - [SSH_TUNING.md](references/SSH_TUNING.md) — ControlMaster, keepalives, retry classification **Operating in fleets and swarms:** - [MULTI_AGENT_CONTENTION.md](references/MULTI_AGENT_CONTENTION.md) — TOCTOU, fleet deploy races, autostart cooldown sharing - [OPERATIONS.md](references/OPERATIONS.md) — full runbook + worker fleet lifecycle - [WORKERS.md](references/WORKERS.md) — worker config, drain/disable/enable, deploy - [CONFIGURATION.md](references/CONFIGURATION.md) — config precedence, env vars, runtime paths - [HOOKS.md](references/HOOKS.md) — hook protocol, install, test - [MACHINE_INTROSPECTION.md](references/MACHINE_INTROSPECTION.md) — `--json`, `--schema`, `--help-json`, `--capabilities` **Automation scripts (in `scripts/`):** - `auto_recover.sh` — heuristic, dry-run-by-default fleet recovery - `worker_disk_triage.sh` — read-only mount-aware disk report per worker - `protocol_test.sh` — directly probe the hook protocol with synthetic input - `multi_agent_safety.sh` — flock wrapper for fleet/setup operations - `mine_rch_history.sh` — find prior agent sessions that hit a given failure - `diagnose-rch.sh` — comprehensive end-to-end diagnostic (the original) **Templates and project docs:** - `assets/workers-template.toml` - Source: --- ## Adjacent Skills - **`sbh`** — disk-pressure defense for AI coding workloads. Use when `RCH-E210/211/215/216` fires. - **`agent-mail`** — file reservations and messaging between agents. Use before `rch fleet deploy` or any worker config edit in a swarm. - **`agent-native`** / **`ntm`** — portable worker lifecycle and NTM mechanics for agents that hit rch failures. - **`cass`** — search prior agent sessions; the skill ships `scripts/mine_rch_history.sh` as a fallback when cass index has dead pointers. --- ## Reading Output: TUI vs Hook `rch` itself, when invoked with **no subcommand**, runs in PreToolUse hook mode (reads JSON from stdin, writes JSON to stdout). Don't run bare `rch` from a terminal expecting help — use `rch --help`. Bare TUIs are at `rch dashboard` (terminal) and `rch web` (browser); both block your session.