AGENTS.md@cli/src/lib/daemon git:20261012.084dfacA

Immutable. This exact content never changes and is served at /api/v1/blob/40d35b8e06dd8579.

# Daemon module

The background process behind `agents __daemon-run` — the interval-driven
services that keep sessions, watchdog checks, device probes,
self-heal, and reapers running on each machine without a human polling them.

## Current architecture

**N independent per-device daemons, no daemon-to-daemon bus.** Every device
that runs agents-cli runs its own daemon process (`agents __daemon-run` →
`runDaemon()`, `daemon.ts:850`). No daemon opens a persistent connection to
another daemon. Cross-device state either rides per-device files in the existing
fleet-synced user repo (`fleet-shared-state.ts`, used by usage/auth sync) or an
on-demand `ssh` invocation of a peer CLI verb — e.g.
`watchFleetFeed` spawns `ssh <peer> agents feed watch --local`
(`cli/src/lib/feed/watch.ts:151`) and `watchFleetSessions` does the same for
sessions (`cli/src/lib/session/remote/watch.ts:245`). There is no shared daemon
protocol and no routine device-to-device usage/auth probe mesh.

**Start, claim, and stop are one serialized lifecycle.** All three mutate the
shared pid/heartbeat/socket inventory behind `<daemonDir>/daemon.lock`.
`stopDaemon()` holds that lock from target resolution through its postcondition,
accepts a pid only when the live process command ends in `__daemon-run`, and
removes a pid or POSIX socket only while it still matches the owner/inode captured
under the lock. A failed process-command inspection is an unverified live owner,
never permission to signal it, erase its files, or launch a duplicate.
`reapStrayDaemons()` applies the same identity gate, waits for SIGTERM death, then
escalates and waits again; it removes a registry marker only after death is
observed, so a wedged duplicate cannot survive while becoming invisible.

**A daemon launch under a redirected HOME is refused (W4, PHNX-3736).**
`startDaemon()` throws `RedirectedHomeDaemonError` when `isolatedHomeSuffix()`
detects HOME pointing away from the account's real home, unless the caller opted
in with `AGENTS_ALLOW_TEST_DAEMON=1` (the deliberate test/e2e seam — a harness
that sets it owns stopping what it starts). The explicit start commands
(`daemon start`, `routines start`) fail loud; auto-start side effects
(`routines add`, webhook fires) catch the refusal, state it,
and leave the foreground command green — the same tier split as the auto-start
circuit breaker. A daemon launched under a temp HOME
keeps its own pid file there, so the real install's pid-file takeover can never
see it: the `HOME=/tmp/pin-e2e-<pid>` daemon a headless e2e session leaked ran
4+ days on yosemite-s1 invisible to every existing surface. Detection lives in
`leaked-daemons.ts` (`findLeakedDaemons`): any same-uid `__daemon-run` whose pid
is neither the service manager's unit main PID nor the recorded `daemon.pid` —
checked against BOTH this process's HOME and the real account home, so a
redirected-HOME caller never accuses the production daemon — is surfaced by
`agents doctor` as a `leaked-daemon` warning with the process's HOME, start
time, and a `kill <pid>` remediation.

**Two runtime models coexist today (RUSH-3193 plus PHNX-3265/PHNX-3608/PHNX-3695
migrated every declared service but one onto the supervisor; 1 declared service
remains inline).** `cli/src/lib/daemon-services.ts` defines `DaemonServiceId` —
the catalog every id in `runDaemon()` is expected to register under, whichever
model it uses. Two services that used to live here, `secrets-broker` and
`keychain-reap`, moved out of this daemon entirely with the standalone
`secrets` engine (PHNX-3989 OWN-1) — this daemon no longer hosts, self-heals,
or reaps that broker at all; `agents daemon status`/`services` only probes its
reachability (`probeSecretsBroker`, `commands/daemon.ts`), reporting a health
record of `null` for it.

- **Supervised (`ServiceSupervisor`, `supervisor.ts`):**
  `browser-ipc`
  (`browser-ipc-service.ts`), `account-state` + `account-auth`
  (`account-state-daemon-service.ts` — PHNX-3608 split the old single
  account-state service into `AccountUsageService` and `AccountAuthService`, two
  periodic services with INDEPENDENT circuit breakers so a run of usage-refresh
  failures parks only usage and never starves the slower auth refresh), `catchup`
  (`catchup-service.ts`, PHNX-3608 — the scheduler's missed-fire recovery pass,
  self-gated on the scheduler being booted), `session-index`
  (`session-index-service.ts`) (all P1/P2), and — since P3 — `watchdog` (`watchdog-service.ts`),
  `device-probe` (`device-probe-service.ts`), `self-heal`
  (`self-heal-service.ts`), and
  `state-dir-check` (`state-dir-check-service.ts`), `session-state`
  (`session-state-service.ts`), `feed-stream` (`feed-stream-service.ts`: owns the
  ONE fleet feed fan-out and serves it to every reader over a socket, so N
  consumers cost one ssh per peer instead of N — PHNX-3999; the socket binds on
  start but the fan-out itself is demand-gated on a connected reader), and
  `daemon-heartbeat`
  (`heartbeat-service.ts`), `tmux-reap` (`tmux-reap-service.ts`), and
  `browser-task-reap` (`browser-task-reap-service.ts`), and `auth-sync`
  (`auth-sync-service.ts`: also mints and pushes owner-notify device tokens to
  workers, PHNX-4267), `usage-sync` (`usage-sync-service.ts`), and — since
  PHNX-3695 — `self-update` (`self-update-service.ts`: checks npm for a newer
  agents-cli roughly every 75 minutes, installs + byte-verifies it with the
  same primitives `agents upgrade` uses, best-effort pulls the `.system`
  companion repo and reconciles with `agents sync --local`, then
  `process.exit(0)`s so the OS supervisor — launchd `KeepAlive` / systemd
  `Restart=always` — relaunches the daemon onto the new code; fails CLOSED on
  any step — a failed install/verify leaves the running daemon untouched and
  retries next tick, never exits into unverified code. Before any of that, the
  tick compares the on-disk `package.json` version (`getCliVersionFresh`) with
  the version the process booted with; when another `agents` process has
  already upgraded the install underneath the daemon, it exits for the relaunch
  without downloading anything — the writer of that install byte-verified it —
  once that install has settled (`installLooksSettled`: package.json at rest for
  a minute and every `bin` entry present; bun's write is not atomic).
  No-ops on a dev build, and on a shadowed install unless the disk is already
  newer (a relaunch installs nothing, so a shadow copy cannot make it unsafe);
  the shadow no-op is logged once per process. `triggerSelfUpdateInBackground`
  (`self-update-service.ts`) starts the same self-update in the background, sharing
  the in-flight guard, so an on-demand caller is never parked behind the full
  check→download→install→verify. Its former on-demand caller, the browser-IPC
  `request-self-update` action, left with the browser engine (PHNX-4101); nothing in
  this repo calls it today.
  When the tick does not exit for a relaunch, it also brings installed host CLIs
  up to their manifest pin (`upgradeOutdatedClis`, `cli-resources.ts`). A
  `clis/<name>.yaml` entry such as `npm: "@phnx-labs/secrets-cli@0.1.8"` is a
  floor: a binary reporting an older `--version` is reinstalled at the pin into
  the npm prefix that owns the copy on PATH (not npm's own prefix, which differs
  on an nvm box), then re-probed; `upgraded` is logged only when the binary on
  PATH answers with the pin. It never installs a missing tool, never downgrades,
  skips a binary that is not an npm install of the pinned package (logged), and
  leaves the old version in place on a failed install. Project-layer manifests
  never drive it: the project layer is whatever `.agents/clis/` sits above the
  daemon's cwd, and an unattended `npm install -g` must not be steerable by a
  checkout, so only the user, system and explicitly added repos count. An install
  that could not finish inside the tick's remaining deadline is deferred to the
  next tick instead of being started and killed mid-write. Async throughout, so
  the event loop is never parked on `npm install`.
  Each implements the
  `DaemonService` contract (`service.ts`) — `id`,
  `start`/`stop`/`restart`/`health()`, plus `intervalMs`/`deadlineMs`/`tick()`
  for the periodic ones — and is `supervisor.register()`ed in `runDaemon()`
  only when `isDaemonServiceEnabled(id)` is true at boot. Every P3 service
  registers alongside the P1/P2 batch before `supervisor.startAll()` except
  `state-dir-check`, which registers (and starts) separately, right after
  `handleShutdown` is declared later in `runDaemon()` — the supervisor fires
  an immediate first tick on registration, and `state-dir-check`'s tick calls
  `handleShutdown` on a marker mismatch, so registering it before that const
  exists would reference it in its temporal dead zone. The supervisor gives
  each periodic service a per-tick deadline race and a per-service try/catch that
  never escapes to the process-wide crash handler. A THROWN tick is recoverable:
  it is recorded (`recordFailure`) and the service keeps ticking on its own
  interval, so a transient failure retries in place and no sibling is disturbed.
  A tick that OVERRUNS its deadline is aborted (its `AbortSignal` fires), logged
  as `tick.breach` with the in-flight snapshot, and recorded as a failure; the
  service then skips its interval ticks until that tick settles and resumes on its
  own, with no process exit (PHNX-4225). A slow tick is not a wedged daemon: on an
  overloaded box the first `getActiveSessions` after boot routinely took 10-15 s,
  and exiting for that made the restart itself the load (see the recordings note
  below). Only a tick still unsettled at 10x its deadline (`HANG_DEADLINES`), or a
  `start()`/`restart()` lifecycle call past `lifecycleDeadlineMs`, is a HANG whose
  promise may never settle, so the supervisor records the cause, flushes it to
  `daemon-health.ts`, logs `tick.hang`, and EXITS the process (code 70); systemd
  (`Restart=always`, `RestartSec=30`, `StartLimitIntervalSec=0`) / launchd
  (`KeepAlive` + `ThrottleInterval=30`) restart the whole daemon within ~30s
  (PHNX-4116). There is NO `parked` state and NO in-process backoff restart — a
  wedged daemon is handed back to its OS supervisor rather than left sitting
  unresponsive. SIGHUP control transitions
  queue on `awaitIdle()` rather than polling, so a requested toggle applies after
  real settlement without another timer owner.
  `getServiceSupervisorHealth()` (`daemon.ts:60`)
  exposes the live in-process `supervisor.health()` map for a future
  same-process reader; a cross-process reader (`agents daemon services`, a
  separate CLI invocation) instead reads the persisted mirror below.
- **A service tick MUST NOT do synchronous IO — the deadline race cannot save
  it (PHNX-3695).** Every service runs on the daemon's single Node event loop, so
  a synchronous `execFileSync`/`readFileSync`/… inside an `onTick`/`onStart` body
  freezes that loop for the whole call. While it is frozen NOTHING else on the
  loop runs — including the supervisor's own per-tick deadline `setTimeout` and
  the browser IPC server's socket handlers, so the deadline race above is
  detection only against ASYNC overruns and is powerless against a sync freeze.
  That freeze is the "accept but never reply" wedge: the trivial `version` probe
  (`browser/ipc.ts:233`) times out only because the loop that would answer it is
  blocked. Use async IO on tick paths — `fs/promises`, and `execFileBounded`
  ([`../exec-bounded.ts`](../exec-bounded.ts)) for a deadline-bounded,
  process-group-killable subprocess spawn. The heartbeat tick
  (`heartbeat-service.ts`) is fully async: it reaps exited routine children via
  `reapExitedRunningJobs` (`runner.ts`) — the async twin of the sync
  `monitorRunningJobs` the off-loop `agents routines list/status` builders still
  use, sharing the reconciliation core `reconcileRunningRecord` and differing
  only in sync-vs-async `fs`/`ps` — whose per-run identity probe is the
  `execFileBounded`-bounded `isPidOursAsync`, whose `host:`-placed runs read
  their remote `.exit` over ssh through `finalizeHostRunAsync` →
  `reconcileTaskAsync` → `sshExecAsync` (a sync `sshExec` there would freeze the
  loop on a 6s host timeout — worse than the local `ps`), plus
  `reapTerminalRoutineProcesses` (also `execFileBounded`).

  **Self-heal broke the rule through its callee, not its tick body.** Its tick
  was async, but `runSelfHeal` → `heal` → `diffVersionResources` byte-compares
  every synced resource in every version home with `readFileSync`. On an operator
  box with 23 version homes that held the loop for over a minute, so
  `attention-notify` breached its 10 s deadline, the supervisor exited (code 70,
  the pre-PHNX-4225 overrun policy), and the restart ran self-heal again 30 s later: 1,843 restarts in under three
  days with a core pinned. The static guard below cannot see a callee, so a
  heavy synchronous pass belongs in a child: the tick now spawns
  `agents __self-heal-run` ([`../self-heal/child.ts`](../self-heal/child.ts))
  through `driveCooperativeChild`, the `__harness-update-run` pattern. It also
  persists the attempt time (`self-heal-last-attempt` in the daemon dir) before
  spawning and skips a tick inside the interval, so a restart cannot re-run it.

  **A slow tick breaches too, even on a live loop.** The since-removed
  monitor engine (PHNX-4241) ran every due poll one after another; with 26
  monitors the first tick after boot ran past its 2-minute deadline, the
  supervisor exited, and the restart found every poll due again: an exit every
  two minutes. Bound a tick's work, not only its IO: cap concurrency, stop
  launching new work partway through the deadline, and leave the rest due.

  **The worst tick-path halt was the event-log LOCK, and it is reached far more
  broadly than one service.** `emit()`/`emitRoutineEnd()` (`feed/events.ts`)
  acquire the log lock with `withFileLock` → `lockfile.lockSync` + `sleepSync`
  (`Atomics.wait`), which HALTS the thread for up to 30s under contention — worse
  than any `ps`. And **every `ctx.log(...)` on every tick** routes there
  (`daemon.ts` `log()` mirrors into `emit()`). The fix is `withFileLockAsync` +
  `emitAsync`/`emitRoutineEndAsync` (`fs-atomic.ts`, `feed/events.ts`): the async
  lock yields with a real timer between retries instead of `sleepSync`. `log()`'s
  event mirror is now a fire-and-forget `emitAsync`; the watchdog tick emits with
  `emitAsync`; the reaper injects `emitRoutineEndAsync` into
  `reconcileRunningRecord`. The other UNCONDITIONAL every-tick halts are fixed the
  same way: **usage-sync + auth-sync** publish through
  `updateFleetSharedDeviceStateAsync` (was a `withFileLock` every tick,
  `fleet-shared-state.ts`); **keychain-reap**'s whole-process-table `ps` is now
  `execFileBounded` (`secrets/reaper.ts`); **watchdog** reads `watchdog.enabled`
  via `getConfigValueAsync` and **browser-task-reap** its idle config via the
  async `resolveBrowserTaskIdleMs` (`device-config.ts`); **tmux-reap**'s per-process
  `/proc/<pid>/environ` scan uses `fs/promises` (`tmux/orphan-reap.ts`).

  The structural guard `guard-no-sync-io.test.ts` (1) scans every `*-service.ts`
  tick/start body for banned sync fs/exec/lock and (2) PINS that each hot tick
  call site uses its async variant (`emitAsync`, `getConfigValueAsync`,
  `await publish…`, `await reap…`) — a full transitive scan is intractable (every
  service imports most of the tree), so it fails if a fix is swapped back to its
  sync twin. Startup and stop/uninstall lifecycle code in `daemon.ts` (pid/lock at
  `:272`/`:343`, `launchctl`/`systemctl`, the `ps` identity probe behind
  `isDaemonRunning`/stop) stays synchronous — it runs in a short-lived
  `agents daemon …` CLI process, never on a tick.

  **Accepted residue (named, not hidden), a follow-up to this PR.** These are
  either µs-scale single-file writes or CONDITIONAL/rare, not the 30s-lock /
  whole-table-`ps` halts above: the tiny `writeHeartbeat` + `log()` daemon-log
  append (µs; `log()` must stay a sync primitive that flushes before a crash); the
  report extraction / transcript archival inside `reconcileRunningRecord`
  (`inferFinalStatusFromLog`, `extractAndSaveReport`, `archiveRoutineTranscripts`)
  which fire ONLY when a run ends; `captureProcessStartTime`'s memoized per-helper
  `ps` fingerprints (keychain-reap, a small pid subset); the browser pid-registry
  `readdirSync`/`readFileSync` (browser-task-reap, bounded by live-session count);
  the opt-in watchdog per-session `statSync`/`readFileSync`; catchup's
  overdue-dispatch (conditional); and
  `session-index`'s synchronous better-sqlite3 (a different class, not a fs/exec
  primitive). Converting these is a separate change and does not block the
  loop the way the halts this PR removed did.
- **Inline declared service, 1 service:** `scheduler` (the routine
  `JobScheduler`) remains outside the supervisor.
  `scheduler` is croner-driven (fires at each job's own cron schedule, not a
  fixed interval) and already has its own live-reload path
  (`schedulerGateTransition`, `bootScheduler`/`stopScheduler`) predating the
  supervisor; folding it in without regressing that reload semantics is
  future work. It has no per-service circuit breaker or state persisted to
  `health.json`; `agents daemon services` infers a coarse `running`/`stopped`
  label from the enable toggle plus whether the daemon process is up.

`runDaemon()` has no bare fixed-cadence maintenance timer left. Its only direct
`setInterval` is scheduler catch-up, which is part of the one remaining inline
declared service and shares that service's live boot/stop/reload lifecycle.
Resource-local timers may still live inside lower-level socket implementations
(for example the browser IPC server's own idle/reap timers); those own the
resource they maintain and are closed by the surrounding service lifecycle. (The
secrets broker's own eviction timers are no longer an example here — that broker
moved out with the standalone `secrets` engine, PHNX-3989 OWN-1.)

Enabled/disabled state lives separately in a `DaemonServicesConfig`, read
and written via `isDaemonServiceEnabled` (`daemon-services.ts`),
`setDaemonServiceEnabled` (`daemon-services.ts`), and
`listDaemonServiceStates` (`daemon-services.ts`) — one `services.yaml` under
`getDaemonConfigDir()`, unioned across both runtime models above.

**Cross-process health is file-backed, keyed by the same `DaemonServiceId`
strings the two models above use.** `cli/src/lib/daemon-health.ts` is the
one health mirror both the daemon (writer) and `agents daemon` /
`agents daemon services` (a separate reader process) use —
`recordSubsystemOk`/`recordSubsystemError` (ok/error streak + last
error/timestamp) and, since RUSH-3193 P4, `recordSubsystemState` (the
supervisor's `idle`/`running`/`stopped` lifecycle state, written on
every transition in `supervisor.ts`'s `startOne`/`stopOne`). A deadline overrun
does not write a new state — it records the cause via `recordSubsystemError`; only
a hang also writes `recordDaemonRestart` and exits (PHNX-4116, PHNX-4225), so
`agents daemon status`'s restart count is real exits, not slow ticks. Only supervised services call
`recordSubsystemState`, so a `SubsystemHealth` record's `state` field being
present is itself the signal `agents daemon services` uses to render
"measured" vs "inferred" (`commands/daemon.ts`'s `buildServiceRows`).
Every update holds a cross-process `proper-lockfile` lease across the entire
read-modify-write and publishes `health.json` by atomic rename. Readers therefore
see complete JSON, and concurrent daemon/foreground writers preserve every
failure-streak increment used by the auto-start circuit breaker.

**Crash model: any uncaught error in the process kills and restarts the
whole daemon, not just the failing service — except for a supervised service's
THROWN tick, which the supervisor's own try/catch isolates (the service keeps
ticking).** A supervised tick that HANGS (unsettled at 10x its deadline) or a
lifecycle call past its deadline is the one case the supervisor deliberately exits
the process for, so systemd/launchd restart the wedged daemon (PHNX-4116); a tick
that merely overruns is aborted and skipped, never an exit (PHNX-4225). A throw that escapes an INLINE service's
local try/catch is still uncaught at the process level. `cli/src/index.ts:96-102` installs the
top-level handlers:

```ts
process.on('uncaughtException', crash('uncaughtException'));
process.on('unhandledRejection', crash('unhandledRejection'));
```

`crash()` logs and then calls `process.exit(1)` (`index.ts:100`), relying on
the OS supervisor (launchd `KeepAlive` on macOS, `systemd Restart=always` on
Linux) to relaunch the whole process — every INLINE service restarts
together, not just the one that threw. The inline scheduler still has no
independent measured health signal.

Two library/callback boundaries are explicitly contained before that terminal
policy. Direct `proper-lockfile` leases use
`logAndContinueOnLockCompromised` because lock refresh happens on the library's
own timer, outside the owning service promise; a lost advisory lease is logged
without throwing from that timer. The process-level `SIGHUP` registration wraps
`handleReload` with `guardSignalHandler`, so a synchronous reload failure is
logged and leaves the daemon and its other services alive.

**Live enable/disable/restart (RUSH-3193 P4).** `agents daemon services
enable|disable|restart <id>` persists the toggle (or, for `restart`, queues
a one-shot restart request via `queueDaemonServiceRestart` —
`daemon-services.ts`) and then signals the running daemon over the same
`SIGHUP` `agents daemon reload` already used (`signalDaemonReload`) — there
is no separate control socket. The daemon's `handleReload` diffs the reloaded
`services.yaml` against the config it booted with and, for any registered
supervised id, calls `supervisor.start(id)` / `supervisor.stop(id)` live — no
daemon restart. It also drains any queued restart via
`supervisor.restartOne(id)`. Most services disabled at boot are not registered
and therefore still require an operator restart to enable. (The former
always-registered `browser-ipc` exception left with the browser engine, PHNX-4101;
the standalone `browser` CLI runs its own IPC service.) The inline scheduler is
re-evaluated directly by `handleReload`; `agents routines start|stop` toggles
that service and sends SIGHUP while the daemon stays up (PHNX-3605).
`agents daemon services` (no subcommand) reads
`readAllSubsystemHealth()` plus `listDaemonServiceStates()` and renders one
row per registered id — state, enabled, consecutive failures, last error —
additively alongside the pre-existing `secretsBroker`/`browserIpc` hosted-
socket fields in `--json` (`commands/daemon.ts`'s `runServices`). An
interactive TTY browser (reusing `dynamicPicker`, `lib/picker.ts`) is
intentionally out of scope for this PR — see the linked follow-up ticket.

**Three cross-device distribution patterns already exist, each hand-rolled
independently — there is no shared abstraction between them:**

- **fanout** (per-peer watch, local process per remote): `watchFleetFeed`
  (`cli/src/lib/feed/watch.ts:133`) and `watchFleetSessions`
  (`cli/src/lib/session/remote/watch.ts:223`) each spawn one `ssh` child per
  peer device and merge their streamed output into one local view.
- **mirror** (every device publishes its own row, readers union all rows):
  `publishLocalFleetStatus` (`cli/src/lib/fleet-status.ts:151`) and
  `writeAuthHealthEntries` (`cli/src/lib/auth-health.ts:373`, called at
  `auth-health.ts:603`) each write a local file that other devices read and
  merge — no single device owns the aggregate. `fleet-shared-state.ts` applies
  the same pattern to an untracked per-device file: headed usage publishers
  write rows that workers merge newest-wins, while auth writes only a safe
  readiness verdict and elects one ready source for an exceptional, async,
  deadline-bounded secret provision. The transport is the `usage-sync` tick's
  SSH exchange (`accounting/usage-sync.ts`, PHNX-4116): a headed box dials each
  dialable peer with `agents __usage-ingest --reply` under a 20-second per-peer
  deadline, pushing its own envelope and storing the peer's reply stamped
  `receivedAt`; a worker never dials. Nothing rides the user repo, so no human
  `agents repo sync user` is involved and no git lock exists to contend for.
The daemon used to host a fourth pattern — **elected-singleton** (first-come
binds, others detect and back off) — for the secrets broker, via
`startHostedBroker`/`shouldTakeOverBroker`. That hosting is gone (PHNX-3989,
OWN-1): the standalone `secrets-cli` engine owns its own broker's process
lifecycle exclusively now, and the daemon only probes its reachability
(`probeSecretsBroker` in `cli/src/commands/daemon.ts`) — there is no longer a
daemon-side elected-singleton example to point at.

**Caching is mostly ad-hoc — the bounded primitive is underused.**
`createMemoryCache` (`cli/src/lib/memory-cache.ts:21-23`) is a bounded
LRU+TTL cache (`max` and `ttlMs` are required, with optional
`fetchMethod` coalescing):

```ts
export function createMemoryCache<K extends {}, V extends {}>(options: MemoryCacheOptions<K, V>): LRUCache<K, V>
```

Only 3 call sites actually use it: `cli/src/lib/session/session-cache.ts:78`
and `cli/src/commands/sessions-picker.ts:142` / `:160`. Roughly 28 other
files implement their own ad-hoc TTL constants plus disk-mirror files with
little or no eviction (`fleet-status.ts`, `auth-health.ts`, `mailbox.ts`,
`linear-cache.ts`, `devices/stats-cache.ts`, `session/presence.ts`,
and others) — so roughly 3 of ~31 cache-like constructs
in this area route through the shared bounded cache. State this as it is:
most daemon-adjacent caching has no shared eviction policy today.

## Diagnosing the daemon: levels, ticks, spans, vitals

The daemon log (`logs.jsonl`, read with `agents daemon logs`) is the record a
crash loop is diagnosed from, so it explains *why*, not just *that*. Every line
is `{ts, level, message, data?}`; `data` carries the structured fields below.
Levels are `debug < info < warn < error`; the daemon writes at or above its
level (`info` by default), set live with `agents daemon logs level <level>` (stored as
`logLevel` in `services.yaml`, re-read on SIGHUP). `log()` in `daemon.ts` redacts
every string in `data` the same way it redacts the message. Every `log()` is a
synchronous append, so `debug` (a line per tick and per span) adds load of its own:
leave it on only while diagnosing, and read a stall at `debug` with that in mind.

| Event | Level | Written by | What it answers |
|---|---|---|---|
| `tick.start`, `tick.ok` | debug | `ServiceSupervisor.runTick` | every supervised tick and its duration |
| `tick.slow` | warn | `runTick` | a tick that used over half its deadline, with the other in-flight ticks |
| `tick.failed` | warn | `recordFailure` | a thrown tick: error, duration, consecutive failures |
| `tick.breach` | error | `recordOverrun` | a tick past its deadline, aborted and skipped until it settles (no exit): elapsed time, every in-flight tick, the synchronous sections that ran during the tick, span totals since the last vitals line, and a vitals snapshot |
| `tick.settled` | warn | `runTick` | an overrun tick finally settling: its duration and how many interval ticks were skipped meanwhile |
| `tick.hang` | error | `exitForRestart` | the restart cause: a tick unsettled at 10x its deadline or a lifecycle call past its deadline, with the same snapshot as `tick.breach`; the process exits 70 |
| `span.slow` | warn | `diagnostics.ts` `spanSync` | one synchronous section that held the event loop past 250 ms |
| `span` | debug | `spanSync` / `span` | every instrumented section and its duration |
| `loop.stall` | warn | `DaemonVitals` | the event loop did not run for over 1 s, and which instrumented sections ran in that window (or that none did: uninstrumented code, or the process was descheduled) |
| `vitals` | info | `DaemonVitals.report`, once a minute | event-loop delay p50/p99/max, process CPU %, RSS, heap, load average, in-flight ticks, and the top sections by total time |

**Recordings encode one file at a time and reap what a dead daemon left
(PHNX-4225).** `ffmpeg` runs detached in its own process group, so a daemon exit
never takes it down. Before the fix every exit orphaned the in-flight encodes and
the next boot started a fresh encode of every queued recording at once: on one
operator laptop that was 21 concurrent full-resolution encodes (about 1,260
threads) and load 200-400 on 18 cores, and attention-notify's first tick after
every boot ran past 10 s, which exited again. The pipeline now records each
transcode's pid and work dir on its ledger row (`TranscodeJob`),
`recoverInterrupted` kills a pid that still writes into that work dir and deletes
the dir before requeueing, and `drain` starts one recording only when none is
active.

**A deadline breach is usually starvation, not a slow service.** The supervisor's
deadline is a timer on the same event loop as every other service and the feed
hub, so a tick that does almost no work still breaches when something else holds
the loop. That is why the breach line carries what *else* was running rather than
only naming the service whose timer fired. Code that runs outside the supervisor
(the feed hub, `getActiveSessions`, the fleet projection) is attributed through
spans: wrap a new hot path in `spanSync(name, fn)` when it is synchronous and
`span(name, fn)` when it is not. Both are passthroughs outside the daemon, where
no span log is installed.

## Service ownership status

**Shipped:** the `DaemonService`/`PeriodicService` contract and
`ServiceSupervisor` (P1); the first 5 services migrated onto it — secrets-broker,
browser-ipc, account-state, session-index, monitors (P2; `monitors`
was later removed with the feature, PHNX-4241); the operator
surface — `agents daemon services` reporting every registered service's
health plus live `enable`/`disable`/`restart` for the supervised set over the
existing SIGHUP reload path (P4); and the remaining 5 interval-driven
services — `watchdog`, `device-probe`, `self-heal`, `keychain-reap`,
`state-dir-check` — migrated onto the supervisor too (P3), bringing the
supervised total to 10. PHNX-3265 moved live-session publishing and webhook
ingress plus every fixed-cadence maintenance loop onto the same supervisor,
and fixed manual periodic restarts so they
replace, rather than duplicate, the timer. PHNX-3695 added `self-update` as a
new periodic supervised service — the daemon previously ran with
`AGENTS_CLI_DISABLE_AUTO_UPDATE=1` forced on (`bootstrap.ts`), so R5 ("the
installed CLI auto-updates") held for the interactive CLI but silently did NOT
hold for the one process that runs unattended for days. `agents daemon
services` now reports measured health for every declared service but one
and infers only `scheduler`. (Two services counted in the P2/P3 history
above, `secrets-broker` and `keychain-reap`, no longer exist in this daemon
at all — they moved out entirely with the standalone `secrets` engine,
PHNX-3989 OWN-1.)
This doc's Current architecture section
above is the source of truth for all of it.

**Still open:**

- **`scheduler` supervision.** `scheduler` is
  croner-driven (fires on each job's own cron schedule, not a fixed
  interval) and already has its own live-reload path
  (`schedulerGateTransition`) that predates the supervisor — folding it in
  needs a `DaemonService` shape that isn't just `PeriodicService`, or a
  dedicated adapter, without regressing that reload semantics.
- **Interactive `agents daemon services` browser** (RUSH-3210) — a TTY-only view reusing
  `dynamicPicker` (`lib/picker.ts`), one row per service with a live-updating
  preview pane (log tail, last error, config) and inline
  enable/disable/restart keybindings, mirroring the session
  browser behind `agents ps` (`commands/sessions-browser.ts`). Deferred out of the P4 PR.
- **Cross-device distribution patterns** (fanout / mirror / elected-singleton,
  described above) formalized as declared `placement` (`every-device` |
  `singleton`) and `distribution` (`fanout` | `mirror` | `local`) fields
  resolved by shared helpers, instead of independent implementations. Usage and
  reserved-auth readiness now share one versioned per-device mirror envelope,
  but the other distribution sites remain independently implemented.
- **Caching** routed through `createMemoryCache` under a declared `cache`
  policy instead of the ~28 files of ad-hoc TTL constants + disk-mirror files
  described above.

## Audit rule: what belongs in a daemon service

Move work behind `DaemonService` when it is autonomous after the initiating
command exits, repeats on a cadence or event edge, owns a long-lived socket or
child, or must expose independent health and restart semantics. By that rule,
the next high-value moves are:

1. **Routine scheduler + catch-up** — the only declared service still inline.
   It needs a lifecycle adapter that preserves croner reload and catch-up claim
   semantics; do not flatten cron into a fake fixed interval.
2. **Fleet feed/session watch coordination** — `watchFleetFeed` and
   `watchFleetSessions` each create an O(devices) SSH fanout per consumer. A
   daemon-hosted coordinator is justified only with one multiplexed local
   subscription surface, bounded buffers/backpressure, and explicit peer
   unavailable envelopes; moving the same fanout unchanged merely relocates it.
3. **Cross-device orphan/session ownership** — the local `session-state` service
   now owns metadata publication, but a peer still must classify its own pane
   and publish that fact. The launching device must not infer peer liveness from
   absent local signals.

Do not turn synchronous launch-time invariants into background services. Host,
cloud, fork, and local session registration currently duplicate metadata
assembly; that belongs in one pure canonical registration API invoked inside
the launch transaction, with the daemon as reconciler/repairer after crashes.
Foreground-only timers (TTY spinners, log tailing, interactive login polling,
one command's progress display) stay with their foreground command because they
have no useful lifetime once that caller exits.

See [PHNX-3193](https://linear.app/getrush/issue/PHNX-3193) for the original design,
[PHNX-3265](https://linear.app/getrush/issue/PHNX-3265) for the composed hardening audit,
and [RUSH-2654](https://linear.app/getrush/issue/RUSH-2654) for the
originating context. Do not describe P3/distribution/caching work as shipped
until the corresponding code lands.