# grove.daemon — the loopback HTTP daemon (multi-repo, SSE)

> ↑ [root](../../../CLAUDE.md)

The FastAPI service that exposes the engine over HTTP. One process serves every repo via the [`RepoRegistry`](../core/CLAUDE.md); responses are [Views](../core/contracts/CLAUDE.md), never engine dataclasses.

## Bind & auth

- **Bind `127.0.0.1` only; never widen it.** `grove daemon serve` takes `--host`/`--port` (default `127.0.0.1:7421`), but binding anything but loopback is unsupported. The constraint is *load-bearing*: it is precisely why endpoint auth is deferred — we trust the user's existing SSH credentials instead of designing token/mTLS. Remote access is an SSH port-forward owned by the [client](../client/CLAUDE.md)'s `SshTransport`; anything more is Tailscale/WireGuard/an authenticated reverse proxy *outside* Grove.
  - **Networked MCP is precedent for this rule, not an exception.** Remote agent harnesses reach Grove because [`grove.mcp`](../mcp/CLAUDE.md) serves Streamable HTTP with its own inbound bearer and still talks to *this* daemon over loopback — a tier that already authenticates callers absorbs the network edge without costing the daemon its invariant. Any future "expose X remotely" request looks for that shape first.
- **Browser auth is cookie → BFF → bearer → daemon.** The webapp's server-side BFF holds the token and attaches the bearer; the token never reaches the browser.

## `/public` — the one namespace served without a bearer

- **A PATH PREFIX is the share boundary, not a scoped principal, and the argument is about the route somebody adds next year.** The alternative — teach `require_session` to recognise a share token and let `/workspaces/{id}/…` serve both — is DRY-er on paper and reuses every existing route. It also makes the boundary an allowlist living somewhere the author of the next route is not looking, so a new `/workspaces/**` route is one forgotten entry away from world-readable. With a prefix, every route under `/workspaces` carries `auth_dep` and always will, and the entire public attack surface is three declarations a reviewer can read at once. It cost three route bodies. `/healthz` is the precedent that this daemon already serves unauthenticated routes; `/public` is that idea with a namespace.
- **`_public.py::PublicWorkspaceReader` is bound to its workspace by `for_token` and exposes NO method taking a workspace id.** That is what makes it structurally impossible for a route to read one workspace while holding another's token — a rule a per-route `if` would only *describe*. It stays synchronous and blocking on purpose: the offload is the route's job, and hiding it behind an `async` seam would take the one fact the route's author most needs out of their view.
- **Every failure is ONE flat 404 (`share_not_found`).** Unknown token, a workspace since unshared, a killed record — a caller must not be able to tell them apart, or the link becomes an oracle answering questions about workspaces the caller holds no token for. This is the deliberate inverse of the `404`-vs-empty-200 distinctions the authenticated routes work so hard to preserve: *there*, telling two absences apart is the feature; *here*, it is the leak.
- **The public transcript route takes no session id — the token names a workspace and the daemon picks the session.** So there is no coordinate an unauthenticated caller can tamper with, and the "which session" question (sub-agent threads, remap candidates) simply does not exist out here. `null` is a real answer: a workspace shared right after creation has no transcript yet.
- **Overview and turns resolve that one session through the SAME manager seam, and turns selects its listing by that id, never `[0]`.** Two locally-correct resolutions silently diverged when a shared cwd held a newer stranger: the overview named one transcript while turns served another. An index can only mean "newest"; it cannot mean "the id we just chose". Test the pair as one contract — separate endpoint tests cannot see an overview naming a different transcript from the one its turns route serves.
- **There is no public SSE and there must not be.** `/events` is a cross-project fan-out carrying every workspace on the host, so it can never be exposed — and it does not need to be, because `after_turn` already gives a follower cheap freshness (the same cursor measured at 426 KB → 46 KB per tick above). The public page polls one composite overview instead of the four requests the authenticated surface makes, since it has no stream to ride.
- **`_turns.py` exists so the two turn routes cannot disagree about what a cursor means.** `_public` cannot import `app` (cycle), and a second copy of `turn_window` would let the public follower and the private one drift on the inclusive-cursor rule that took measurement to settle.

## Mail, and the native owners' control stream

`daemon/mailboxes.py` carries two concerns that share one dependency — the
registry of connected owner workers — and nothing else. Keeping them legible as
two is the point: mail reaches every live agent, owners are how one KIND of
agent is steered.

**Both sit behind the daemon's ordinary `require_session` bearer, and the second credential system that used to gate mail is gone.** Grove is loopback-only for one user, so an agent writing to a peer is the same principal as the human driving the dashboard; a scoped per-peer token bought isolation between parties that were never separate, and the cost was that an interactive terminal session could not participate at all. `POST /mailboxes/messages` therefore takes an addressed message and delegates to `MailboxDelivery`, which calls the manager's own `send_message` — so a terminal agent, a native session and a container's named second agent are reached by one road. `GET /mailboxes/contacts` reconciles every workspace against live tmux, so it goes off-loop like every other manager call.

**The daemon steers a native workspace IN-PROCESS, and the reason is that the owner worker is connected here and nowhere else.** `OwnerSteerClient` (`daemon/mailboxes.py`) is the `NativeSteerClient` `build_app` injects into the `RepoRegistry`, so every manager the daemon mints queues `steer` / `interrupt` / `set_model` frames straight onto the owner's SSE delivery from `/message`, `/interrupt` and `/controls/model` — no network hop, no second daemon. The queues are loop-owned and the manager verb runs on a pool thread, so the client marshals back with `run_coroutine_threadsafe` and AWAITS the result; its loop is bound at lifespan start (`native_steer.bind`), and an unbound client honestly answers "no owner". A CLI or TUI process holds no registry, so its managers default to `DaemonSteerClient`, which POSTs the same three routes with a same-host bearer minted off `auth.json` — the identical rendezvous the client SDK's local backend uses, and NOT `grove.client`, which the engine may not import. **A missing owner is `PaneNotFound` → 409**, never 501: the workspace's live shape has nowhere to deliver and a respawn fixes it, the same remedy a session with no window gets. `/mailboxes/connection` reads `WorkspaceState.native` off the RECORD to admit an owner, never the roster — a later config edit must not turn a running owner away. **That gate is about the control STREAM only:** a terminal workspace is refused there (`native_session_unsupported`) and is an ordinary mailbox contact in the same breath, which is the distinction the previous design collapsed.

**A DAEMON STOP IS NOT A WORKSPACE EDGE, and `aclose` treating it as one killed every native session on the host at each reinstall.** The router used to revoke each owner's registration credential on shutdown, so the worker's reconnect answered 401 and it exited — leaving a pane holding one `ValueError` line and a workspace no verb could steer. Only a lifecycle event (`_INVALIDATING_EVENTS`: killed, paused, …) may unregister; a close now just drops the in-memory bindings so the streams end and the workers reconnect. The worker's half of that contract is in [core](../core/CLAUDE.md). Now that the worker mints an ordinary same-host bearer per connection rather than carrying a launch-issued one, this is structural rather than a rule to remember — there is no longer a credential a shutdown *could* revoke. `OwnerSteerClient.owner_connected` is the registry read the manager uses to decide whether a steer should revive first — it must run ON the registry's loop (the bindings are loop-owned state), and unbound means "no owner" rather than an error, because no owner can have registered before the loop exists.

The primary slot is the empty string, never the configured agent label. A private Unix listener mounts only this router, not the daemon app; it does not widen the TCP bind. Stop the router's streams before its listener. Invalidation must wake idle delivery waiters, not merely set a flag, and an ACK cannot acquire the lifecycle lock held by the send awaiting it. Native input evidence is not model compliance; a `delivered` receipt says Grove handed over the bytes and never that a model read them, so an `unknown` never triggers an automatic retry.

## Knowing when the daemon is running old code

**`LoadedSource` snapshots the package's newest source mtime when `build_app` runs, and `/whoami` reports `restart_required` once disk moves past it.** It checks mtime rather than a git revision or a hash, because mtime covers every way code arrives: a pull into the editable checkout, a wheel reinstall, and a hand edit. It errs toward "changed", which costs a restart, and never misses a real update. The scan is about 3 ms over 263 files and runs off-loop, per `/whoami`, never on a timer. **The field is publish-only by design. The daemon does not restart itself**, because a restart kills the SSE streams and the in-flight lifecycle verbs of every client on the host, and that call belongs to a person. Reason for existing: a tolerant reader (the phase file's) turns a vocabulary skew between the CLI and a stale daemon into silence, and silence is indistinguishable from "nothing happened" (root guide, process lessons).

## Request/response shape

- **Request-body Pydantic models live at module scope, never in a closure.** With `from __future__ import annotations` every handler signature is a string forward-ref, and Pydantic v2's `TypeAdapter` rebuild resolves those against the function's *globals* — it cannot see closure scope. A `class _Body(BaseModel)` inside `build_app()` raises `PydanticUserError: TypeAdapter[_Body] is not fully defined` on the first request. Underscore-prefixed at module top: the underscore keeps the surface daemon-internal, module scope makes the symbol resolvable. Same trap for any inline `Annotated[Union, Field(discriminator=...)]`.
- **`POST /workspaces/{id}/keys` acknowledges one delivered key, not an application action.** It uses the strict shared request model, authenticated workspace namespace and off-thread manager dispatch. Do not alias it to `/interrupt`: remote/paneless agents have a cancellation API but no terminal, and sending Escape cannot stand in for that API.
- **Refusal codes split on whether a retry could ever succeed:** `pane_not_found` → 409 (state-shaped — a respawn fixes it), `steering_unsupported` → 501 (capability gap; a 409 there promises a retry that can never work), `resume_not_supported` → 422 (well-formed request, semantically invalid for the agent kind).
- **The session-remap response deliberately does NOT carry the new session id.** `WorkspaceStateView` omits `agent_session_id` (views never expose it); a client confirms the remap took via the activity/sessions streams, not the response body. A bad `session_ref` resolves engine-side to 404 `agent_session_not_found`.
- **`GET /agents` rows carry a resolved per-agent model catalog (`AgentSummaryView.models`) via `grove.core.agents.resolve_models`, the single catalog seam.** Codex discovery shells out (`codex debug models`), so the whole row list is built in a closure and off-loaded with `asyncio.to_thread(_build)` — never resolve per-agent models on the event loop.
- **Error envelope is `{"detail": {"error": <code>, "message": <text>}}`, and `_grove_error_to_http`'s `code_map` is a linear `isinstance` scan, so SUBCLASS ENTRIES MUST PRECEDE THEIR PARENT.** `BranchError` is the deliberate catch-all for branch subclasses nobody remembered to map; bare `GroveError` falls through to 500. Clients re-raise `ProtocolError(code, message, status)`, so **the codes are a wire contract** — keep them in sync with [contracts](../core/contracts/CLAUDE.md). Mind the two session domains: `agent_session_not_found` (coding-agent transcripts) vs the auth router's `session_not_found` (revoked bearer sessions) — the names collide easily, the domains never do.
- **`repo=` is not a lookup key, it SELECTS CONFIGURATION — so every repo-dispatched route resolves it through one `_known_root` helper.** `registry.get(path)` resolves *that* directory's config cascade, and the daemon then acts on what it finds: `GET /agents` **executes** the configured command (`codex debug models`), and the ticket routes take their base URLs from it while sending the user's token. Without the gate, any directory on the host chooses what a read-only-looking GET runs and where it talks. Two lesser symptoms close with the same 404: a nonexistent path reaching `subprocess(cwd=…)` escapes as a bare **500** (an `OSError` is not a `GroveError`, so no handler sees it), and an existing non-repo directory answers **`200 []`** — a typo'd path masquerading as "this repo has no branches". **The residual is deliberate:** `POST /workspaces` still accepts an unregistered root, because creating the first workspace is how a repo becomes known; the alternative entry is `grove config add-project`.
- **An unrecognized `repo` root is 404 `unknown_repo_root` on every listing, never an empty 200** — a typo'd path must not masquerade as "nothing here yet". Malformed *filters* refuse earlier still: `GET /workspaces?ticket=` (wire format `<provider>:<id>`) is 422 `invalid_ticket_filter` checked BEFORE the repo-root validation, so a bad filter never reports as a bad path.

## Concurrency (the root's `### Concurrency` doctrine, as it lands here)

- **State the rule as "NO ROUTE CALLS A MANAGER METHOD ON THE LOOP", because that is greppable; a per-route judgement about what counts as slow rots.** The routes that look cheap and are not: `GET /workspaces` reconciles against live tmux (~500 ms for 23 workspaces, and it is the most-polled route in the product), the steer routes inject tmux keystrokes with settle delays, and the ticket routes make real upstream HTTP.
- **Lifecycle work reaches its pool through ONE named seam, never a raw executor handle in a route body (`_lifecycle.py::_LifecycleRunner`).** It owns the bounded pool (8 workers, `grove-lifecycle`-prefixed threads — high enough that a fleet doesn't serialize, low enough that it can never become N simultaneous `devcontainer up` builds), the per-workspace key and the shutdown, so those three cannot drift apart across call sites. Everything else is `asyncio.to_thread`, which makes a hand-rolled `get_running_loop()` in a route read as "someone meant a *different* pool" — worth a second look. **The one legitimate bare `run_in_executor(None, …)` is `_PollCoalescer`:** it hands the same `Future` to every joining caller, and `to_thread` returns a coroutine, which cannot be shared or shielded.
- **The restored exclusion keys on the WORKSPACE ID and waits on the LOOP.** A coarser key — one global lock, the manager, the repo — passes a naive "do they serialize?" test and quietly restores the very freeze the offload removed. And an `asyncio.Lock` held by a suspended coroutine costs nothing, while a `threading.Lock` blocked *inside* a pool thread burns one of only 8 workers, so a queue of verbs on one workspace could starve every other workspace out of the pool meant to protect it. `create` is deliberately **unkeyed**: its id is minted *by* the call, so two creates address disjoint workspaces by construction. The damage the key prevents is a `kill` removing a worktree while a `respawn` is midway through `devcontainer up`.
- **A lock map keyed by a lifetime-bounded id is a leak unless entries are reference-counted.** Workspace ids come and go for the life of a daemon that runs for weeks. `_KeyedLocks` counts holders and deletes at zero — no reaper task, and the bookkeeping needs no mutex because every mutation sits between two `await`s on the single-threaded loop. Claim the entry *before* the first await, or a releaser can drop it out from under a coroutine already queued on it.
- **`registry.get` must stay ON the loop even when its consumer moves off it — this is the trap in every offload here.** It mints a project's Manager, and `on_project_registered` schedules the image prebuild via `asyncio.get_running_loop()`; from a pool thread there is none, so every project's prebuild silently never happens. Resolve managers in the handler and close over them, and when you convert a site check what else got dragged inside the closure.

## Listings

- **`GET /projects` is the one listing that is deliberately NOT repo-scoped** — it is the route a client calls to *learn* a repo root, so requiring one would be circular. It returns `RepoRegistry.known_projects()`, so empty-project visibility reaches remote clients that cannot read `known_projects()` in-process the way the TUI's picker does. It sorts server-side because the engine builds its result from set-derived scans whose iteration order is not stable across calls.
- **`GET /sessions` is scoped by a VALUE of its `repo` param, not by a second route.** `repo` given → the project listing; omitted → the host-wide catalog (`SessionCatalog.scan`, including repos Grove has never managed). A sibling `/sessions/catalog` would leave two routes answering one question, and `GET /workspaces` had already set the "omitting `repo` widens the listing" precedent. `limit` is bounded (≤200) and applied after the newest-first sort.

## Bounding a response: which instrument, and why they differ

**One vocabulary, three instruments, and the discriminator is what the CLIENT does with the tail — never how big the payload is.** Picking by size alone is how an API grows four idioms that disagree. Answer these in order; the first "yes" is the instrument.

| Ask | Instrument | Shape |
|---|---|---|
| Does the client FOLLOW something that grows, holding what it already read? | **resumption cursor** | `after_<unit>=<n>` in, `incremental` + `first_<unit>_index` + `total_<unit>s` out; not-honoured ⇒ whole payload + `incremental: false` |
| Does the client SCROLL forward through history it has not seen? | **opaque forward cursor** | `cursor=` in, `{rows, next_cursor}` out |
| Does the client consume a bounded HEAD (ranked, or a list it renders whole)? | **limit + total** | `limit=` in, items + `total` out |

Two invariants bind all three, and they are what make it one vocabulary rather than three:

- **The response always states what it withheld** — `total`, or `incremental: false`, or a `next_cursor`. No silent caps, ever.
- **A response that cannot honour its own bound answers WHOLE and says so**, never partially and silently. `_SseHub.can_replay` is the original; `_turn_window` is the copy.

A route with a **list → drill-in** shape (`/sessions` → `/sessions/{id}/turns`) is not a fourth idiom: it is the limit+total instrument on the list plus a selector on the member, and any new route wanting per-item granularity reuses that pairing rather than inventing one.

The three applications below are that table resolved, with the measurements that decided each.

- **A table a human SCROLLS gets a cursor.** `/usage/sessions` is the one, and its `UsageSessionPageView{rows, next_cursor}` is the shape any future page must copy rather than re-invent. A cursor earns its keep because the underlying set is stable enough to continue through and a refresh appends rows underneath the reader.
- **A RANKED list gets a `limit` plus a `total`, and a cursor there would be a regression.** `/usage/findings` measured **2475 findings / 1.4 MB / 2.5 s** on a real host while the TUI renders the top 5 and the webapp rendered all of them into one `<ul>`. The tail of a ranked list is noise by construction, and the detectors are six independent SQL passes over the whole filtered range — so *every* cursor page would re-run all six for rows nobody reads, making the server busier to send less. `UsageFindingsView.total` is filled in `insights.py` at the one site holding the complete list, so the in-process TUI reader and a daemon-sliced page report the same count; the drill-down that genuinely pages is already there, since each finding's `evidence_filters` reproduces it on the cursor-paginated sessions route.
- **A bare-array response can only take an OPT-IN `limit`.** `/workspaces/{id}/commits` returns `list[CommitSummaryView]` with nowhere to admit a truncation, so a default cap is precisely the silent truncation the parameter exists to avoid — both shipped consumers (the webapp and the client SDK) read it as the complete log. Default stays uncapped; ~186 B/commit measured, so the real exposure is a long-lived branch, not the common case. **Capping it by default requires an envelope first**, which is a breaking shape change for both.
- **`GET /workspaces/{id}/diff` is the head+drill-in pairing, not a fourth idiom.** The file list is bounded by BYTES (1 MB, `truncated` when cut) and `?path=` is the member selector — the same shape `/sessions` → `/sessions/{id}/turns` already has, which is why per-file granularity needed no new vocabulary. Unlike `/commits` this response is an envelope, so it *can* admit truncation and therefore *can* carry a real default cap. The patch crosses RAW; the reasoning for parsing nothing lives in [core](../core/CLAUDE.md).
- **A cap that cannot be seen is a lie, so the withheld count crosses the wire.** `total == len(findings)` is the client's "nothing was withheld" signal. **Do not read a short list as a quiet range** — that inversion is exactly what a silent cap produces.
- **`/turns` is bounded on TURNS and on nothing else, and that is now the whole contract.** It takes `last` (a tail of turns) / `after_turn` (a cursor), both defaulting to `None` = the entire session. The payload is dominated by per-entry text — measured **378 KB for SIX turns** — so a row cap is a weak bound on bytes, and it is deliberately the only one: the per-payload character caps that used to sit under it are gone (see [contracts](../core/contracts/CLAUDE.md)). **Before "adding a limit" here, note that the last one silently cut a 7092-character agent reply to 4000 and that is the bug this route was fixed for.** A caller wanting fewer bytes asks for fewer turns.
- **A live transcript follower needs RESUMPTION, not pagination, and the precedent to copy is the SSE replay contract rather than the usage cursor.** `/usage/sessions` pages *forward through history* a reader has not seen; a transcript follower already holds the history and wants only what changed, which is exactly `Last-Event-ID` → `can_replay` → *replay or full snapshot*. So `after_turn=<n>` reuses those semantics (`_turn_window`): honour the cursor, or answer whole with `incremental: false` and let the client replace. Forcing the `{rows, next_cursor}` shape onto it would have been the second pagination style, not the reuse.
- **The cursor is an ORDINAL and it is INCLUSIVE, and both halves are forced by measurement.** A turn's identity is its position in a forward walk of an append-only transcript — appending extends the last turn, a new human prompt starts turn N+1 — so an integer is a sufficient key and no id field was needed on `SessionTurnView`. But turns are append-**mostly**: measured over 45 s of a live agent, **6 of 7 turns stayed byte-identical and only the tail moved**, so an *exclusive* cursor would freeze a half-finished turn on screen for the rest of the session. Re-sending the client's own last turn is the minimum correct delta. **`after_turn == total` is not a gap** but an empty incremental page — treating an off-by-one as desync would make the common "nothing happened" tick the most expensive request on the route.
- **The win is bounded by the tail turn's size, so quote 9× rather than the amplification figure.** Measured on a live workspace: a progress tick re-downloaded **426 KB to gain 269 B**, but the tail turn is **10.9%** of the response, so a cursor costs 46 KB — ~12.6 MB/min → ~1.4 MB/min. **The remaining fat is per-turn, not per-session:** `file_edit` is 35.5% of the payload and `todo` 29.6% (62 whole-list snapshots in one session, ~2 KB each, only the last one current).
- **The todo share is now measured AND fixed.** A `?last=40` window on a real 133-turn session: **3,810,728 bytes total, 1,100,957 bytes (28.9%) in 119 separate todo entries, of which exactly ONE is current** — a todo write is a full-list rewrite, so every earlier one is pure waste on every read (`file_edit` 208 KB / 5.5% and text 225 KB / 5.9% of the same window, for scale — tool bodies at 2,119 KB / 55.6% remain the dominant share and are a deliberate separate decision, not touched here). `contracts.sessions.SessionDetailView._drop_superseded_todos` nulls every `todo` payload but the newest across the whole windowed turn list (not per turn — 53 rewrites landed in one turn on a real session), keeping the entry itself (role + text) so position and count never shift; see [contracts](../core/contracts/CLAUDE.md) for why this is a lossless projection rather than a capped field and therefore carries no `todo_truncated` signal. No wire-schema change — `todo` was already nullable — so no client regenerate was required.
- **Both turns routes window through the one helper, and only the workspace-scoped one takes a cursor.** `/sessions/{sid}/turns` browses history for a session that may belong to no workspace and mostly is not running, so nothing follows a growing tail there; it still reports `total_turns`/`first_turn_index` so `last` stops being a silent truncation. The route reads the WHOLE turn list and windows in the daemon rather than pushing `last` into the adapter — the adapter memoizes per `last`, so a client alternating between a tail and a cursor would otherwise hold two projections of one parse, and only the complete list can report an honest `total_turns`.

## Uploading a file: two steps, and JSON rather than multipart

`POST /workspaces/{id}/attachments` stores one file and returns its id; `POST /workspaces/{id}/message` then names ids in `attachments`. Three decisions worth not re-litigating.

- **Two steps rather than one multipart message body.** An upload that fails should fail on its own rather than taking a typed message down with it, and the browser's requests reach the daemon through a BFF proxy that reads bodies as **text** — a binary multipart body does not survive that, and widening the proxy would make every future route there a content-type question.
- **Base64 in JSON, with the cost stated rather than hidden.** It inflates a payload by a third where multipart would not; multipart costs a server dependency, a proxy change and a second content type on a surface that has exactly one. For files a person attaches to a chat message that is the cheaper side, and `AttachmentStore.MAX_BYTES` is what keeps "a chat message" true. The `_ENCODED_CAP` on the wire is deliberately looser than the real limit: it only stops an oversized string being allocated and decoded before the engine can check the DECODED length.
- **The response's `path` is display-only and the request never accepts one.** A path chosen by the caller is the caller choosing which file the agent is told to open; ids resolve against one directory the engine owns. `path` still crosses because it is what the agent will actually read, already translated into its namespace — a container workspace's is a container path.

## Session history endpoints

- **Usage pagination annotations stay module-scoped and policy clamps in the
  handler.** A `Query(le=closure_value)` annotation makes Pydantic's forward
  reference unresolved during OpenAPI generation; keep only stable validation
  in the module-level annotation and apply `cfg.usage.max_sessions_per_page`
  inside the blocking service call.
- **A single-flight initiator is shielded too.** Shielding only joiners still
  lets the first disconnected request cancel the shared executor future; every
  waiter must await the same shielded future. Only the underlying future's
  completion callback may release the slot; a cancelled initiator does not mean
  its executor job stopped. The daemon owns one usage service for the process
  and closes it from lifespan shutdown, off the event loop. Independent owners
  are failure-isolated so one broken close cannot skip the remaining cleanup.

- **Session history is fetch-on-demand, never pushed over SSE** — turns are unbounded; the stream stays small. The scans run off the loop and are bounded to one workspace's cwd via `SessionExplorer.for_workspace`. `session_id` on the turns route must be the full id — prefix resolution stays a CLI affordance. **The turns route also serves a sub-agent THREAD id:** listing match first, then `SessionExplorer.subagent_turns` fallback (see [core](../core/CLAUDE.md)), then the 404 — fleet rows on the activity wire carry thread ids as their `session_id`.
- **The host-wide catalog is event-maintained, and metadata reads are not liveness reads.** `_CatalogMemo.snapshot()` serves retained metadata to gallery/find without a `/proc` walk; `rows()` refreshes liveness only for a listing that consumes it. Reject non-diagram events before touching the gallery's session join. Turn-count work is single-flight per completed generation, with explicit completeness distinct from a zero changed count; register Future callbacks outside the lock because an already-completed Future invokes them synchronously.
- **Measured on a host with 544 sessions: cold `GET /sessions` 1.39 s, warm 2.7 ms (50 rows) / 6.3 ms (200 rows); the drill-in 78 ms**, with zero full transcript parses. The cold time is **dominated by `SessionCatalog._workspace_maps`, not by the head reads**: it calls `WorkspaceManager.list()` per known repo, reconciling status against live tmux (~500 ms for 23 workspaces) though the catalog only wants each state's id/cwd/kind. Head reads cost ~166 ms, the `/proc` liveness walk ~72 ms, an unauthenticated mewbo `discover_all` ~98 ms. If cold latency ever matters the fix is a cheaper unreconciled listing seam engine-side, not an index.
- **`/gallery` joins `_CatalogMemo.snapshot()` metadata at its current event generation, without re-probing process liveness.** Items are addressed by an opaque id (a hash of the path) so no path ever enters a URL; `GET /gallery/{id}` serves the XML, `GET|POST /gallery/{id}/preview` reads and writes the browser-rendered PNG keyed by the item's content digest, and a POST whose echoed digest disagrees with the scan's is a 409, never filed against bytes it does not depict. Measured cold `GET /gallery` 2.8 s (it is the catalog's cold cost plus one `git ls-files` per worktree), warm 7 ms.
- **`GET /sessions/{sid}/turns?kind=&cwd=` is the workspace-LESS drill-in — the one route that resolves a session without a workspace.** Most sessions on a host were never launched by Grove. It resolves by `(kind, cwd, session_id)`, which is what a catalog row carries and what an adapter's `read_turns` needs. **`cwd` is a plain `str`, not a `Path`, on purpose:** the adapters match a *recorded* cwd by string, so the value must round-trip byte-for-byte from the row that produced it. A row whose head read never recovered a cwd is not drillable. Any coordinate mismatch is one typed 404 — never another session's transcript. The workspace-scoped route stays the right one for a workspace with a pinned transcript config dir, since the host scan only sees what the daemon's own environment reaches.
- **`GET /workspaces/{id}/todo` distinguishes 404 from an empty 200 and must keep doing so:** 404 `agent_session_not_found` means the workspace has no session at all, while a session that has called no todo tool yet is a real 200 with an empty `TodoListView`. **"No session" is NOT "no `agent_session_id`"** — a codex workspace mints no id by construction, so keying the 404 on the id alone made an entire provider's todo unreachable from every client while the adapter parsed it fine. There is deliberately no write half (the todo axis is pull-only, argued in [core](../core/CLAUDE.md)); the issueops sticky-comment publisher calls `WorkspaceManager.latest_todo` in-process rather than over HTTP.
- **`GET /workspaces/{id}/runtime-facts` is GONE, and the reason it existed is the lesson.** It spawned a `codex app-server` child per request to fetch the model's context window — a number that turned out to be on disk in every Codex rollout since 0.98 (`token_count.info.model_context_window`), and which the app server cannot answer for a LIVE session at all (`thread/resume` on a thread an open TUI holds refuses with `already has an active writer`, measured 0.154.0). Context-window pressure now rides `AgentActivityView.context` on the ordinary ~1 Hz stream at zero subprocess cost. **Before adding a route that spawns anything, grep the artifacts the poll already reads for the fact you are about to buy.**
- **`GET /workspaces/{id}/history` is the ONE per-workspace route that does NOT gate on the workspace existing, and the sibling gate is what broke it.** `_manager_for` is right for `/todo`, `/queue` and every other read here, because those describe a live workspace. This route serves a DURABLE store whose whole purpose is answering for a workspace `kill` has deleted — so the copied gate 404'd exactly the case the feature was built for, verified against a real tombstoned record. The refusal is the STORE's instead: an id with nothing recorded returns an empty view, identical to a live workspace predating the store, and there is deliberately no 404 separating "unknown id" from "nothing recorded" — a reader holding an id off a usage row cannot tell those apart and does not need to. **This is the inverse of the 404-vs-empty-200 distinctions the routes above work to preserve; check whether your route's subject outlives its record before copying a neighbour's gate.** Its six original tests all passed while the motivating case 404'd, because every one of them read a LIVE workspace — the missing test was create → record → kill → read.
- **`GET /workspaces/{id}/phase` deliberately returns 200 `null`, not 404, for "the agent has not reported".** The 404-vs-empty split the todo route uses does NOT apply: an unreported phase and workspace-not-found are different facts a fleet watcher must tell apart, so `_manager_for` alone gates the 404 and the phase read never raises on "nothing reported yet". The write side is the manual/operator counterpart to the agent's own file-channel report; the agent itself never calls this route (see [mcp](../mcp/CLAUDE.md) on why the file is the primary channel).

## Activity stream (SSE)

- **`GET /events` is a hand-rolled `StreamingResponse`, not `sse-starlette`.** Any new dependency needs `uv lock`, which re-resolves the graph and 404s on a dead wheel already in it. SSE framing (`id:` / `event:` / `data:`) is ~3 lines. Documented in OpenAPI via `responses={200: {"model": DashboardEvent}}` so webapp codegen still picks up the envelope.
- **A quiet `/events` beats with a NAMED `heartbeat` frame, and BOTH of its framing rules are load-bearing.** A `: keepalive` **comment** holds a proxy connection open but fires no `EventSource` listener, so the webapp's `lastEventAt` only advances on real fleet activity: a quiet fleet force-reconnects and refetches a full snapshot on every backgrounded-tab return, and the self-heal cannot tell a **dead** stream from a **quiet** one — the one distinction a heartbeat exists to provide. **(1) The frame MUST carry `data:`.** The HTML spec's dispatch algorithm returns early on an empty data buffer, so `event: heartbeat` with no body is delivered and silently **dropped**. *"This frame carries no payload"* is therefore never a reason to omit the body — and because it looks like one, a test pins the body rather than trusting prose. **(2) It must NOT carry `id:`.** The last-event-ID buffer is not reset between events, so an id-less frame leaves `lastEventId` — and the `Last-Event-ID` reconnect header — at the last REAL event; stamping a beat would move the client's resume point onto a frame `_SseHub`'s ring never held and the reconnect would ask `can_replay` for a range it cannot honour. `_ID_LESS_KINDS` derives this from the event KIND inside `_sse_frame`, never a caller flag, so no call site can emit an id-carrying heartbeat by accident. `_HEARTBEAT_INTERVAL_SECONDS` (15 s) is coupled to the webapp's 20 s `STREAM_STALE_MS` — move either alone and the heal silently breaks in one direction. **The pane stream deliberately KEEPS its comment:** `useWorkspacePane` has no `lastEventAt`, no stale bound and no `visibilitychange`, so a named beat there would be a producer with no consumer. Check the consumer before adding one. Rider: a route's OpenAPI **description** is embedded in `types.gen.ts` as JSDoc, so editing prose here alone drifts `codegen:check` — a webapp regen is a landing step for a docs-only daemon change.
- **`_sse.py::_SseHub` is the sync → async bridge.** It subscribes once to the *synchronous* `ActivityService` bus; each `DashboardDelta` hops to the loop via `loop.call_soon_threadsafe`, then fans out to per-connection **bounded** queues that **drop-oldest** on overflow — a wedged browser never back-pressures the engine. A shared ring buffer holds the last N events for `Last-Event-ID` replay; when the gap exceeds the buffer, send a fresh `snapshot` instead.
- **The lifespan poll runs `ActivityService.poll_once()` in a thread** (blocking git/tmux I/O). That is why the service's `seq` is an `itertools.count` (atomic under the GIL) exposed via `next_seq()` — both the pool thread (deltas) and the loop thread (snapshots) stamp ids from it.
- **`poll_once()` has TWO independent triggers — the lifespan timer (2 s) AND every hook-ingest POST — and they share ONE `_poll_coalescer.py::_PollCoalescer` so they can never run concurrently.** `POST /hooks/agent-events` is the agent's native http hook, fired per tracked event with **no debounce**, so active fleet coding is a burst of POSTs. The per-workspace fingerprint diff scopes the *wire* cost but NOT the *computation* — every trigger still walks every repo/workspace — so N independently-scheduled triggers mean N full-fleet scans running AT ONCE on a thread pool, which is what "many cores pegged" looks like. A caller arriving mid-run `await`s that SAME `asyncio.Future` (via `asyncio.shield`, so a disconnected HTTP client's cancellation never kills a run other callers still want); the check-and-assign has no `await` between them so it cannot race on the single-threaded loop. **A THIRD trigger must go through the same coalescer or the invariant breaks silently.**
- **The poll's live-consumer gate is `_audience.py::_PollAudience`, and the ROOM IS BIGGER THAN SSE.** "Consumer" is not "`/events` connection": the notification broker and the issue-ops status publisher ride the same delta bus and are exactly the ones that matter when no dashboard is open — a push notification exists to reach a user who is *not* watching, so gating on SSE alone would silence notifications on precisely the idle host they were built for. Hence a **count**, with `_SseHub` joining per connection and the always-on two joining for the process lifetime: **a daemon with notifications or issue-ops configured polls continuously, by design**, and the second browser tab closing doesn't silence the poll for the three still streaming. Gate *before* the tick, not after the interval sleep, or a user opening the dashboard waits out a full interval for the first frame. The hook-ingest trigger stays ungated — it is an edge, not periodic work.
- **Do NOT add an `is_disconnected()` poll to the generator.** Starlette's `StreamingResponse` runs its own disconnect watcher that cancels the generator (the `finally: unsubscribe()` runs then). A manual poll only adds deadlock surface.
- **The live focused pane is a SEPARATE per-workspace stream (`GET /workspaces/{id}/pane/stream`), NOT a kind on `/events`.** `_pane_stream.py::_PaneStreamer` self-paces (~1 Hz), captures via the same best-effort `peek_pane` seam the one-shot read uses, and **diff-guards** — a frame only when the ANSI changed, else a `: keepalive`. Muxing it into `/events` would let pane frames dominate the bounded queue the activity deltas were sized for (cross-project, fanned to many, ~2 s vs per-workspace, per-focus, ~1 Hz). **The visibility + WORKING gate is the CLIENT's job, by design:** opening the stream IS the subscribe hint, and the client already holds each session's `AgentActivityState`, so recomputing that policy per pane-tick server-side would duplicate it. An open stream self-throttles on its own write, so a slow client never back-pressures the engine.
- **A `session_activity` frame is a STRICT SUPERSET of `GET /workspaces/{id}/peek` minus the pane, so a client polling peek beside a healthy stream is paying twice for one answer.** Both sides run the *same four* git reads (`ahead_behind`, `diff_stats`, `recent_commits`, `dirty_file_count`) — `manager.peek` per request, `ActivityService._workspace_activity` per workspace per tick — and the tick is unconditional whenever the audience is non-empty, which an open `/events` connection alone guarantees. So **"should the working-tree counters ride the stream" is already answered yes**; the only open question a future reader should ask is why anyone is still polling. The stream's copy is also the *more correct* one: it computes against `_live_branch`, where peek still uses the create-time `state.branch`. Two exceptions to the superset, both deliberate: the pane pair (`agent_snapshot`/`snapshot_taken_at`), which has the dedicated stream above, and `WorkspaceActivity.branch`, which the engine derives per tick and `WorkspaceActivityView` then **drops** — so every client renders the stale recorded branch off `state.branch`, exactly the trap `_live_branch`'s docstring describes. `tests/core/contracts/test_activity_views.py` pins the superset (nothing in the type system does), because the moment a field lands on peek and not on the activity view, every client's poll-gating silently goes stale.
- **`GET /activity` and the stream's connect-time `snapshot` frame are the same `DashboardSnapshotView` off the same `activity_service.snapshot()` call** — a client that opens `/events` has already been handed the snapshot and needs no separate fetch. Measured on this host: the payload is ~85% `sessions` and ~70% *sub-agent fleet entries* (15 of 17 sessions on one row); **empty projects are 4.7% of it**, so dropping them to slim the wire trades a documented affordance (`known_projects()` exists precisely so a zero-workspace repo stays visible to "create a workspace here") for noise-level bytes. Slim this by measuring the `sessions` array, never the project list.
- **`workspace_changed` carries NO workspace payload, and attaching one would be a regression rather than an optimization.** It is emitted from `ActivityService._bridge_callback`, a *synchronous* subscriber running on whichever `grove-lifecycle` pool thread ran the verb — so building a `WorkspaceActivity` there would put four git subprocesses plus a per-session transcript parse inside the verb's own critical path, on a pool of 8, to deliver data the next `poll_once` hands over free within 2 s (a new or changed workspace fails the fingerprint compare by construction). For `killed` it is not merely expensive but impossible: the record is deleted on the next line. **The client's lever is `detail.event`, which already names the lifecycle kind**, and reading it removes the refetch instead of shrinking it — `killed` → drop the row locally, `updated` → one targeted `GET /workspaces/{id}` (title/description/`ticket_refs` are the fields NOT in the activity fingerprint), everything else → nothing, because status *is* in the fingerprint and the next `session_activity` carries the whole row. `message_sent`/`question_answered`/`control_invoked` are steering acks with no state change at all. **The one thing only this kind can express is a DELETION**, since `poll_once` signals a vanished workspace by ceasing to emit for it, which no delta can say.
- **Test the infinite stream by driving the ASGI app directly.** Build the `scope`, feed `http.request`, capture the first `http.response.body`, return `http.disconnect`. The sync `TestClient` deadlocks on it and httpx's `ASGITransport` buffers the whole never-ending body.

## Notifications and issue-ops wiring

- **Both subscribers are pure engine bound in the lifespan, so the daemon adds zero routes, wire shapes or helpers for either.** `NotificationBroker.from_config` / `TicketStatusPublisher.from_config` return an object or `None`, are `bind(activity_service.subscribe)`ed right after `sse_hub.start`, and closed on shutdown; `build_app(..., notification_broker=…, status_publisher=…, issue_ops_engine=…)` are the test-injection seams, mirroring `auth_store`. **The one thing the daemon hands the publisher beyond the registry is a `transcript_probe` closed over `_CatalogMemo.find`** — the publisher's finale links a transcript only where a reader could open it, and answering that from anywhere but the reader itself lets "we linked it" and "the route 404s" drift apart. It costs no scan of its own (the memo is the same one `/sessions/{id}/turns` uses) and is asked once per workspace death. Designs live in [notifications](../core/notifications/CLAUDE.md) and [issueops](../core/issueops/CLAUDE.md).
- **The assignee poller is the one lifespan-bound object that rides NO bus, and therefore does NOT `audience.join()`.** `AssigneePoller.from_config` / `bind()` / `close()` mirror the publisher's shape, but it consumes no `DashboardDelta` — it drives its own timer against the trackers — so joining the audience would hold the fleet's per-workspace git/tmux scan open for work it never reads. It is also the only daemon-owned worker whose `close()` uses `shutdown(wait=False, cancel_futures=True)`: a tick can be midway through a `create`, which for a container workspace is minutes of `devcontainer up`, and shutdown must never block on side-effecting work already in flight. It adds no routes and no wire shapes; design in [issueops](../core/issueops/CLAUDE.md).
- **`POST /issue-ops/events` answers 202 = accepted-and-acted**, because the create/steer/reply already happened synchronously in the engine (off the loop — it does blocking git/tmux/network I/O). An ordinary refusal comes back as an `IssueOpsOutcome`, never a raised error.

## Session lessons

(Folded into the sections above. Add daemon-only HTTP/SSE/auth lessons here; engine facts go to [core](../core/CLAUDE.md), wire shapes to [contracts](../core/contracts/CLAUDE.md), transport to [client](../client/CLAUDE.md).)