AGENTS.md · git:20261001.2b2cd96 · 2026-10-01 · sha256 7f1f33b22c025f42
AGENTS.md git:20261001.2b2cd96B
Immutable. This exact content never changes and is served at /api/v1/blob/7f1f33b22c025f42.
# Haive
Deterministic multi-CLI orchestration and AI agentic workflow utility. Reimplements a legacy markdown-driven Claude Code workflow, the autonomous `/workflow` implementation loop, and a sandboxed local environment replication step set as a deterministic web project. Agentic CLI invocations only happen for parts that genuinely need reasoning. Everything else runs as TypeScript step modules with web forms.
The legacy markdown step content has been ported into TypeScript step modules under `packages/worker/src/step-engine/steps/`. The original source archive is no longer vendored in the repo.
## Stack
- pnpm workspace monorepo, turborepo
- Node 26, TypeScript 5.7, ES2024 target, NodeNext modules
- Hono 4 REST API on port 3001
- Next.js 16 + React 19 + Tailwind 4 web UI on port 3000
- Drizzle ORM on PostgreSQL 18, postgres.js driver
- BullMQ on Redis 8 (noeviction policy is required)
- Uploaded repo archives live in the `haive_repos` named volume (shared by api and worker); Mailpit for dev SMTP
- Docker Compose orchestrates everything; the dev stack is driven by `scripts/dev.sh` (aliased `pnpm docker:dev`), which wraps `docker compose up` with the dev override and GPU layering
- clawker (Apache 2.0 Go binary) wrapped via child_process in `sandbox/clawker-client.ts`. The binary is installed only when the worker image is built with `CLAWKER_RELEASE_URL` plus a matching `CLAWKER_SHA256`; nothing in this repo sets either, so the binary is absent from the shipped image and the wrapper is inert until someone wires it
## Monorepo layout
```
haive/
package.json pnpm workspace root
pnpm-workspace.yaml
turbo.json
tsconfig.base.json
docker-compose.yml postgres, redis, mailpit, api, worker, web
docker-compose.dev.yml dev override: port exposure, hot reload, db-migrate one-shot
packages/
shared/ @haive/shared types, schemas, crypto, config, logger
database/ @haive/database Drizzle schema and migrations
api/ @haive/api Hono REST + WebSocket terminal proxy
worker/ @haive/worker BullMQ workers, step engine, sandbox manager, CLI adapters
web/ @haive/web Next.js Conductor-style UI
tests/
e2e/ Playwright specs
fixtures/
```
## Package boundaries
- `@haive/shared` holds zod schemas, types, crypto, logger, ConfigService, SecretsService, and UserSecretsService. May be imported by every other server-side package. May import `@haive/database` for the Drizzle schema namespace used by the secrets services; the actual db client is injected at runtime via `initialize(db)` so shared never instantiates a connection.
- `@haive/database` exports the Drizzle schema and a `createDatabase(url)` factory. Instantiated only by the api and worker packages.
- `@haive/api` is HTTP + WebSocket only. It must never spawn child processes for CLI execution; that responsibility lives in the worker.
- `@haive/worker` owns BullMQ queues, the step engine, the CLI adapter registry, the sandbox/clawker wrapper, and the dispatcher priority chain. It holds no HTTP routes.
- `@haive/web` consumes only the public REST API of `@haive/api`. It must not import from `@haive/database` or `@haive/worker`.
## Architecture summary
Three queues carry the core pipeline (`QUEUE_NAMES` in `@haive/shared` declares more —
repo, bundle, runtime/ide ensure, ddev control, usage and PR polling, and plan mirror):
- `task-queue` runs the orchestrator. One job per task. Owns the step machine and persists every transition to Postgres.
- `cli-exec-queue` runs the sandbox worker. One job per CLI invocation. Spawns a per-task Docker sandbox via `docker run`/`create` (`sandbox/docker-runner.ts`) — NOT clawker (clawker backs the persistent terminal/login containers in `sandbox/container-manager.ts`); captures piped stdout/stderr and streams to a Redis Stream (`cli-stream:<invocationId>`). Steerable Claude-family runs keep stdin open (`-i`) so a user steer reaches the CLI mid-run.
- `env-replicate-queue` runs Dockerfile builds for environment replication.
- `plan-mirror-queue` reconciles the repository-backed plan snapshot: a `refresh` rewrites
`.haive-data/plan.{json,md}`, a `save` also commits and optionally pushes, and a 10s
scheduled sweep drains `plan_mirror_state` rows whose `written_revision < revision`.
State source of truth is Postgres. Every step transition, every CLI invocation, every form submission is a row. Crash recovery reads the last row.
## Step engine
Every legacy markdown step becomes a `StepDefinition<TInput, TOutput>` with four phases:
1. `detect` runs first. Pure or shells out via clawker. No LLM. Always runs.
2. `form` returns a `FormSchema`. The web UI renders the schema and the user submits values.
3. `llm` is optional. Spawns a CLI invocation through the dispatcher. For CLIs without native sub-agents the splitter emits a sequential script.
4. `apply` runs last. Writes outputs to `task_steps.output` and to actual files under the workspace.
Step lifecycle: `pending` to `running(detect)` to `waiting_form` to `running(apply)` to optional `waiting_cli` then back to `running(apply)` to `done` or `failed` or `skipped`.
### Fix loop
A downstream step that finds a blocking defect returns `loop_back` instead of failing: the
queue bumps the round, records the diagnosis as a `fix_loop.requested` task event and
re-enters at `FIX_LOOP_TARGET_STEP_ID` (`_fix-loop.ts`), which is `07-phase-2-implement` and
is HARDCODED. Nine steps emit one (07b, 07c, 08, 08a, 08b, 08c, 08d2, 09), and 07 is the only
reader of `loadFixLoopDiagnosis`, so a round in which 07 does not run is a round in which
nobody reads the defect.
**The target is fixed; what varies is whether 07 runs.** In DAG mode
(`06b-sprint-planning.output.mode === 'dag'`) 07 used to skip at EVERY round, so every
loop_back wrote a diagnosis nothing consumed and re-ran the whole review chain against
unchanged code until the round cap. MEASURED on task 681f0f99: 07 `skipped` at rounds 1-4,
three `fix_loop.requested`/`fix_loop.started` pairs all sourced from `08b-test-management`,
none consumed. That was permanent, not transient — the loop enqueues only the target and the
forward walk never goes backwards, so 06b never re-runs and its round-0 mode stands for the
life of the task. `shouldRun` splits by ROUND instead: `06c-dag-execute` owns the initial
build (round 0 still skips), 07 owns every fix round. That is also what makes
`guidance-context.ts`'s standing claim — "a DAG-mode run gets the guidance only once the fix
loop routes back to 07" — true rather than false.
Repointing the target at `06c-dag-execute` is NOT the alternative, for four structural
reasons. `resolveDagPhase` derives its cursor as `levels.find(l => l.checkpointedAt === null)`,
so after a successful build every level is checkpointed and it resolves without dispatching an
agent; the per-issue worktrees were removed at checkpoint; `buildCoderPrompt` has no diagnosis
channel; and `PATH_REQUIRED_TARGETS` (`execution-paths.ts`) is a `Record<string, string>` that
cannot hold two targets for one emitter, with a second independent copy of the same mapping in
`_prompt-defect.ts`. The tree a fix pass must edit is `01-worktree-setup`'s integration
worktree in BOTH modes, which is exactly what 07 already uses.
**A round must not be spent on a failure no agent can repair, and that verdict has to be
reachable without a human.** `08b-test-management` is the worked example. Its fix loop is
driven by `testsPassed`, and `null` there means "not a defect" — `fixLoop.evaluate` returns
nothing, so no round is spent and a `degradedNote` reaches the person instead. Two guards
produce it, and they cover different classes: the enumerate guard (`buildCollectCommand`, a
`--list` whose non-zero exit means the runner loaded nothing) catches a LOAD failure, while
`classifyTestEnvFailure` (`_test-env-guard.ts`) reads the RUN's own output for an ENVIRONMENT
failure. The second exists because the first is structurally blind to it: Playwright builds its
list task set without `createGlobalSetupTasks`, so `--list` never runs globalSetup — MEASURED,
a container with no browser binaries listed 50 tests and exited 0 while the real run died at
`chromium.launch()`, and five consecutive fix agents each re-derived that diagnosis and wrote
it to a field nothing read. The classifier keys on error IDs Playwright raises itself
(`registry/index.js`'s "Executable doesn't exist at", `registry/dependencies.js`'s "Host system
is missing dependencies"), never on the decorative install box beside them, and claims nothing
for a framework whose output has not been measured. It applies from pass 0, unlike the
enumerate guard — a browser missing from the container is never something the tester's own
first pass could have caused.
**Each fix pass is a fresh CLI process, so what earlier passes concluded has to be carried
explicitly.** `priorPassNotes` (08b) does it inside one step's loop and `loadPriorFixContext`
(`_fix-loop.ts`) does it across rounds; both dedupe with the ledger's `contentFingerprint` and
share the same budget (400 chars per entry, 4000 per block). That dedupe collapses a verbatim
repeat but NOT two rewordings of one finding — the same limit `review_findings` measured for
prose keys — so the cap, not the dedupe, is what bounds a loop that keeps re-deriving itself.
`contentFingerprint` normalises before hashing — uuids, PATHS (everything from the first
slash) and all digits — so texts differing only in those are ONE entry. That is the intended
behaviour and MEASURED it is not over-collapsing: across all 170 real ledger entries and
diagnoses on the dev install it yields 148 fingerprints, 21 of the dupes byte-identical (the
case it exists for) and exactly ONE pair with differing text — two 08b diagnoses identical
apart from one filename in a "tests written" list, which is the same complaint. NO collision
came from the digit rule. Write test seeds accordingly: short strings differing only by a
number (`defect 1`, `defect 2`) all collapse to one entry, which real ~3000-char diagnosis
prose does not do.
**Two background blocks reach the fix prompt, and they carry different things on purpose.**
`loadPriorFixContext` carries the DIAGNOSES; the workspace/tooling facts earlier rounds
established arrive separately through `augmentPromptWithLedger` (`step-runner.ts`, ahead of
the dispatch). It used to render both in one block, ordered changes → findings → diagnoses
and head-sliced at the cap, so the diagnoses were always the part cut — MEASURED on task
681f0f99, 44 ledger entries totalling 48,523 chars against a 4000-char cap, and not one
diagnosis reached 07, the only step that reads one. The slice also defeated the ledger's own
dedupe, which suppresses an entry only on an exact text match and so missed every entry the
block had cut mid-string. Both blocks now drop WHOLE oldest entries and state the omission:
a truncated fact reads as a complete one, and the newest rounds are what the pass is
downstream of.
### Step summaries
The "What the agent did" panel (`task_steps.summary`) has two producers, and the cheap one wins. `resolveCuratedSummary` (`_step-summary.ts`) lifts `findingsSummary`/`summary`/`notes` straight off the apply output for the steps that emit one — no LLM, and it mirrors its own task-ledger entry. Only when the output carries none of those keys does `maybeEnqueueStepSummary` (`step-runner.ts`) spend a CLI call, and that pass is best-effort throughout: a missing provider, a `skip` dispatch, an empty agent text or a failure all leave `summary` null and never touch the step machine.
**Which CLI writes it is a per-task choice, not the step's.** `tasks.summary_cli_provider_id` (NULL = inherit the step's chain: per-step pref, then `tasks.cli_provider_id`) and `tasks.summary_llm_enabled` (false = skip the pass entirely) are set on the New Task form. The chosen provider is honored only while it is still `enabled`, and that check is load-bearing rather than defensive: handing a disabled id to the dispatcher does NOT fail, because `resolveDispatch` filters to enabled providers and merely ORDERS the preferred one first — an id matching nothing leaves the recap on whichever provider comes first. Falling back to the step's own chain is the predictable answer. MEASURED before the setting existed: claude-code spent 33,945 tokens (24,064 of them cache reads) and 22s writing three sentences.
**Both New Task CLI dropdowns remember their last choice per repository, and the memory is the CHOICE, not the column.** `GET /tasks/last-cli` used to read the FK columns alone, and a NULL there cannot say whether the user picked "none"/"inherit" or never picked — so only a picked provider was ever remembered and the summary dropdown reset to inherit every time. `tasks.cli_choice_recorded` / `summary_cli_choice_recorded` carry that missing bit, set from the PRESENCE of the field in the create request (the form always names all three; the plan, kb_author and upgrade-rollback spawners name none, which is why "most recent task on this repo" could not be read as a choice). The route's WHERE still admits the legacy predicate and orders flag-first, so rows written before the columns existed answer exactly as they did and no backfill was needed — `packages/database/migrations/pre-baseline/` is a parity record and the applier (`drizzle-kit push --force`) syncs DDL only, so an `UPDATE` there would never have run. `cliProviderId` in that response keeps its old meaning — the provider this repo last RAN on — because the plan chat and merge-conflict pickers render it as "the CLI that will run" and must stay aligned with `resolveProvider` in `routes/plan.ts`. A plan task started with a picked CLI is spawned with `ignore_saved_step_clis` set, or a saved per-step preference runs instead of the pick — MEASURED, a plan chat whose picker named codex ran on Claude through an explicit `01-plan-chat` preference saved days earlier. The chat panel reads the step's effective CLI (`stepCliProviderIds`) rather than the task column for the same reason: changing the CLI mid-conversation records a step choice for that task and leaves the task column alone.
**A step's CLI picked during a task belongs to that task.** The step card, the CLIs tab and the skill-verification gate's "CLI for the fix" (`09_6`, for the repair or regeneration it routes to) write `task_step_cli_choices` (task, step, role), which every resolver reads before the saved `user_step_cli_preferences` (`resolvePreferredCli`, `enrichStepsWithCliPreferences`, `resolveCurrentStepCliProviderIds`), and the saved preference is rewritten only when the person ticks "Also for my later tasks" (`remember`). A pick used to rewrite it every time, so a CLI switched to rescue one stuck run became the default of every later task of that user — observed twice: a plan chat switched twice in 8 s moved `01-plan-chat` for good, and a live check left `01-plan-build` on a test model. A NULL choice is the task clearing the slot, so the saved preference stays out of it there; a role it clears falls to the step's default slot, as an unset role always did. `task_step_cli_touched` is no longer written and is read only for the tasks that predate the choices.
**The invocation is unlinked but attributed.** `task_step_id` stays NULL — that column is what places a row in the step terminal, the retry blocker, park folding, the step's invocation count and the `cli_invocations_one_live_per_step_idx` partial unique index, and a recap written after the step finished is none of those. `summary_for_step_id` carries the SPEND home instead, so `enrichStepsWithCliStats` folds it into the step's token/cost badge by `coalesce(task_step_id, summary_for_step_id)` while both COUNTERS stay on `task_step_id`. `sumTaskTokens` and `sumTaskProviderBreakdown` accept `task_step_id IS NOT NULL OR summary_for_step_id IS NOT NULL` — deliberately not "no filter", because a summary row written before that column existed carries neither and must stay out of both sides, which keeps a task's list total equal to the sum of its per-step badges on old tasks with no backfill. Cost resolution needed no change: `resolveInvocationCost` already runs before the `step_summary` branch returns.
**The fan-out prompt is capped, and the cap is the point.** `buildAgentMiningSummaryPrompt` used to divide 7,000 chars by the agent count but floor the result at 800, which grows without bound past ~8 agents; its comment assumed "2-6 depending on QA level", from before plan mining fanned out to 61-623 agents on one step. MEASURED across every step-summary invocation on the dev install: 71 agents produced a 62,212-character prompt, 70 produced 52,875 and 623 produced 53,720, and all three were killed by the pass's own 60s budget having written nothing, while the three near 8,000 characters all succeeded. `SUMMARY_AGENT_TEXT_BUDGET` and `SUMMARY_AGENT_LIMIT` are both hard caps now, the elision is stated in the prompt, and past the limit the closing instruction asks for an overall summary instead of one line per agent — 70 lines was never the recap the panel is sized for.
### Staged agent bodies
**A step whose agent returns DOCUMENTS must not carry them in the reply.** One fenced JSON
block holding a whole corpus hits the model's single-message ceiling, and what happens there
is not truncation — it is the model shortening its own answer to fit, which trips NO detector:
`isOutputTruncationMessage` has never matched once on this install, across every step, all
time. MEASURED on one repo, 08 inline emitted 199,202 chars on its first run and then 135,433
chars for 378,008 output tokens on its second — 3.7x the tokens of the run before it for FEWER
bytes, all of it spent shortening and re-emitting. Staged, the same step emitted 18,187 chars
with MORE content in it (29 entries, 338 sections) and wrote 30 KB files with zero failures.
`_kb-body-file.ts` owns the pattern: the agent writes each body to `KB_DRAFT_DIR`
(`.haive/kb-draft/`) and names it in the JSON, and `apply` reads it back BEFORE anything is
routed or stored, so nothing downstream learns a second shape. Four rules, each of which cost
a measured failure:
- **Inline still wins.** `sections`/`content` present is used as-is, so a model that ignores
the contract — or a `detect_output`/step output replayed from before it existed — behaves
exactly as it always did.
- **The dir must be handed to the sandbox user.** The worker runs as root and the CLI sandbox
as uid 1000, so a plain `mkdir` yields `root:root 0755` and the agent writes NOTHING into it
— MEASURED, an 8-minute run produced zero bodies. `prepareAgentWritableDir` chowns the whole
created chain to the repo root's owner.
- **A declared-but-missing body is a FAILURE, never an empty entry.** Publishing a blank page
under a canonical KB name is worse than failing: an empty `ARCHITECTURE.md` reads as "this
project has no architecture" to every later reader. Failures are reported per item so one
bad body cannot discard the fifteen beside it.
- **The drafts are deleted once filed.** `.haive/` is NOT gitignored and `12-post-onboarding`
stages `.haive/install.json` by name, so leftovers sit in the user's `git status` for good —
and on a repo onboarding has to `git init` itself that step stages the whole tree with
`git add -A` and commits every draft. `discardKbDrafts` runs at the end of apply, and KEEPS
them when a body failed to resolve, since that is the one case where the files are evidence.
Two steps use it: `08-knowledge-acquisition` (a whole KB per reply) and `09_2-qa-resolve`
(`proposedWrite.contentPath`, one proposed section per answer — the second-largest emitter in
the system at 112,718 chars; MEASURED, one step row run three times on the same questions gave
40,218 / 28,717 / 40,706 chars for 23 answers each, the middle run 29% fewer characters for 16%
MORE output tokens).
**Do NOT reach for it for every large step — most are already bounded, by a different
mechanism, and adding a second one buys nothing.** `01-plan-build`, `02-plan-coverage`,
`03-plan-sequence`, `08c` and `08d` throw `MiningWaveError` and go wave-by-wave (one pass
cannot emit 400 nodes without truncating); `09_5`/`09_5b`, `03-phase-0a` and `11d` fan out one
agent per item; `05`, `07a`, `07b` and `08a` take one item per loop iteration. What makes a
step a candidate is not its SIZE but its SHAPE: one reply carrying N document bodies, with no
per-item dispatch. `06_5-agent-discovery` has that shape on paper (`custom[].body` is a whole
agent definition) and is deliberately NOT converted — MEASURED across every run on this
install it emits ZERO custom agents, so the path is dead and the large output is its
`skipped`/`declined` reasoning instead.
### Fan-out dispatch
**A fan-out reserves every agent before it sends any.** `dispatchMiningAgents` first inserts one
`pending` row per agent, carrying the step's own prompt, its requirements and its timeout rung,
in one transaction, and only then sends them one at a time. A worker that died part-way used to
leave the agents it had not reached with no row at all, so the barrier concluded the step without
them. They now wait `pending` with no invocation, and the barrier sends them from the recorded
prompt without charging an attempt (`dispatchReservedAgents`). A reserved row whose prompt has
aged out of retention is failed rather than waited on.
**Every write that links or fails a row is a compare-and-swap on the state read**
(`sameMiningState`: id, status, invocation link). Two passes over one step, such as a duplicate
advance or a completion racing a retry, used to both re-roll an agent, each superseding the
other's run. Now the loser supersedes only its own new invocation and sends nothing, and only the
winner supersedes the run it replaced. The orphan reconcile's fail write swaps on the link it read
for the same reason: a pass beside it may already have re-rolled the row onto a live run.
**A pass that lost agents to another pass reads the barrier again before it settles.** A first
fan-out, a wave, a re-roll or a user-requested re-run can find its agents taken by a pass running
beside it, and that pass's run may already have finished, so every row the loser read can be
stale: settling on them applied the old failure in place of the new result, or finished a wave
without folding it. `dispatchMiningAgents` therefore reports what it `lost` beside what it sent,
and the loser reads the rows again, once, and settles on those (`rereadMiningBarrier` in the apply
loop). Once is the bound: a pass that loses a second race settles on its second read. A pass that
sent nothing still parks while any row is live (`hasLiveMiningAgents`).
A dispatch that throws part-way fails what it reserved or linked and did not queue
(`releaseUnsentAgents`), and ends the one run it had recorded. A pass that fails its step before
sending, such as a `selectAgents` that refuses, fails every reservation still unsent
(`failReservedAgents`). Left `pending`, such a row would make the api's Resume refuse the step as
still running.
**A run's end and its mining result land together, on the row still linked to it.**
`handleCliExecJob` writes both in one transaction, on the success path and the failure path, so no
crash can leave a run ended beside an agent row still `running`. Every write it makes to a mining
row matches the run the row is linked to (`ownMiningRow`): a re-roll moves the row to a new run, and
a late completion of the old one used to overwrite it. What follows that transaction (learning a
model limit, the recap, `markCliParkBegin` and handing the step back) sits outside the `try` whose
catch records a failed run, so a queue error there no longer rewrites a finished run as exit -1.
### Worker restarts
**A step keeps the status that says what it is waiting on, even while its apply runs.** A
continuation (the advance an ended CLI run queues) re-enters a `waiting_cli` step. For a
form-bearing step it used to re-validate the saved answers and flip the row to `running`. A worker
that died from that point on left a `running` row, which boot reads as a pass that died before
doing anything, so it reset the step: every agent row deleted and the answers cleared. Now a
continuation that finds its answers saved uses them as they are, and the row stays `waiting_cli`.
Values the job carries are ignored there, since a parked step's only source of them is a submit
redelivered after it was applied. A submission, a retry and a first run still flip to `running`.
**Boot hands a dead worker's active jobs back before it starts its own.** BullMQ keeps a job
`active` under its lock when its worker dies, and the task and cli-exec locks are 30 minutes, so
such a job used to run only once that lock expired. `requeueOrphanedActiveJobs`
(`queues/boot-requeue.ts`) moves each one back to waiting before this process starts a Worker, and
only while `getWorkersCount()` reads 0: a connected worker may still be running them, and a count
that cannot be read moves nothing. The count is read once and nothing locks it, so like the reap and
the reconcile around it this relies on the one worker per install compose runs (a fixed
`container_name`): two workers booting together on one Redis could each read 0 and requeue a job the
other had just taken. Every task job is moved; on cli-exec only an agent run and the
version refresh are (`cliExecJobRequeuedAtBoot`), the kinds known safe to run again at once, and the
rest wait out their lock as before. The reconcile below ends a parked step's started runs, so their
jobs exit on the finalized row; any other started run runs again, as its lock's expiry would have
made it. A move never counts toward
`maxStalledCount`, so a job that kills the worker would come back at every boot: past
`BOOT_REQUEUE_LIMIT` requeues of one job, it is left to its lock and BullMQ's own count.
**Boot recovers a parked step and resets only a step that has nothing to lose.**
`reconcileOrphanedSteps` runs before any queue starts:
- A `waiting_cli` step the task is on has its orphaned runs ended, and unstarted runs no queued job
owes are ended too. The task's epoch is then fenced by a compare-and-swap on the state read, and
the step is re-driven at the new epoch, so every advance queued before the restart is stale. A
step the task has moved past is requeued rather than re-driven, and every run it still has is
superseded: one that ended later would resume it at whatever epoch the task was at by then. An
advance for it queued before the restart is dropped once it finds the task pointing elsewhere,
since every advance that carries no answer is queued for the step the task points at.
- A `running` step the task is on is decided by `bootRecoveryAction`. With agent work behind it (a
finished loop pass, an agent row, or a run of its own nothing superseded), it is a parked step
whose park write was lost. It is demoted to `waiting_cli`, guarded on `running`, and recovered
like the parked ones. Without agent work it is reset and re-run, as a deterministic step always
was.
**A running task that nothing drives is taken over.** A `running` task whose current step row is
missing, `pending` and not parked, or finished (`done`/`skipped`) with nothing after it, while the
task queue holds no job for it, is re-driven by `redriveStalledTasks` (`queues/stalled-redrive.ts`):
once at boot after the reconcile, whose own re-drive enqueue can still be lost, and every minute for
a task idle five minutes, for a hand-off lost between boots. It bumps the epoch under the task row's
lock and re-checks the row in the same statement, so a step a pass claimed since the candidate read
is left to that pass. A finished row is the job that died between ending the step and pointing the
task on, and its advance takes the duplicate-delivery branch below, which re-drives the hand-off. A
current row that FAILED is the job that died between failing the step and failing the task; no
advance can run it again, since every write refuses a failed row, so the sweep fails the task at
the epoch it read through that job's own hand-off (`finishFailedStep`), which also records the
failure's hint, arms its allowance watch and logs `step.failed`. An advance that meets such a row
first does the same rather than running it, since every reopen (a step Retry, both Resumes,
`retry_ai`, a CLI switch, the allowance auto-resume) resets the row first; `advanceStep` hands a
failed row back as its failure, as it hands back a done or skipped one. A
START on the queue is no such job (`taskQueueOwed`): it only claims a task still waiting to
start, so one a dead worker left `active` under its 30-minute lock would otherwise hold a claimed
task for that long and then do nothing.
The same sweep covers the two other jobs that can be lost with nothing to notice. A task left
`queued` past the cutoff with no START on the queue gets one, which the claim makes safe to
duplicate; `created` is left alone, since a deferred-start draft waits there on purpose. Every
route that starts a task moves it from `created` to `queued` before it queues START, after every
write that can fail (`markQueuedForStart`, `api/src/lib/task-start.ts`), so a START that was lost
or could not be queued is the sweep's to give, and the route answers as though it had been
queued. A task whose creation failed partway stays `created`, where nothing starts it. And a `waiting_cli`
current step with a run recorded before the cutoff that no cli-exec job owes, no run of its own
running and no task-queue job, is recovered exactly as boot recovers a parked step
(`recoverParkedStep`), bounded by `bornBefore`: only runs started or recorded before the cutoff are
ended, since a live worker may be starting younger ones. A step whose old runs all still have a job
(a pause, the per-task cap and the runtime reserve all hold jobs `delayed`) is left to them. The
lost run never started, so its re-dispatch charges no orphan budget.
**START claims the task, or does nothing.** `claimTaskStart` moves a `created` or `queued` task to
`running`, pointed at its first step, in one statement fenced on the epoch START read. A START that
finds the task anywhere else — redelivered after the task ran on, a second START a task Retry
queued, a task a repository deletion cancelled — logs and returns, where it used to restart the
task from its first step or revive a cancelled one. The pointer is load-bearing: a task Retry
leaves it on the step that failed, so under a pause the advance START queues for a first step
already `done` read the chain as moved and was dropped, leaving a `running` task nothing drove. A
START that throws before its claim fails the task only while it is still `created` or `queued` at
the epoch it read, since a Retry can re-queue the task in between, and fails nothing when it could
not read the task at all; `markTaskFailed` never writes over `cancelled` or `completed` from any
caller. Once START has claimed, its redelivery is
no longer a recovery path: a START that dies after the claim is recovered by the reconcile and the
re-driver above, as any advance is.
**A duplicate delivery re-drives the hand-off the step finished with.** An advance that finds its
row already `done`, while the task still points at that step and round, re-drives the hand-off from
the row (`finishedStepResult`, `step-runner.ts`) rather than as a plain `done`. The apply tail writes
`done` for a fix-loop, restart or revise verdict too, and for a `fixLoopOnError` failure, whose
diagnosis is the row's `error_message`, so a job that died between that write and its hand-off used
to walk forward past the round its step had asked for. The verdict is recomputed from the stored
output by the one function the tail itself routes with (`finishedRoutingVerdict`), so the two
cannot disagree on precedence. An answered fix-loop gate is finished from its answer instead
(`resolveFixLoopGate`): the gate writes its row `done` before it acts, so a verdict recomputed
there would replace the person's decision with the step's own. On a task that failed before the
answer was sent, the pickup guard admits the answer onto the `done` row as it would onto the parked
form (`gateAnswerSentAfterFailure`, the task's `completed_at` against the job's own timestamp); a
failure after it was sent, such as a Stop, keeps it out. `advanceStep`'s own `done` short-circuit
is left as it is: the START path that reaches it has no moved-chain check, so a verdict re-driven
there could bump a round.
**A step's advances run one at a time.** A continuation leaves the row `waiting_cli` through
apply, and a fan-out's agents each queue an advance as they finish, so two advances of one step can
both reach apply. `holdStepAdvance` (`task-queue.ts`) holds each task, step and round to one advance
at a time in the worker and defers a second with `moveToDelayed` rather than dropping it: the
advance already running may be a barrier check that parks without seeing what the second was
queued for. It replaced a guard that let a second advance skip only while the row read `running`,
which no continuation does any more. An advance that cannot be deferred (no job token, or the move
failed) waits for the holder in this process instead: running beside it is what the hold prevents,
and failing the attempt could lose it, since a continuation is queued with no retries. The hold is
taken outside the job's own catch, since that
catch fails the task, and it is per process, like the queue's single worker. A deferred advance can
then run after the pass it waited behind failed the step, so an advance on a `failed` task is
dropped (`failedTaskRefusesAdvance`): a Retry, a Resume and the allowance auto-resume each set the
task `running` first. An answer submitted to a form still parked is the one exception, since
answering it is what reopens the task; an advance onto that form that carries no answer is dropped
like any other. A clarification answer is one of those, since it rides `task_events` rather than the
job, so its route sets a failed task `running` itself before it queues the advance.
A Retry's advance waits behind a pass still running, so that pass must not keep what the Retry
reset. A task Retry's START, which only a task that never reached a step still takes, runs its first
step itself rather than through an advance, so it takes that step's hold as well. Every write step-runner makes to a pass's row, and every status the DAG executor and the
merge resolver set on it, goes through `updateOwnedStep` (`step-ownership.ts`), which lands only
while the row is still the pass's own, not `pending` (a Retry), `skipped` (a Skip) or `failed` (a
Stop, which fails the row without moving the task's epoch). A run a pass records for its row goes
the same way (`insertOwnedRun`, at every insert site in step-runner, the merge resolver and the DAG
executor), and so does a fan-out's reservation: the row is locked while it is still the pass's own,
in the transaction that inserts. The lock comes after the insert, the order a Retry takes them in
(runs and agent rows, then steps), because an insert can wait on a run or an agent row the Retry is
removing; the other order deadlocked the two. A Retry supersedes a step's runs and deletes its agent rows before
it writes the row, and that write waits on the lock, so its reset sweeps a second time once the rows
are written (`resetRowsForRerun`, and `resetStepAndDownstream` worker-side): what a pass recorded
before the Retry took the row is ended there, and a pass after it is refused. A run swept that way
can still be queued by the pass that recorded it, so its cli-exec job starts it only by a
compare-and-swap on `ended_at` and `superseded_at`: the Retry's sandbox kill ran before that job
had a container. That kill runs once, and it can land while a job is still preparing its sandbox, so
the run is read again right before the container is created (`beforeRun`), as soon as it first
prints, when it is certainly running and so either the kill found it or this read sees the
supersede, and every few seconds after (`run-superseded.ts`); a run that reads superseded gets no
sandbox or loses the one it has. A Stop takes the Retry's steps in one transaction
(`stopActiveCliInvocations`): runs, steps, a second sweep of runs, then the task, and the sandboxes
are killed only once it commits. No reader sees a run cancelled beside a step still running, and a
Stop that fails leaves every run as it was and kills nothing. The allowance auto-resume
(`autoResumeFailedStep`) writes in the same order, its claim on the failed task last, and a claim it
loses takes back every write before it. A pass a Stop cut off
mid-apply then releases what the failed task holds (`settleFailedTask`), as its own failure would
have. The cancel poll also stops a
pass whose task moved to a newer epoch. Either way the pass stops there, records no recap and
hands nothing off (`superseded`), and the Retry's pass runs once it lets go. The two writes that open
a pass on its `pending` row, claiming or skipping it (`openPendingStep`, `step-ownership.ts`), land
only while it is still `pending`, so a Skip stands. A Retry leaves the row `pending` too, so the same
transaction then reads the task under the job's fence (`taskWriteTarget`: its epoch, and not failed
or finished) FOR SHARE and rolls the flip back when the fence fails. A reset's epoch bump in flight
holds that read until it commits, so no row is ever activated at an epoch a reset has moved past. A
claim that committed before the bump is on the rows the reset reads after it: the api Retry and the
fan-out Resume reset every row still active once they have bumped (`rowsActivatedMeanwhile`, NOWAIT,
since they hold the task row, answering 503 when a pass is mid-write). Both kill the task's CLI
sandboxes only once that has committed: a refused action then leaves every run alive, and a dying
run's completion finds its invocation already superseded.
The loop Resume, `retry_ai`, Skip and a CLI switch on a failed or parked step take the task back
to their step the same way, through one helper (`moveTaskToStep`), and queue their advance at
the epoch it moved to; a CLI switch on a `pending` row only invalidates its cached form, since
the task has either not reached that row or already queued the advance that claims it. Those four,
the step Retry and the fan-out Resume refuse a `cancelled` or `completed` task with a 409 inside
their transaction, before anything is reset or killed: the task page offers no step action on
either, and a failed task stays fully recoverable.
A task Retry re-runs the step the task stopped on, at its round (`retryTaskAtStep`), rather than
replaying the task from START, which re-derives every stored loop_back and pays each fix round
again. One transaction ends the task's live runs, re-offers a parked row as it was (a form keeps its
schema and answers, a runtime park its detection), resets the failed or live row, its downstream
and any other live row, and moves the task back through `moveTaskToStep` under a bump fenced on
`failed`, so a Cancel that landed first stands (409) and nothing is reset. A `done` or `skipped`
row is left as it is: its advance re-drives the hand-off it finished with. A task that never
reached a step still restarts through START, and settles its rows again after its bump, since
answering a parked form revives the task and can open a row in between. That settle locks every
active row before it writes (`settleActiveSteps`): a pass that got in before the bump can still
move its row from a form to `running` between two unlocked writes and escape both. The recap goes to the ledger, or to a recap run,
only after the outcome has landed and only while the row still reads `done`: the ledger entry is
inserted by one statement that checks it, and a recap run is queued only once inserted and checked,
since a Retry's reset supersedes only the runs that already exist. That check narrows the window
without closing it: a Retry supersedes a step's runs before it writes the rows, so a recap inserted in
between still reads the row as `done`. The summary write closes it (`writeStepSummary`): it lands only
on the row version it summarized, keyed on the row's `ended_at`, which a Retry nulls and a re-run
rewrites, and only while its own run stands, read under a lock that waits out a supersede in flight.
Its ledger entry follows only a write that landed. The handoff is fenced on the
epoch the pass ran under: `handleResult` does nothing once the task has moved on, and every write it
makes to the task carries that epoch and refuses a task a Stop failed meanwhile, so a Retry or a Stop
landing after that check stands: pointing the task at the next step, parking it on a form, a run or a
fix-loop gate, and completing or failing it, the last two also reaping the task's containers.
The allowance watch a failure arms is fenced the same way, and only while the task still reads
`failed`: the teardown before it can outlast a Retry clicked on that failure, which clears the
watch as every other exit from `failed` does. The hint it leaves on the row (`writeStepHint`) lands
only while the row still reads `failed` and the task is still failed at that epoch, since a reset
clears a hint written before it but not one written after; a finished fan-out's timeout hint lands
only on a row still `done` under the pass's fence, and a login hint only while its run stands.
Answering a fix-loop gate does the same, and closes the gate only while its row is still the pass's
own. So does the advance that parks or starts a step. A pause or runtime park writes its row and
points the task at it in one transaction (`writeFencedPark`), the row first as a Retry takes rows
first, and is dropped whole when the fence no longer holds: a park a Retry overtook leaves the
signature the Retry's own advance reads as a live loop and drops itself behind. A step starts only
while the fence holds. A job revives a failed task only when the task was already failed at pickup,
which the pickup guard allows only for an answer to a form still parked, or to a fix-loop gate
that answer closed before a worker died (above), so a Stop landing after pickup stands; the answer
to a fix-loop gate hands off under the same rule. That holds
for the job's own catch too, which fails the task only at the epoch the job holds it at: the one it
read, or the one a reset the job made itself moved it to. Such a reset (a fix-loop re-entry, a
revise, boot recovery's) compare-and-swaps that epoch in the write that bumps it, kept last as the
api Retry keeps its own, and a lost swap rolls the whole reset back and hands nothing off. The
hand-off after it points the task at the target only while the task is still at that epoch, and a
fix round's request and `started` event are written in the same transaction: a new round has no row
to reset, so no swap is taken there, and a round a Retry overtook would otherwise count toward the
cap. That transaction first locks the source row while it is still the pass's own
(`lockOwnedStep`), since a Retry writes the steps before it moves the epoch.
A form submit carries no epoch on purpose, so it cannot be fenced. `isStaleSubmit` drops one that
lands on a form parked after the job was queued, such as a form a `ReopenStepFormError` reopened,
which would otherwise answer the new form with what was typed into the old one.
### Merge conflicts
The DAG's level merge and the merge resolver (`12-worktree-cleanup`, `00a-sync-base`,
`13-onboarding-push`) share the git core in `git-merge.ts`. Each fix agent edits the conflicted
files, and the host verifies, stages and commits, so a fixer must never start from what an earlier one
left. Five rules keep it that way, each MEASURED on git 2.43 and 2.54:
- **Unmerged paths are read with `-z`** (`unmergedPaths`). Without it git quotes a name holding a quote
or a non-ASCII byte, the quoted name reads as a deleted file, and its conflict markers were committed.
- **A conflict is told from a refused merge by MERGE_HEAD naming the merged ref** (`openMerge`). A
refusal (a local change or an untracked file in the way, a ref git cannot resolve) opens nothing and
exits with the codes a conflict does, and used to send fixers into a tree with no merge in it. It
now halts with git's own reason.
- **`merge --abort` refuses once a file the merge staged is edited**, a fixer's edit to a cleanly
merged file, and used to leave that half merge for the next fixer. `abortMerge` puts those paths
back from the index, each by its bytes, and aborts again. An abort that still fails halts the step
with a `merge.abort_failed` event, and nothing is dispatched into the merge. A merge open for
another ref is an earlier attempt's and is aborted before a new one opens.
- **A person's own checkout is never written to.** The worker's `/host-fs` mount is read-write, so under
`HOST_REPO_ROOT` nothing is put back from the index and a merge open for another ref is left alone:
both halt and name what blocks them.
- **What a fixer changes outside the conflicted files is moved aside, never committed or lost.**
Before a fixer is sent in, the merge dir is recorded as one git tree built in a scratch index,
untracked files included, with what the merge staged (a tree built from
`diff-index --cached HEAD`, so its cost follows the merge, and written through
`update-index --index-info` on stdin, so a name that is not UTF-8 keeps its bytes), the paths it
is sent to resolve, `HEAD` and `MERGE_HEAD` (`captureFixBaseline`, 1.8 s cold on a 39,378-file
repository). Every listing these read is bounded at 64 MiB rather than execFile's 1 MiB default,
which 300 changed paths under a deep directory pass, and a listing cut short checked nothing. For
a directory/file conflict those paths include the one git moved the file away from: git gives the
file a name of its own (`foo~HEAD`), reports only that and stages the directory side, so keeping
the file means deleting the directory and putting the file back (MEASURED on git 2.43 and 2.54).
The partner is found by content, the same blob beside it on its side where the other side holds a
directory, never by the name git chose, and only where that side changed it since the merge base:
git takes the directory cleanly where it did not, so an identical sibling is no partner. Files git
ignores are left out and stay where a fixer leaves them, never committed: they are where a
person's running tools write (build output, logs, caches), and recording them would hash every
dependency tree, MEASURED at 43,305 ignored files (835 MB of `node_modules`) beside 1,972 tracked
in a worktree of this repository. What counts as ignored is settled then, kept as a blob
(`git status --ignored=matching`, which names a directory only when a rule ignores the directory
itself: 26 entries for those 43,305 files). A fixer resolving `.gitignore` otherwise made a file
under a rule it dropped read as new and be moved, and hid a recorded file under a rule it added,
which the restore then wrote over (MEASURED on git 2.43). So the tree taken once it ends starts
from the recorded one, and only a file new since is judged by the rules it leaves. While secret
masking is on (the default), the deny-list's files (`.env`, `.env.*`) that the merge's own index
does not track are read-only empty mounts in the fixer's sandbox, so it cannot change those, and
such a file is left out of both snapshots, masking on or off: at a same-branch root the sandbox
holds `.git`, where a snapshot's blob would hand the fixer back what the mask hides. One the merge
added is tracked there, so the sandbox leaves it readable and both snapshots read it. Both
snapshots and the restore read
`.gitattributes` from the empty tree (`GIT_ATTR_SOURCE`, git 2.40) with `core.autocrlf` off and
`core.fileMode` on, so no clean filter or line-ending conversion stands between the bytes recorded
and the bytes put back, and a change of the executable bit is seen where the repository ignores
modes (MEASURED on git 2.43 and 2.54: otherwise a lossy clean filter's output came back, and LF
for CRLF). That bit is the only permission git records, so a fixer's other permission changes are
neither compared nor reported: recording them would take a walk of the whole tree before and after
every fixer, and setting modes to git's view would widen a person's own 0600 files. Once it ends,
on every outcome and before the merge is committed or aborted, whatever it changed outside those
paths moves to `.haive/merge-leftovers/<task>/<run>/files/` beside a `manifest.json`, and the
paths are put back from the tree (`relocateFixerChanges`). git recreates a file it puts back with
its default mode, so each keeps the permission bits of the file it replaced, never wider than git
wrote it, the executable bits following git where reading is allowed: a person's own 0600 file
came back 0644. A deletion there is put back too, and reported like a move, since at a same-branch
root it can be the person's own. The commit then
stages only the paths the fixer was sent to resolve, so neither a stray nor a person's own
uncommitted work at a same-branch root is swept into it, where `add -A` took both. What a fixer
staged itself outside those paths is put back in the index as the merge had it, byte for byte so a
name that is not UTF-8 is unstaged too, `.haive-data/` and gitlinks included although they stay in
the tree, since `commit` takes the whole index and a fixer at a same-branch root can run git;
MEASURED on git 2.43, git refuses to open a merge while the index holds a staged change, so the
recorded index is the merge's own, and the manifest names each blob taken out. An index that
cannot be read, or that still holds what the fixer staged after a few tries (another git process
holding its lock), stops the step before anything is committed, and the recorded tree is kept for
the retry. The index is put back even when the tree cannot be read (a mask policy, snapshot or
diff that fails), since it needs only the recorded index. A person's own edit made there while the
fixer ran cannot be told from the fixer's and is moved with it. The index is refreshed after the
restore, since git rewrites the files and `merge --abort` refuses their stale stat data. git
writes what it puts back as the worker, so those paths, and the directories git created for them,
are handed back to the merge dir's owner; a directory that already stood keeps its own. `.haive/`,
`.haive-data/` (other writers keep them) and gitlinks are never moved; a link or a name that is
not UTF-8 stays and is reported, a link with its target in the manifest, since a scratch worktree
removed later takes the link with it. A name is judged by its bytes, since a valid one can hold
U+FFFD, the character an undecodable byte reads as. A directory a fixer made where a file stood
gives way to the file only once everything in it has moved: git replaces such a directory whole
and takes what is left in it along, so one still holding something is kept and named. The recorded
tree is spent once used, and one whose merge a person finished or aborted meanwhile moves nothing:
putting its paths back would write merged files into a tree with no merge open. A tree git could
not record is reported the same way rather than read as a fixer that changed nothing; the fixer is
still sent, since halting there would stop every merge whose scratch `add` fails (a clean filter
the worker lacks). Each relocation is a `merge.fixer_leftovers` event and a step warning. Its
manifest is written before anything moves and marked once that event is written (a `reported` file
beside it), so a relocation a restart cut short, or one whose event was never written, is reported
by the task's next relocation. A base worktree is removed without `--force`, so one still holding
something git will not discard is kept and named. The plan merge's agent pass (`01-plan-merge`) is
handled the same way, its note going into the conversation rather than a step warning, since the
revise loop resets the row every turn. Its recorded tree is kept for the whole merge rather than
spent, as a `plan_merge.fix_baseline` task event beside the transcript, since a Retry resets the
row and the next conversation resumes a merge a cancelled one left open. Nothing but its fixers
writes in the scratch worktree, so what one that failed, was stopped or was re-dispatched left is
moved aside before the next is sent in (`llm.prepareWorkspace`) and before Save or Pull removes
the worktree, reported by the event alone; Save or Pull fails and keeps the worktree when that
cannot be done. A tree git could not record is kept the same way, so every later fixer of that
merge is reported unchecked, a later pass's before its fixer is sent in, rather than a fresh
record taking in what one left. A merge a pass opens is recorded afresh, since an earlier merge of
the same two commits was another tree. A conversation that resumes another's merge also reports
what that one's relocations never did. Nothing is recorded under `HOST_REPO_ROOT`: the sandbox
mounts a local-path repository read-only, so no fixer can write there. A cancel runs no step
code, so it moves nothing aside: a task's worktree is removed whole, a fixer's changes with the
rest of the task's work, a same-branch root is left as the fixer left it, merge still open, and a
plan merge's scratch worktree waits for the next pass or a Save or Pull, as above.
The resolver checks for a committed merge before its budget halt, so a merge finished by hand after
a halt finishes the step on a Retry.
## CLI adapter system
`packages/worker/src/cli-adapters/base-adapter.ts` defines `BaseCliAdapter`. Implemented adapters: `claude-code`, `codex`, `gemini`, `amp`, `zai`, `antigravity`, `ollama`, `muse`, `grok`, `openrouter`. Each declares `supportsSubagents`, `supportsCliAuth`, `supportsMcp`, `supportsPlugins`, `defaultAuthMode` (`subscription` or `api_key`), and `apiKeyEnvName`. `supportsSteering` defaults to false; the Claude-family adapters (`claude-code`, `zai`, `ollama`, `muse`, `openrouter`) override it to true, and so do `amp` and `codex` — codex only through its app-server, and only once that is verified for the task (`steeringTransportReady`) — see Steering below.
Four adapters are the same trick: the stock `claude` binary pointed at a non-Anthropic Anthropic-wire endpoint via `ANTHROPIC_BASE_URL` — `zai` (api.z.ai), `ollama` (in-stack daemon or ollama.com), `muse` (api.meta.ai), `openrouter` (openrouter.ai/api). All four piggyback the `claude-code` install, so they build no new sandbox image. Their per-endpoint quirks are MEASURED against the live API and recorded in the adapter, not taken from vendor docs — `muse` exists as its own adapter purely because its effort scale rejects `max`, and `openrouter` records the opposite finding (its effort enum is validated globally at the gateway, so every level is safe on every model). Re-probe before "correcting" any of those comments.
`ollama` records the third effort finding, and it is the only scale in the codebase whose default is not its top level: `low|medium|high|max` with default `high`. The level arrives as Anthropic `output_config.effort`, which Ollama's compat layer reads only because the claude binary sends `thinking:{"type":"adaptive"}` — the layer honors `thinking.type` for the literals `enabled`/`disabled` and falls through to effort for anything else. An out-of-range level never 400s; it leaves the think value unset, which Ollama then forces on for a thinking-capable model. `max` is exposed but not the default because gpt-oss does not recognise it as a harmony level and reasons LESS at `max` than at `high`, and deepseek-v4-pro at `max` spent an entire output budget in the thinking channel and returned empty text — the same failure the disable-thinking proxy exists for. `high` is additionally what the binary already sent before the scale existed, so the default changed nothing. Orthogonally, the level is on/off only for most LOCAL models: every built-in renderer in ollama 0.24.0 reads `think.Bool()` and none reads `ThinkLevel`, while Ollama Cloud models are rendered upstream and honor all four. Do not "correct" this scale to match zai's — re-probe.
`openrouter` traffic goes through the `openrouter-compat-proxy` sidecar (`docker/openrouter-compat-proxy/`), not straight to openrouter.ai. The claude binary appends a trailing `role:"system"` message (the Agent tool's agent-type listing) on top of the top-level `system` field; OpenRouter passes Anthropic models through natively, but every other vendor needs an Anthropic→OpenAI translation in which that message keeps its position, and vLLM-style backends answer `400 "System message must be at the beginning."` (measured on `qwen/qwen3.8-27b` across all three of its upstreams; `--disallowedTools Agent` does not stop the binary emitting it). The proxy hoists it into `system`. Unconditional rather than toggled — the rewrite is a no-op when no such message is present and was verified not to change already-working models — so unlike ollama's thinking proxy there is no per-provider switch. `resolveOpenRouterBaseUrl` is the single place that endpoint is chosen, shared by the adapter and the Test-connection probe; a provider that sets `ANTHROPIC_BASE_URL` by hand bypasses the proxy and the probe then says so.
`openrouter` fronts 400+ models, so the provider form picks from a cached catalog rather than a free-text slug: the worker's `REFRESH_VERSIONS` job also refreshes `openrouter_model_cache` (`cli-versions/openrouter-models.ts`), the API serves it at `GET /cli-providers/openrouter/models`, and the picker refuses models whose `supported_parameters` lacks `tools` because Claude Code cannot run a step without native tool use. The cache is never a gate: an empty or errored catalog degrades to a free-text model field.
### Steering
Mid-run steering is a user message delivered to a RUNNING CLI, applied at its next
boundary — an NDJSON line on stdin for most CLIs, a `turn/steer` JSON-RPC request for codex.
`supportsSteering` is read in exactly ONE place (`dispatcher.ts`, ANDed with the step's
`steeringRequested` and with `steeringTransportReady`, the per-task half only codex overrides)
and reaches the browser only as the per-invocation `cli_invocations.steerable` column — no
capability travels to web, which is why there is nothing to keep in sync in `@haive/shared`.
**The capability set goes stale silently, so re-probe it.** It was set in June 2026 and by
September amp had shipped steering without anything noticing — `adapter-steering.test.ts`
asserted seven adapters and amp was not one of them. It now asserts ALL ten, false ones
included. Verdicts as of 2026-09-12 (codex revised 2026-09-13), each against the vendor's
current docs and, where installed, the binary's own `--help`:
- **claude family** (`claude-code`, `zai`, `ollama`, `muse`, `openrouter`) — `--input-format
stream-json`, stdin held open, one NDJSON user-message per steer.
- **amp** — `--stream-json-input` ("Read JSON Lines user messages from stdin. Requires both
--execute and --stream-json", quoted from the shipped image's own `--help`) plus a
top-level `"steer": true`, which means "apply at the next interruption point while the
agent is busy". `-x, --execute [message]` takes an OPTIONAL value, so bare `-x` with the
message on stdin is the shape the adapter already used for an oversized prompt.
MEASURED against 0.0.1789200043-gdb3b35, because two things had to hold before the flag
could be set. Its stream DOES emit the `user` + `tool_result` event `onBoundary` keys on
(`system, user, assistant, user, assistant, result` for a one-tool run) — without it
`steer_consumed` would never publish and every amp steer would sit at "queued" until exit
relabelled it "the run ended before this was applied", telling the user their steer was
ignored when it was not. And a `steer: true` line written 6s INTO a run was received and
applied: the agent abandoned its original task and answered the steer verbatim, then
exited 0 once stdin closed — which is what makes the forwarder's `onResult` latch plus its
750 ms grace the right shutdown for it. amp also echoes each stdin message back as a
text-only `user` event, which is harmless in both directions: `onBoundary` ignores a user
event carrying no `tool_result`, and `onText` reads assistant blocks only.
- **codex** — YES, through `codex app-server`, and only where it was verified; `codex exec`
stays the default and the fallback. `exec` is fire-and-forget (it prints `Reading additional
input from stdin...` and waits for EOF before the first turn), and `codex queue --thread <id>
--message <text>` is NOT a way in: MEASURED against 0.154.0 on a live authenticated run, it
accepts and persists the message (`Queued message <id> for thread <id>`, exit 0) while the
running `exec` never sees it, and a later `exec resume` with its own prompt did not deliver it
either — the `thread/queue/*` RPCs belong to the app-server the TUI talks to. The app-server is
JSON-RPC 2.0 over plain piped stdio (`cli-executor/codex-app-server.ts`): `initialize` →
`thread/start` → `turn/start`, each steer a `turn/steer` carrying `expectedTurnId` and a
`clientUserMessageId`, and consumption is the turn's `userMessage` item echoing that client id
(drained in order when a binary echoes none). MEASURED on a live plan_chat turn: a steer sent
during the third of three `sleep 20` calls answered with the live `turnId`, the `userMessage`
carrying its client id arrived once that call returned, and the reply answered the steer. How it
is verified and abandoned is below.
- **grok** — NO. Headless `-p` streams are read-only and the REPL needs a TTY (piped stdin
dies ENXIO, already recorded in `grok.ts`). Only ACP (`grok agent stdio`) is bidirectional.
- **antigravity** — NO, and it is the closest miss. It already passes `--input-format
stream-json` and writes an NDJSON prompt on stdin, but as `stdinPrompt`, which CLOSES; its
docs say to wait for the `result` event before writing again. That is a queued follow-up
TURN, not a mid-turn steer, and there is no `steer:true` equivalent to queue one.
- **gemini** — NO. Stdin is never opened at all; mid-run injection is an open upstream
feature request.
**codex steers through an EXPERIMENTAL protocol, so each task verifies it before relying on it
and falls back to `codex exec` the moment it does not hold.** The app-server is marked
`[experimental]`, and older binaries lack it or part of it — MEASURED on real images: 0.40.0 has
no `app-server` subcommand, 0.78.0 has an app-server with `turn/start` and `turn/interrupt` but no
`turn/steer`, and 0.122.0 and 0.154.0 pass the probe. Three layers:
- **A zero-token probe on a provider's first steerable dispatch in a task**
(`cli-adapters/codex-app-server-probe.ts`, called from `resolveTaskDispatch`; in onboarding and
workflow tasks that dispatch IS `00-model-health`, whose progress line states the verdict). The
provider's own image and argv run with NO credentials and NO network, which is what makes it
free: MEASURED on 0.154.0, `turn/start` still answers `inProgress` and the turn stays active for
20+ s while codex retries its connection, so `turn/steer` (must answer `result.turnId`),
`turn/interrupt` (only accepted after `turn/started`) and `turn/completed` are all exercised with
no model call. The verdict lands in `tasks.codex_app_server` per provider (migration 0157) and is
current only while the provider's `cli_version` still matches. An inconclusive probe (docker
exit 125, a turn that ended before its steer was answered) records nothing and the next
dispatch probes again. ASSUMED, not measured: a future codex that refuses turns without a login
would read as `unsupported` — which fails toward exec, the safe direction.
- **An in-invocation fallback when the app-server cannot accept the turn** (spawn, initialize,
thread/start, turn/start, or no accepted turn within 5 min): no work was done, so exec-core
re-runs the SAME invocation from `codexExecFallbackSpec` (the adapter's exec argv, prompt over
stdin) and returns that. Before that exec half starts the row stops claiming `steerable` and a
`steerable` stream frame drops an open terminal's steer box, so no steer is accepted that nothing
will read. A binary that exits before answering `initialize` is `spawn` here
exactly as in the probe, with its stderr tail as the detail — MEASURED, that tail was clap's
`unexpected argument '--json' found`, the one line naming what changed, where the session alone
could only say the process had exited. A failure after the turn was accepted — a server->client
request under `never`/`dangerFullAccess`, a stream that ended without `turn/completed`, a
completed turn with neither usage nor a message — cannot be re-run blindly, since the agent may
have edited files, so it fails as `CODEX_APP_SERVER_FAILED_HEADLINE`, a transient the step's
existing re-dispatch re-runs. Either way `handleCliExecJob` first records `unsupported`/`runtime`
for that provider, so every later dispatch in the task — that re-run included — builds
`codex exec`, and the step carries a `warningMessage` naming the stage and codex version. That
banner cannot be the only trace: a self-revising step resets its own row at the end of the turn
that set it — MEASURED on plan_chat, the warning was cleared 0.3 s after it was written — so
every `unsupported` verdict, probe or run, is also a `codex_app_server.unavailable` task event on
the Activity tab. A run that behaved still reports `thread.cliVersion`; one that differs from the
verdict's binary (a provider left on "latest" whose image was rebuilt) drops the verdict, so the
next dispatch re-probes the new binary. Haive's own kills (timeout, cancel, preemption) never
count, and neither does Docker's own exit 125: a container that never started says nothing about
codex, which is how the probe already read it.
- **`CONFIG_KEYS.CODEX_APP_SERVER_ENABLED`** (default on, Admin > CLI execution) for what the probe
cannot see: off sends every newly dispatched codex run through `codex exec`. The same card lists
the last 30 days of `unsupported` verdicts with their codex version and stage — the report Haive's
protocol support is updated from when a codex release changes the API.
Four measured facts to keep. EVERY app-server error is `-32600` — an unknown method, a malformed
request, `thread not found` and `no active turn to steer` alike — so nothing may key on an error's
code or wording, only on success results and structural fields; that is also why a single refused
steer is NOT a downgrade, since a steer racing the turn's end is refused the same way. codex exits
~0.12 s after stdin EOF when no turn ran but ~5.1 s after an interrupted one, with the image's
entrypoint and without it. And the MCP surface needs no reconciling: both transports read the same
`~/.codex/config.toml` that `cli-merge` writes (`initialize` reports `codexHome:
/home/node/.codex`), and `codex exec` runs already expose `codex_apps`. Those servers boot
asynchronously once `thread/start` has answered — MEASURED on live runs, ~0.3 s for `thread/start`
and ~5 s from container start to `turn/started` — so they never hold up the handshake the 5-min
deadline guards. Finally, codex's multi-agent feature goes off with
`-c features.multi_agent_v2=false` and never `--disable multi_agent_v2`, although codex's help calls
them equivalent: `--disable` refuses a feature name the binary does not know (0.154.0 answers
`--disable no_such_feature` with "Unknown feature flag", exit 1, and 0.78.0 refused every exec and
app-server start that way) while `-c` ignores one, and `codex features list` shows both forms
setting the same flag.
**The steer is echoed by US, because the binary never echoes it.** `steer-echo.ts` fans one
written steer to three places from the forwarder's `onWritten`: a `steer` stream frame
(live), the Clean transcript (persisted), and the stream-log buffer (the persisted Raw tab).
Three details are load-bearing:
- The frame is its OWN top-level type, not a value on `output`'s `stream` union. The viewer
routes any non-`text` output straight into xterm, so an output-shaped frame would be
echoed raw by any client that has not shipped the inline rendering — and `output` is in
the viewer's stall-clock frame set, so a steer published there would restamp "the CLI is
talking" on exactly the frozen run someone is steering.
- The Raw line goes into the buffer ONLY, never a second publish. The viewer draws the live
line from the steer frame itself; publishing both draws it twice.
- The soft-timeout wind-down rides the same channel and is marked `system` AT ITS SOURCE, so
it still reaches the CLI but is not rendered as something a person typed. Marked rather
than inferred from an empty id, because an empty id already means "legacy bare-string
steer", which IS human — the same distinction that makes `publishCliSteerConsumed` drop an
empty id while `publishCliSteer` keeps one.
**Both sides of the optimistic append have to be idempotent.** The sender adds its own turn
when the POST resolves, but the worker publishes the frame the moment the text reaches
stdin, so the frame routinely arrives FIRST. Guarding only `applySteerFrame` left the race
open from the other direction — MEASURED in the browser against a live plan-chat run, one
steer rendered as two identical `You ✓` turns while the persisted transcript held exactly
one. `appendUserTurn` is idempotent on a non-empty id for that reason.
`cli_invocations.clean_transcript` is the durable half (migration 0156): ordered segments,
model prose interleaved with the user turns at the positions they were injected. It is a
SECOND column rather than part of `raw_output` because that column is also the step parsers'
input, so a human sentence there is handed to a JSON parser as the agent's answer. NULL
means "not recorded" — every pre-existing row, and any run with no model prose, where
`raw_output` is the only copy of the answer and a transcript would hide it.
`model_identity` (`queues/cli-exec/model-identity.ts`) records which model ANSWERED, not which one was configured. Two DISTINCT channels, not two names for one value: the claude-family stream-json `system`/`init` event carries what the binary ASKED for, each `assistant` event's `message.model` carries what the endpoint SERVED, and they disagree — api.z.ai answers a `glm-5.3[1m]` request as `glm-5.3`, and recorded streams show the same provider served `glm-5.2` on 2026-07-24 and `glm-5.3` on 2026-08-18, i.e. an endpoint can swap models with no config change here. Coverage is MEASURED per provider: `claude-code`/`zai`/`ollama`/`muse`/`grok`/`openrouter` report both channels; `gemini` names its models only as the keys of `stats.models`; `antigravity` names one only in its `--log-file` and only as a human LABEL (`Gemini 3.7 Flash (High)`), parsed from ONE constant marked volatile that returns null on a reword; `amp` and codex's `exec` report NOTHING. codex's `exec --json` carries no model on any typed event (verified against a complete 3.4 MB SUCCESSFUL run — `model_provider` strings found in such a stream come from old `~/.codex/sessions` rollout files an agent happened to read, not from live events), and amp emits `agent_mode` instead because it abstracts the model away. codex exec still records `requested`, read off its own `--model` argument — the adapter passed the short `-m` until 2026-09-13, which the argv reader never matched, and MEASURED, 0 of 624 codex invocations carried any identity. codex's app-server does better on one channel: `thread/start` names the model it resolved, recorded as `requested`, while a served model appears only when it reroutes (`model/rerouted`, source `codex-app-server`). So amp is permanently `match: 'unknown'` and codex nearly always is, which is why `unknown` never fails a run.
Two traps in that data. `result.modelUsage` keys are recorded as `billed` and are NOT an identity source: `grok` serves `grok-4.6` while billing `grok-4.6-build`, and `claude-code` bills a `claude-haiku-*` call for its own session titling. And an `assistant` event with `model:"<synthetic>"` is a message the BINARY authored (an API error), not a model reply — filtered on the angle-bracket convention rather than the literal word, so a future sentinel is excluded too. `requested`/`served` are stored verbatim; only `match` is lenient, and only for one case: an endpoint that DROPS a trailing variant tag while naming the same model (`glm-5.3[1m]` to `glm-5.3`) is `exact`. That keys on the tag's PRESENCE differing, not on tag-stripped equality, so a version swap (`glm-5.2[1m]` to `glm-5.3`) and a different variant (`[1m]` to `[200k]`) both stay `differs`, and ollama's colon marker (`glm-5.2:cloud`) is untouched. A mismatch warns and never blocks — claude-code legitimately resolves an alias to a dated snapshot — unless `CONFIG_KEYS.MODEL_IDENTITY_STRICT` (default false, admin toggle) is on. Capture rides the same parse as token usage, so it costs no extra call, prompt or tokens; the canary (`00-model-health`) then copies the task-default provider's identity onto `tasks.model_identity`, while per-invocation truth stays on `cli_invocations.model_identity` because per-step CLI preferences let one task run several models. Re-measure with `packages/worker/test/model-report-discover.ts` before "correcting" any of this — it runs each adapter's real invocation and feeds the live output through the shipped parser.
`tool_usage` (`cli-executor/tool-usage.ts`) records what a run USED — native tool calls, MCP tool calls (`mcp__<server>__<tool>`, grok's `use_tool`, codex's `mcp_tool_call`), native sub-agent activity, `Skill` calls, and the agent-definition / skill FILES the agent opened — tallied by the same pass as `token_usage` and `model_identity`, so it costs nothing extra, and persisted to `cli_invocations.tool_usage` (migration 0159). Three facts to keep. NULL means "not yet examined"; a written `coverage: 'none'` means "examined, not observable on this path" — amp (MEASURED: its stream-json assistant events carry text blocks only, across all 39 stored runs), gemini (no stream parser), antigravity (unmeasured), plain output and the sequential sub-agent script — the one deliberate departure from `compaction`'s "never write an empty object", because it is what lets the backfill's `IS NULL` predicate converge. Codex items arrive TWICE, `item.started` then `item.completed` (app-server: `item/started`, `item/completed`), so only the completed one counts, and the app-server's camelCase item types are folded onto the exec spellings so one record covers both transports. And a read is classified against the provider catalog's `projectAgentsDir`/`projectSkillsDir` union, matched on whole segments from the sandbox MOUNT ROOT (`/haive/workdir`, never the resolved cwd — a worktree run's cwd sits under `.haive/worktrees/`), with `README.md`, the `-legacy` quarantine and a CLI's own skills outside the tree (`/home/node/.codex/skills`, grok's `bundled/skills`) never counting — nor a glob or shell variable standing where a filename stem should (`cat .claude/agents/*.md` reads every persona or none, and the tally cannot say which; MEASURED, `*` and `$f` once ranked beside real personas). Every CLI naming quirk sits in ONE volatile constant in that module; re-measure against stored rows before "correcting" it. `agents.assigned` is the one field no CLI reports: the dispatcher reads the `[[HAIVE_AGENT_DEFINITION:<id>]]` marker ids off the prompt BEFORE `adaptPromptForCliCapabilities` rewrites them (`agentGuidanceIds`), a mining step whose agent IS a persona names it (`AgentMiningDispatch.personaIds` — 03's roster, whose prompt carries no marker), and the union rides the spec as `assignedAgentIds` to the completion write, which stamps it on the success and the failure path alike. It cannot be backfilled: the stored prompt is the rewritten one — MEASURED, 0 of 3,476 stored prompts hold a marker, and the 228 that name a `.claude/agents/<id>.md` path are matching the repository file-tree listing in the prompt, not a pointer — so the stats tab says "recorded from" the first stamped row instead of reading an empty list as "unused".
The dispatcher (`resolveDispatch`) filters to enabled providers, orders the resolved preferred provider first, and picks the first whose adapter is registered and has `supportsCliAuth` — plus `supportsSubagents` when the step declares the `subagents` capability. If none matches, the step is skipped. Every plan it emits is a CLI invocation; there is no API-mode branch. Auth mode selects which credentials the CLI is given, not whether the dispatcher bypasses the CLI.
The sub-agent emulator splits a single sub-agent specification into either a native `Task()` call (Claude Code) or a sequential prompt script (everything else). A sequential script runs inside a single `cli-exec-queue` job — the runner is an in-memory for-loop over the sub-steps, with no per-sub-step DB writes. A crash mid-script therefore fails the whole invocation; restart re-runs from sub-step 0. (Mid-script resume would require persisting each sub-step's parsed output to `cli_invocations` before moving on — not implemented.)
### Skills per CLI
**Every CLI with a skills mechanism needed its own lever, and each was MEASURED on the wire**:
the CLI in its own sandbox image, Haive's exact argv, and the API base URL pointed at a local
server that records the request and answers 400, so no tokens are spent. What the model receives
is only visible there — an `init` event lists skill NAMES, never the listing text.
- **claude family** reads `.claude/skills`, and `Skill` is one of its 11 eager tools. The listing
is capped at context window x 4 x 1% (8,000 chars on a 200K model) and the binary charges its
OWN bundled skills first: on 2.1.270, 1 of 18 repo skills kept its description.
`CLAUDE_FAMILY_SKILLS_ENV` (steering.ts) sets `CLAUDE_CODE_DISABLE_BUNDLED_SKILLS=1`, which
exists from 2.1.169 and gives 18 of 18, plus `SLASH_COMMAND_TOOL_CHAR_BUDGET=40000` for the
offered builds before it: 2.1.105-2.1.168 cut repo skills with no switch to stop it (2.1.150:
10 of 18), and the budget restores all of them. A SKILL.md without frontmatter is listed with
its H1 as the description.
- **codex** reads only `<root>/.agents/skills` (walk-up) and its home, REJECTS a SKILL.md without
frontmatter, and has no skill tool: it lists name, description and path and the model reads the
file — its intended lazy load, which 63 of 95 observable workflow runs used unprompted. Its five
`.system` skills are off (`-c skills.bundled.enabled=false`, honoured from 0.114.0 and on every
offered build): `skill-creator` had been pulled into 33 runs of `09_5`/`09_5b`.
- **grok** skips project skills AND AGENTS.md in an untrusted folder (from 1.0.25), and a headless
run is never trusted: every grok run listed 0 repo skills until `GROK_FOLDER_TRUST=0`, which
also ungates the repo's Claude-compat hooks, held off with `GROK_CLAUDE_HOOKS_ENABLED=0`. Its 23
bundled skills are hidden by `[skills] ignore` in `/etc/grok/managed_config.toml`, grok's
managed layer, read from outside the auth volume and mounted read-only through
`CliCommandSpec.configFiles`, so Haive never merges into the `config.toml` grok writes itself.
- **gemini** exits 55 before its first request without `GEMINI_CLI_TRUST_WORKSPACE` (from
0.39.1), and without `--yolo` non-interactive mode drops every tool that needs approval,
`activate_skill` and the write tools included. Skills reach the model from 0.26.0; its two
built-ins are switched off by name (`skills.disabled`) in the system settings file, the only
place gemini honours it, and only while that file is ROOT-owned.
- **amp** builds its prompt server-side, so it is not capturable offline.
- **agy** (antigravity) ignores the workspace's `.agents/` in a headless run. MEASURED on 1.2.2 with
Haive's argv and a throwaway copy of a real login: it listed only its built-in skills and
subagents, and `--agent <name>` answered `Agent "<name>" not found`. It loads customizations
from `~/.gemini/config` instead, so the sandbox MIRRORS them there read-only at dispatch
(`CliCommandSpec.repoMirrors`, `queues/cli-exec/repo-mirrors.ts`). `.agents/skills` is mounted
as it is, and each `.agents/agents/<id>.md` becomes `<id>/agent.md` with only `name` and
`description` kept. agy loads neither a flat file nor one carrying Haive's claude keys. Both
come from the invocation's own tree (the worktree when there is one), and nothing on disk
changes shape. agy also needs `~/.gemini/config` owned by `node` (`nodeOwnedDirs`): Haive binds
its MCP config there, and with the dir root-owned every print run died before its first turn
(`failed to get/create default project`).
Every version range above was measured per downloaded build, zero-token, across the versions the
picker offers. Two feeds reach back past Haive's command line, so `minRunnableVersion`
(install-metadata) keeps those builds out of the picker and refuses a save naming one: grok
0.2.116 (`streaming-messages-json`) and gemini 0.6.0 (`--output-format`).
`resolveSkillTargetDirs` writes one mirror per ENABLED provider's `projectSkillsDir`, so a provider
enabled after onboarding has no generated skills until something rewrites them — generated skills
are outside onboarding-upgrade scope. Every lever above is version-bound vendor behaviour:
re-capture after a CLI bump before trusting it.
**Agent and skill frontmatter has to parse in every CLI's YAML, not just claude's.** Every value
goes through `yamlScalar` (`steps/_yaml-scalar.ts`): plain when it reads back as itself under
YAML 1.1 and 1.2, double-quoted otherwise, so a value that was always valid keeps its bytes and
no template hash moves. MEASURED with a plain `description:` holding `: ` (four baseline agents,
and LLM-written ones on live repos): claude 2.1.270 listed the agent, grok 1.0.34 left it out of
`spawn_subagent` without a word, and gemini 0.60.0 refused it ("YAML frontmatter parsing
failed"). A claude-only check therefore never shows the breakage. Haive's own line readers
(`_agent-loader`, `parseSkillMarkdown`, the bundle parser's `splitFrontmatter`) decode through
`unquoteYamlScalar`. Haive's own gemini runs never load `.gemini/agents` (`enableAgents:false`
returns first), but the files are committed to the repository, where gemini runs with agents on.
### Agent rules
**Every dispatch carries its provider's rules, whatever the checkout holds.** Onboarding writes
the merged `haive:cli-rules` block into AGENTS.md, but a workflow task runs in a worktree checked
out from HEAD, onboarding's commit defaults off, and an upgrade never re-committed the file —
MEASURED before this existed, 598 of 3,509 recorded invocations (17%) ran with the rules loaded,
and none since the last manual regeneration. So `buildCliSidePlan` puts the provider's effective
rules (`resolveEffectiveRules(provider.rulesContent)`, not the merged block) at the top of every
`kind: 'prompt'` dispatch through `withAgentRules` (`orchestrator/agent-rules.ts`). It is applied
LAST in `adaptPrompt`, so no capability or marker rewrite ever touches the operator's text, under
a framing line: the step's own instructions and output contract win where they differ, and the
block supersedes a differing AGENTS.md copy.
It is never skipped because the repository "already delivers" the rules, which was unsafe four
ways: the block written into every repository was damaged by the dedup bug until #232, codex reads
only the first 32 KiB of AGENTS.md, antigravity's native loading is unmeasured, and a block merged
from several providers never equals one provider's rules. The cost is a duplicate where the
repository copy also loads.
- **Only a block at position 0 is Haive's.** A stored prompt dispatched again has its block
REPLACED with today's rules, and null rules strip a leading block, so a switched-off dispatch
never keeps a stored one. The strip runs BEFORE the other adapters, because every one of them
prepends: one that applies now but did not on the stored run (a model newly learned to lack
vision) would otherwise bury the old block and leave two. The isolation scan reads the prompt
with that block removed for the same reason. A marker anywhere else is text the prompt carries
and can neither suppress nor replace the injection.
- **Unfenced**, because providers are per-user and edited by their owner: this is operator text.
- **Opt-outs** (`skipAgentRules`): the step recap, `01-env-detect` and the model-health canary,
whose prompts carry their whole task and whose replies are parsed as they are. The sub-agent
kinds, which no step builds, get no block.
- **`CONFIG_KEYS.AGENT_RULES_INJECTION_ENABLED`** (default on, Admin > CLI execution). ABSENT in
the pure resolver means off, which keeps every direct `resolveDispatch` caller's prompt
byte-identical; a failed read means off.
- **Never a new failure.** gemini takes its prompt only as an argument, so when the rules push a
prompt that fits past `PROMPT_ARGV_LIMIT_BYTES` the dispatch is rebuilt without them and says so.
- **Recorded per run.** `spec.agentRules` (`{ hash, injected, reason? }`, the hash of the
normalised effective rules, kept even when they were not injected) rides the job to exec and is
written by the UPDATE that sets `started_at` into `cli_invocations.agent_rules` (migration 0166).
NULL means not recorded: older rows, a run that never started, a sub-agent split.
## Retrieval protocol
`_retrieval-guidance.ts` owns the one block that tells every agent how to find code and
knowledge: DISCOVER with `rag_search`, then GROUND on disk. Grounding runs on HITS as well as
misses and is not a fallback — the index is written by `02-pre-rag-sync` at run start and
`11c-rag-reindex` after the commit gate, so for any file the task itself edited it is not
stale but WRONG, and a DAG coder's siblings work in worktrees it has never seen. That fact is
stated on the `rag_search` bullet of the MCP surface block rather than in the retrieval block,
because it is true of a RUN and agent definition files outlive every run.
It renders against TWO axes, both resolved at DISPATCH and never inferred from prompt wording:
`supportsLsp` (claude-code/zai/ollama/muse/openrouter/grok) and `ragWired` (`adapter.
supportsMcp && surface.rag.enabled` — false for `amp`, and for any repo whose onboarding chose
`ragMode: 'none'`, which is a first-class option and not a fault). Builders always emit the
(LSP, rag) arm; `adaptRetrievalProtocol` rewrites it to the earned cell in ONE pass over a
four-cell table. Not two chained passes: the legacy strings named LSP and `rag_search` in the
same sentence, so each pass would rewrite the other's key out from under it and the second
would then match nothing — silently. Those strings have since been retired into the shared
block (04-phase-0b, 05-phase-0b5), which is what leaves the table with one rag-agnostic entry.
No variant may contain an EMPTY line. Six splice sites join with `.filter(Boolean)` and nine
without, so a blank element renders the same block two ways and the exact-string rewrite finds
only one class of them — half the prompts silently keeping the wrong arm. Asserted in
`_retrieval-guidance.test.ts`.
The rewrite only reaches the canonical block, so ~15 builders that name `rag_search` in their
OWN prose are not covered by it. `mcpSurfacePrompt` carries the backstop: with rag absent it
states so positively and voids any rag instruction from the prompt or from an agent definition
file read off disk. That is why an empty surface — and amp's `null` — is no longer suppressed;
silence there was exactly the dispatch whose body still named the tool.
Grep-first is CORRECT below onboarding step index 14 (`10-rag-populate`): `08-knowledge-
acquisition` (9), `09_5-skill-generation` (11) and `09_5b-skill-repair` (11.5) run before any
index exists. The boundary is that index, not a list of step ids.
## Prompt containment
Every agent reads text somebody else wrote, and no agent can tell an honest comment from one
placed to steer it. `steps/_untrusted-repo.ts` is the one home for the rules that answer that,
and `prompt-agent-paths.test.ts` is what stops them drifting: it builds every prompt the step
registry can reach — loop arms and detect variants included — and asserts these properties over
each one INDIVIDUALLY, never over their concatenation. A joined text passes as soon as one arm
carries the guard, which is how `01-plan-build`'s expand wave and `08b`'s fix pass each shipped
unguarded.
Read that file's own header before trusting it further, because its boundary is systemic: a
permissive proxy fails every strict comparison, so a builder driven that way emits the DEFAULT
branch of each conditional and no other, and an unmodelled branch would still pass. Nor does it
see what the dispatch-time augmenters ADD — `augmentPromptWithLedger` and its siblings run after
every builder — which is why the ledger's own fence is pinned in `task-ledger.test.ts` instead.
**FOUR VARIANTS, AND THE ENDING IS THE WHOLE CHOICE.** They share a first half; what differs is
what the pass is told to DO about text that tried to steer it, which has to match what its
output can carry:
- `REPO_IS_DATA_LINES` — REPORT it as a finding. For a pass with a findings array that has room
for one: 07b's validator, 08c, 08d, the lenses.
- `REPO_IS_DATA_ONE_CLASS_LINES` — protection with NO reporting duty, because the findings array
holds exactly one kind of thing. `07_7-secret-sweep`'s holds CREDENTIALS; `04a` and `05`'s
hold problems with the SPEC, so a prompt-injection report filed there arrives as a spec defect
and `05a`'s corrector EDITS the spec in response to something it read in a source file.
Routing it to `## INSIGHTS` is no better — `08e` interpolates what lands there.
- `REPO_IS_DATA_ACTING_LINES` — for a pass that EDITS. No reporting duty either, and for a
sharper reason: a coder's `concerns` reaches `recordLedgerEntry`, so quoting hostile text
there manufactures a relay into every later prompt in the task.
- `REPO_IS_DATA_AUTHORING_LINES` — for a pass whose output becomes another agent's ASSIGNMENT:
the spec writer, the sprint planner, every plan writer. The longest reach of the three shapes
these four blocks cover — a reviewer that swallows an injected line files a skewed report, a
coder WRITES it, an author turns it into an order nobody authorised.
The plan-writer set is DERIVED, not listed: whatever emits `PLAN_PATCH_CONTRACT`. A hand list
missed `01f-external-plan-sync` and `11f-plan-reconcile`, which are plan writers living under
`workflow/`. `01-advisory-research` is the one deliberate omission — its output is read by a
PERSON at a gate rather than executed, and no variant's ending fits.
**EVERY VARIANT MAKES THE PERSONA CARVE-OUT, AND IT GOVERNS METHOD.** `agentDefinitionGuidance`
tells an agent to follow the repo's own `.claude/agents/<id>.md`, so a block saying nothing
under `.claude/` is an instruction would contradict the file the same prompt just pointed at.
Two halves, both load-bearing: the definition this prompt HANDS you (points you at, or pastes
in) is your persona — one it merely NAMES, including one you were sent to edit, stays data; and
the persona says HOW to work and never what you may FIND, so narrowing, skipping, suppressing,
omitting, downgrading and reaching a verdict are all refused. A carve-out with no limit is a
licence, and the limit is the half that is easy to leave out.
**FENCE AGENT PROSE CARRIED INTO ANOTHER PROMPT.** `fencedAgentBlock` wraps a block between
`UNTRUSTED_OPEN`/`CLOSE` and runs `fenceSafe`, which collapses any run of four or more `=` —
keyed on the run rather than either banner's wording, so a reworded banner cannot reopen the
hole. Instructions ABOUT a fenced block stay OUTSIDE it: "copy the ids VERBATIM", "every one
MUST appear", "validate and fix what these DESCRIBE". `renderBoundedPlanIndex`'s omission notice
is the worked example — it is HAIVE telling the agent not to invent an id, and fencing it voided
a guard rail by its own containment.
This is not belt-and-braces over the guards: `REPO_IS_DATA_LINES` REQUIRES its reviewer to quote
the hostile string with its file and line, so hostile content reaches the next prompt BY DESIGN.
The reporting duty and the relay are two halves of one design, and the fences are that split
enforced downstream. One validator's output needed three: 07b's fixer, 07's fix-loop diagnosis,
and 07b's own re-validation.
**TWO RULES, AND THE SECOND IS THE COST OF THE FIRST.** _Fence where a person's words and a
machine's are JOINED_ — `formatRejectDiagnosis` contains the browser output and the audit
findings beside the developer's feedback, `buildGateDirectiveDiagnosis` contains the failure it
quotes under the operator's instruction — so the markers travel with the string and every later
reader inherits them. _Balance wherever the result is CUT_: `balanceFences` repairs a slice that
kept one banner and dropped the other. A TAIL slice (`cleanDiagnosis`, the prior-round entry)
drops the BEGIN and leaves the contents loose; a HEAD slice (`loadHonoredConstraints`) drops the
END, and an unmatched BEGIN swallows the rest of the prompt. A block fenced WHOLE whose budget
drops WHOLE entries — the task ledger — needs no balancing, because nothing can cut one in half.
**NEVER FENCE THE OPERATOR.** A fence tells the agent not to follow what is inside it, which is
containment for AGENT prose and sabotage for a person's. `REPO_IS_DATA_MERGE_LINES` exists
because reusing the coder guard in the merge prompt pitted it against the operator's own
conflict guidance. `fixIsHuman` keeps a gate-2 rejection out of the fence; `HUMAN_REJECT_SOURCES`
splits the prior-round block; plan chat fences its own turns and never the user's. The ASSIGNMENT
itself is never fenced — a coder must follow it — so hostile text entering one is stopped
upstream, at the writer, which is what the authoring variant is for.
**A VALUE NAMED ON A PROMPT LINE.** Above the guard or below it, a value interpolated onto a line
of its own can open an instruction line. What must not be REWRITTEN is FILTERED — a filename or
a KB id the agent then opens, where `isSingleLine` and `survivesFence` decide and a mangled name
is worse than an absent one. What is prose is COLLAPSED (`collapseToLine`, `safeTitle`). The
character class is every Unicode control — C0, DEL and C1 — plus U+2028 and U+2029, with TAB the
one carve-out. A RANGE, because enumerating cost three review rounds and was wrong every time:
`\s` misses U+0085, a C0-only class misses the line separators, an ASCII class misses
U+001C-U+001E.
**FILTER AND FENCE AT PROMPT-BUILD TIME.** `detect_output` is PERSISTED and `step-runner` replays
it, so anything applied in `detect()` never reaches a step detected before it shipped — the
reason `fencedDebtBlock` and `assertReviewableChange` live where the prompt is assembled, and the
bug `listKbFiles` shipped with.
## RAG embedding failures
**A failed embed never becomes a hash vector.** `hashEmbed` is a deterministic SHA-256
vector with no semantic content, so once an index holds real vectors a hash row is NOISE
in the dense half of the RRF fusion — it can outrank a genuine lexical hit, and nothing
downstream can tell the two apart. Every ingest loop used to `catch` any embed error and
substitute one, logging a `warn`. MEASURED on an 8-core CPU-only host with
`qwen3-embedding:4b`: a batch of 8 real code chunks takes 50-69s (10.5s on GPU) against
the old hard 60s budget, so on a CPU host that substitution was the DEFAULT outcome, not
an edge case — and the index it produced looked healthy.
`embedBatch` (`step-engine/steps/_rag-embed-health.ts`) is the one embed path for all three
loops (`workflow/_rag-index.ts` for 02/11c, `onboarding/10-rag-populate.ts`,
`queues/global-kb-sync-queue.ts` — which was also UNBATCHED, sending up to
`MAX_CHUNKS_PER_FILE` 80 chunks in one call). It returns `embedded` / `hashed` / `failed`.
`hashed` is reachable only two ways, and neither is a mid-run accident: the repo has no
embedding endpoint at all (every chunk hashes, which is homogeneous and therefore honest),
or `CONFIG_KEYS.RAG_EMBED_STRICT_ENABLED` is off, which restores the old behaviour byte for
byte and is the no-deploy rollback.
**What a failure does depends on WHEN the step runs.** `10-rag-populate` FAILS — "the index
is populated" is its whole contract and a human is watching onboarding. `02-pre-rag-sync`
and `11c-rag-reindex` leave the chunks UNINDEXED and carry on: 02 runs at the start of every
workflow task, so failing it would block all work on the repo over a broken index. An
absent row is honest; a stale row left by a skipped update still points at the right file.
**Two timeouts, because one call serves both jobs.** `ollamaEmbed` is used by bulk ingestion
AND by the interactive `rag_search` query embed, so a single budget is simultaneously too
tight for a CPU batch and far too loose for a tool call (MEASURED: one query embed is 0.44s
on the same CPU host). `RAG_EMBED_TIMEOUT_MS` (240000) and `RAG_QUERY_EMBED_TIMEOUT_MS`
(20000) split it; `resolveEmbedBudget` falls back to the module constants when ConfigService
is uninitialized, because `worker/scripts/rag-eval.ts` imports the module standalone.
Lowering `RAG_EMBED_BATCH_SIZE` is the better fix for a slow host than raising the timeout.
**Per-repo memory, because a step row does not survive a retry.** `repositories.
rag_embed_degraded_at` is the STRUCTURAL flag every reader gates on;
`rag_embed_degraded_reason` beside it is display copy that outlives the state it describes
(the message-column rule above). Only a completed run with REAL embeddings clears it — a
hash-mode run proves nothing about whether embeddings work.
**`rag_embed_lexical_only` is the accepted verdict, and it is NOT "keep hashing".** It forces
`ragHybridSearch`'s existing lexical-only branch (the one a jsonb-only store already takes),
so an accepted repo gets honest full-text ranking instead of full-text plus noise — VERIFIED
on a live 9,197-row index: the same query returns 5 hits either way, `maxDense` 0.7233 with
the dense half on and 0.0000 with it off. The api reaches that branch by two routes and both
skip the vector rather than hashing one: the repo flag, and `embedQueryOrNull` returning null
for a single query that could not be embedded. `embedQuery` keeps its hash fallback for
callers that must have a vector of the right width.
Recovery re-embeds only where hash rows can actually exist — leaving lexical-only mode, or
the explicit Rebuild action for a repo indexed before this existed (those carry no
degradation record and have no other route back). A repo that merely failed under strict mode
has no hash rows, so its next incremental sync picks the missing chunks up as ordinary
inserts. `forceRagReembed` nulls `chunk_hash` rather than deleting: rows stay searchable
until they are replaced, and that RECOVERY path issues no DDL of its own.
Do not read that as "Haive never issues DDL against a user-owned store" — it does.
`ensureRagSchema` never reads `conn.mode`, and `ddev` is routed through `resolveExternal`, so
the full path (`CREATE EXTENSION vector`, `CREATE TABLE`, the indexes, `CREATE OR REPLACE
FUNCTION update_content_tsv`, the dedupe `DELETE`) runs against `external` and `ddev` stores
exactly as against `internal`, on every sync. The mode-aware protection is on the DESTRUCTIVE
side only: `cleanupRagForRepository` drops databases and is kept off external/ddev by its
CALLER filtering `projectNames`, not by the function itself.
## Identifier search
**Postgres' text-search PARSER splits snake_case, so a repo's own identifiers were not
searchable as themselves.** MEASURED: `to_tsvector('english','user_has_monter_role')` yields
`user`/`monter`/`role`, `pdf_generator` yields `pdf`/`generat`, and `'simple'` splits
identically — the split happens before any dictionary, so no configuration fixes it. Only
dotted paths (`d_product.inc`) and camelCase (`CNotificationEmail`) survived whole. The
lexical half was therefore matching common English words, which is why `plainto_tsquery`'s
AND of every term matched ZERO rows on 17 of 18 real queries, and why simply switching it to
OR measured WORSE: it promoted `includes/module.inc#module_enable` (dense sim 0.2632) to rank
4 of 12 where every other hit was above 0.65.
**Coverage ranking is the trap to avoid.** Ranking OR-matches by how many query terms a chunk
contains scores better in aggregate (mean dense of added rows 0.509 vs 0.454) but put the
`user_has_monter_role` DEFINITION at rank 137 — a focused function chunk contains few of a
query's terms, so coverage rewards long rambling chunks. Rejected on that measurement.
`identifiers.ts` (`shared/src/rag`) holds ONE extraction rule. `identifierTsvSql` builds the
`update_content_tsv` trigger's SQL and `extractIdentifiers` runs on the query, both from
`IDENTIFIER_PATTERN`, so the index and query sides cannot disagree about what an identifier
is — VERIFIED across 2,000 real chunks and 20,295 lexemes, zero mismatches. Three details are
load-bearing and each cost a measured failure to find:
- **Dotted tokens also emit their SEGMENTS.** The pattern matches greedily, so
`pdf_generator.class.inc` is one token and the bare `pdf_generator` would never be indexed
— which alone made the feature miss the case it exists for.
- **The pattern is embedded with SINGLE backslashes.** `standard_conforming_strings` is on, so
doubling them makes the regex engine read `\\.` as "a literal backslash, then any
character"; MEASURED, `d_product.inc` then came back as `d_product` + `inc` and the
whole-path lexeme was never stored.
- **Two camelCase humps, not one.** `[a-z][A-Z]` misses PascalCase with a single-letter prefix
— `CProduct` and `CPDF` contain no lowercase-then-uppercase pair — so a second clause wants
an uppercase-then-lowercase pair NOT at the start, which admits `CProduct` while still
rejecting capitalised prose (`Postgres`, `Excel`) and all-caps words (`PDF`).
Lexemes go in via `array_to_tsvector`, which BYPASSES the parser, and are matched with the
literal-lexeme cast `'pdf_generator'::tsquery` — never `to_tsquery`, which re-runs the parser
and re-splits them (`to_tsquery('simple','pdf_generator')` is `'pdf' <-> 'generator'`). The
`english` prose half is untouched, so no configuration mismatch is introduced. Cost measured
on a 9,197-row store: tsvector total 8.9 MB to 10 MB, 9.3 lexemes added per chunk.
**Ranked by IDF, and that is forced, not preferred.** `array_to_tsvector` stores POSITIONLESS
lexemes and MEASURED `ts_rank_cd` returns exactly 0 for them, so the usual ranker cannot order
identifier matches at all. IDF also fixes flooding: `mpdf` is in 22.8% of chunks and
`pdf_generator` in 0.03%, and an unweighted OR let the common term bury the rare one —
`ln(N/df)` separates them 1.48 vs 8.03 and needs no tuned threshold.
**`identifierCandidates` (3) bounds the INJECTION, and lowering the weight is the wrong knob.**
The ranker is a third RRF input at full weight, but drawing `candidatePool` candidates let it
rewrite the page: MEASURED it displaced `notification_bell.tpl.php` (dense 0.725) from a query
that literally says "notification bell". At HALF weight the motivating case stops surfacing
entirely; at a cap of 3 and full weight it surfaces and displaced top-3 code hits fall from 3
of 18 to 2. An exact identifier match is worth a couple of slots, not the right to rewrite the
ranking — the same bargain the knowledge reserve strikes.
**A query with no identifiers omits the CTE entirely**, so the statement is byte-identical to
the one that shipped before this existed — VERIFIED on the 3 such queries of 18.
`CONFIG_KEYS.RAG_IDENTIFIER_SEARCH_ENABLED` off restores the previous ranking exactly even
against a store that already carries the lexemes, because they are then never queried; that is
the rollback, and no data has to be undone.
**The backfill is not optional, and the reason is not obvious.** `_rag-index.ts` skips a chunk
whose `chunk_hash` is unchanged, and a change to how `content_tsv` is DERIVED leaves `content`
— and the hash — identical, so no existing row is ever re-upserted and the new trigger never
fires on it. `backfillIdentifierTsv` therefore rides `ensureRagSchema` (which runs at the start
of every workflow task via `02-pre-rag-sync`), in the same shape as
`dedupeAndEnforceRepoUniqueness`: warns rather than throws, keeps no applied-record, and
converges structurally — `SET content = content` fires the trigger, which stamps
`IDENTIFIER_TSV_SENTINEL`, and a stamped row never matches the predicate again. MEASURED,
9,197 rows in 17.5s, and the next run is 1.2s writing nothing. The sentinel deliberately
contains no `_`, `.` or hump so `extractIdentifiers` can never emit it from a chunk's own text
— otherwise indexing Haive's own source would stamp a row as upgraded when it was not, and a
version bump would skip exactly those rows. It is capped per invocation so a large store
finishes over the next few syncs rather than making one task wait for the whole rewrite.
## Knowledge reserve in retrieval
**A repo indexes two orders of magnitude more code than KB, so the KB has to be given
slots — it cannot win them.** MEASURED on a live 9197-row index: 9156 code chunks against
41 KB chunks (223:1), and the KB chunk that ANSWERED the query ("major functional areas of
the activit module") sat at dense rank 41 behind 40 code chunks, so a `top_k` of 12 returned
zero KB. 11 of 18 logged queries returned no KB at all. Per chunk the KB was the BETTER
match (avg dense 0.4133 vs code 0.3170); code won on volume alone. Not a floor problem —
`denseFloor` is 0.3 and the best KB chunk scored 0.5926, and two KB rows were already inside
the 50-row candidate pool. They were outranked, not gated. This is the mirror of `b8bcb2af`,
which added `codeDenseFloor` because code was losing to KB prose: a scalar floor tuned
against corpus size flip-flops, so the fix is a QUOTA.
`applyKnowledgeReserve` (`shared/src/rag/search.ts`) holds `knowledgeReserve` (2) slots of
every page for `KNOWLEDGE_SOURCE_TYPES`. A quota rather than an RRF multiplier because a
multiplier cannot close a rank-41 gap without over-promoting the queries where the KB already
surfaces, and the gap varies per query. It is a floor on PRESENCE, never a re-ordering: only
rows the page would otherwise CUT are promoted, so a KB hit that earned rank 2 stays at rank
2, and promoted rows keep the low `rrf` that got them cut rather than a fabricated score
(`rrf` is shown to agents and logged as `maxRrf`). Admission is RELATIVE — `denseSim >=
knowledgeReserveRatio` (0.75) × the best CODE hit's — because MEASURED across 18 real queries
best-KB/best-code ran 0.630-1.043 and the only two under 0.75 were pure symbol lookups
(`CProduct class d_product.inc …`, `PDF certificate generation mPDF …`) where the KB
genuinely has less to say. So no absolute threshold needs re-tuning per corpus. The effective
reserve is `min(knowledgeReserve, floor(topK / 3))`: `top_k` reaches this route as low as 4,
where a flat 2 is half the page.
**`task` is not knowledge**, which is why `KNOWLEDGE_SOURCE_TYPES` names its three members
positively instead of testing `<> 'code'`. `workflow/_task-embedding.ts` writes one row per
task keyed by task UUID rather than a file, for the effort estimator alone, and it has no
counter in `logRagQuery` — one surfacing in an agent's results would be an invisible hit.
The positive form is also what uses `idx_rag_source_type`: MEASURED 1.6 ms against 8.6 ms for
a `<> 'code'` sequential scan on the same index.
Candidates come from a SECOND small query, not another CTE unioned into the ranking SQL: a
row reached only through such a union carries no `d_rank` and no `l_rank`, scores `rrf = 0`,
and the main statement's own `ORDER BY rrf DESC LIMIT` discards it before any caller sees it
— the reserve would have had nothing to draw on. It applies `denseFloor` (never
`codeDenseFloor`), so a promoted row has cleared the same relevance gate the main query holds
its own rows to. The reserve is INERT wherever the dense half is (`usePgvector` false — a
jsonb store, an accepted `rag_embed_lexical_only` repo, or a query that could not be
embedded), since every row there reports `dense_sim = 0` and the ratio has no signal, and
inert on any page with no code hits, which is the global KB store (every row `kb`) — so
neither global caller is perturbed and neither pays for the extra query.
`mergeHits` re-applies it when trimming the local page for the global KB. That is not
belt-and-braces: a promoted hit carries exactly the low `rrf` that got it cut, so a plain rrf
slice drops it FIRST and the reserve would die at the merge. `knowledgeReserve: 0` restores
the previous ranking exactly.
## Plan canvas
`plan_nodes` and friends (`packages/database/src/schema/plan.ts`, migrations 0130/0131) are the durable, per-repo tree of what a project is MEANT to be — the intentional counterpart to the KB, which only describes how the code currently is. One plan per repo, enforced by a partial unique index on the root node rather than a `plans` table. A repo with no remote and no local tree is a first-class `source: 'blank'` repository: the repo-queue INIT job creates the storage dir, `git init`s it and lands one commit, so worktrees, task attachments and the `.haive-data/` mirror all work on a project that does not exist yet.
**One writer.** Every write — LLM turn, UI edit, markdown import — goes through `applyPlanPatch` in `@haive/shared/plan` (NOT the worker: the api writes plans too and cannot import the worker; `global-kb/task-context.ts` is the precedent for a shared package taking an injected `db`). It owns the materialised ancestry (`path`, self-inclusive and slash-terminated so one `LIKE 'prefix%'` selects a subtree), the `version` optimistic-concurrency check (a stale `expectedVersion` is a 409 the UI must SHOW, never a silent refetch), and subtree moves, which rewrite every descendant in one prefix-substitution UPDATE. An agent's patch must carry that version for every existing node it changes or deletes (`requireExpectedVersion`, set by `applyAgentPatch`): an op without one is dropped and reported, where it used to skip the check and replace whatever a person had changed on the canvas since the agent read the plan. For the same reason it deletes only a node with no children, so a subtree goes leaf by leaf, each delete with its own version: the cascade from a parent would take descendants whose versions it never sent. An ordinal-only reorder is exempt, because the sequencing step writes a whole build order at once and a node-level version would reject it over an unrelated edit to one child (MEASURED: 1,350 of its upserts, none versioned); plan chat is shown every node's version (`withVersions`), which the committed mirror is not. The SQL offset must be cast — `substring(path from $n::int)` — because an untyped bind resolves to Postgres' POSIX-REGEX `substring(text, text)` overload and returns NULL. `importPlanMirror` is the ONE deliberate bypass: a mirror restore recreates nodes with their original ids verbatim (the spec writer and plan.md quote those ids), which the patch contract deliberately cannot express; it runs only when the repo has zero plan rows and refuses whole rather than half-import when the ids collide.
**Plan task types are not in `workflowTypeSchema`.** That zod enum is the create-task REQUEST validator; `plan_build` / `plan_chat` / `advisory` need a node id the generic form has no field for, so they are spawned by their own endpoints (`POST /repositories/:id/plan/...`) — the same arrangement `kb_author` already has. They live in the DB enum and in web's local `WorkflowType` union only.
**Completion greens the node** from the worker's `markTaskCompleted` (the only completion write; cancel and fail have their own functions, which is the point). Only `todo`/`in_progress` advance — `blocked_human` and `not_applicable` are verdicts a person entered, and a task finishing is weaker evidence than that. Status roll-up is DERIVED at read time (`rollUpStatus` in shared, one source for every view) and never stored: a `blocked_human` descendant makes every ancestor render blocked; green requires every descendant `done` or `not_applicable` — `not_applicable` must not prevent green, but must never render green itself either.
**Two triggers, one builder.** `createPlanBuildStep` (steps/plan/01-plan-build.ts) serves both the standalone `plan_build` task and the onboarding wrapper `10_8-plan-build` (index 14.5, after the KB and RAG exist, before the mirror is staged), because `onboarding_upgrade` reconciles template artifacts only and never re-runs onboarding steps — an already-onboarded repo can reach the builder no other way. The build is LEVEL-BY-LEVEL (wave 0 drafts root + level 1 via `selectAgents`; each later wave is thrown from apply() as a `MiningWaveError`, one agent per frontier node) because one pass cannot emit 400 nodes without truncating. Do NOT move it to `loop`: the runner's loop re-entry calls `resolveLlmPhase`, which asserts `stepDef.llm` exists — a mining-only loop step dies on its second pass with "reading 'prepare'" (measured, on a real repo). The frontier is recomputed from the DB every wave, and excludes `research`/`external` nodes (a blocker waiting on a person is not a system to decompose) and `taskable` leaves; per-wave agent ids (`plan-expand-<nodeId>-p<N>`) double as the asked-set, so a node is never re-asked even though apply passes are independent calls. Because a wave re-runs apply() with the CUMULATIVE result set and temp-ref creation is not idempotent, the fold is over `newAgentMiningResults` — `task_step_agent_minings.consumed_at` (migration 0132) is stamped on every row apply has folded right before the next wave dispatches and cleared on a re-roll, each row only at the run apply read (`consumeFoldedMiningRows`), so one another pass re-rolled meanwhile keeps its reply; for steps that never throw MiningWaveError the new field equals the cumulative set, so existing steps are unchanged. That set is decided once per pass, so each reply is also CLAIMED as it is folded (`applyAgentPatchOnce`, `_plan-prompt.ts`): its row's `consumed_at` is set, only while unset, in the same transaction as its patch, and so is the row's partial-apply note, since every later pass skips a claimed reply and a note written after the commit could be lost for good. Two apply passes over one wave, or one pass that re-runs apply in place after a wave that sent nothing, then write a reply once, and a reply another pass claimed counts as applied, so the builder's whole-wave re-roll does not fire on a wave that did land. 01, 02 and 03 all fold this way. A `ReopenStepFormError` consumes the rows apply folded and clears its detect output, form and answers in ONE transaction with `pause_form_on_retry` set, before it detects afresh, and writes the row only while the pass still owns it, since a Retry's reset keeps its own hold; a worker that dies before the new form is parked re-detects and stops at the form instead of re-sending the answered items with the old answers. The submit that led there can be redelivered, still carrying those answers, and a form submit carries no epoch to drop it by. While the hold is set, values a job carries are ignored, since the form has not been offered since. Once the form is parked, `isStaleSubmit` drops a submit older than the park. Every form park is stamped (`waiting_started_at`) in the write that parks it, not after, so there is no moment when a parked form has no park to be older than. A pass that dies between that write and marking the task waiting leaves the task `running`, and a stale submit landing on one parks the form again (`staleSubmitAction`) rather than only being dropped.
**Two modes, and the difference is the node status.** `from_repo` mines the KB and its nodes
arrive `done` because they describe code that exists; `greenfield` takes a written brief plus
any number of attachments and leaves them `todo`. A brief used to ride `from_repo`, which was
not inaccurate prompt copy but a project rendered as already finished. `from_md` (one inline
markdown document) is RETIRED from `planBuildRequestSchema` and deliberately alive in the
worker's `BuildMode`, so stored tasks stay retryable. An unrecognised stored value falls to
`from_repo` — a default rather than a throw, because that is what a revert would also produce.
**Attachments exist before the worker reads them, or the task never starts.** The plan
builder's first step reads the uploads dir and the worker picks a job up immediately, so
`POST /plan/build` with `deferStart` creates the row WITHOUT enqueueing; the client streams
files through the ordinary attachment route and finalizes with the `start` task action, which
claims `status = 'created'` in one atomic UPDATE (a second click matches no row and enqueues
nothing). An upload that fails therefore cannot start anything — the draft is the repair
surface, reachable from its Attachments tab. This replaced the inline-document seed hook,
which could carry exactly one file. `_ATTACHMENTS.md` is regenerated after every write: the
prompt notice names that file unconditionally, so a path that skipped it handed the agent a
pointer to nothing. Every writer — upload, delete and archive expansion — goes through ONE
function, `rewriteAttachmentsManifest`
(`@haive/shared/attachments-fs`, Node-only and outside every subpath web imports), and it NEVER
throws: it runs after the work it describes, so a refusal there answered 500 for an upload that had
already happened, and the client's retry stored the file twice. A link planted at a GENERATED name
(the manifest, `_PLAN_INPUTS.md`, a sidecar) is replaced as a link (`writeFileNoFollow`'s
`replaceLeafLink`, which never follows it and inherits nothing from it) rather than blocking every
later write; anything else standing there is logged and left, and `00-plan-inputs` carries on
without that index or sidecar instead of failing the step.
**An attachment's `filename` is a RELATIVE PATH, not a basename.** That single fact is both
folder upload and archive expansion: the browser sends `webkitRelativePath` from a
`webkitdirectory` input (its own button beside the file picker, never a mode on it), and
expanding an uploaded archive produces exactly the rows a folder upload would have. Nothing
downstream needed a special case, because `resolveTaskUploadsMount` already bind-mounts the
WHOLE uploads dir and every consumer joins the name onto it. The rules live in
`@haive/shared/attachments` so the api (on upload) and the worker (on expansion) cannot
disagree about a name: a `..` segment THROWS rather than being dropped — every legitimate
producer can express a path without one, so its presence is an upload to refuse (400), not a
name to repair quietly — and leading dots are stripped per SEGMENT, so no dotfile or
dot-directory survives anywhere. De-dupe is per DIRECTORY: two folders' `README.md` are two
documents. It keeps an archive's two-part extension whole (`splitAttachmentExtension`): a second
`spec.tar.gz` is `spec (2).tar.gz`, never `spec.tar (2).gz`, which no archive rule recognises.
`ensureArchivesExpanded` (`worker/src/attachments/expand-archives.ts`) runs lazily wherever the
attachments are read for an agent: `00-plan-inputs.detect`, `loadLiveAttachments` (so the live plan
inputs see an archive's members, not a `binary` zip), and before `augmentPromptWithAttachments` in
`step-runner` and in `dag-executor`, each time either builds an agent's prompt, so
every task type gets it and the notice never names a `.zip` no agent can open. It reuses
`repo/clone.ts`'s `extractArchive` (`unzip` and `tar` are in the WORKER image, not the api's;
its `flattenSingleTopLevel` is what stops `spec.zip` becoming `spec/spec/`), then walks the
result with `lstat` and keeps REGULAR FILES ONLY — a symlink is how an archive reaches out of
the directory it was extracted into — re-checking every path even though the extractor claims
to sanitise. All-or-nothing per archive: over `ATTACHMENT_ARCHIVE_MAX_FILES` (500) or
`ATTACHMENT_ARCHIVE_MAX_TOTAL_BYTES` (512 MiB) it inserts NOTHING and records
`expansion_note`, because half a specification is worse than none — nothing downstream can
tell which half it was given. The cap is measured AFTER extraction on purpose: the alternative
parses `unzip -Z`/`tar -tv` human-facing output, and a bomb is exactly the input that lies in
it. `expanded_at` is stamped for every archive that will not expand — a cap breach, an unreadable
archive, no free folder name — or a failed archive is re-extracted on every step for the life of
the task. A PLACEMENT that fails is the one exception: it takes back all it placed and leaves the
archive a candidate, since a lost database round trip is no verdict on the archive.
`expanded_from_id` cascades the rows AND is what makes "nested archives are not recursed"
structural (a row with a parent is never a candidate). The FK
cannot reach the disk, so a delete removes what the attachment left there too
(`attachmentRemovalPlan`): an archive's MEMBERS, which a folder delete has to reach at the uploads
ROOT (`docs/x.zip` expands into `x/`), and each document's extracted-text SIDECAR. A delete that takes
the LAST file extracted from an archive takes the archive too (`archivesEmptiedBy`,
`@haive/shared/attachments`, which the panel's confirmation names from the same rule), or it would
stay listed as expanded with nothing of it left and never be expanded again. A file a SURVIVING row
still names is never removed: the upload claim reads the disk, not the rows, so two rows can name one
file, and deleting one must not take the other's. It removes FILES,
never a directory, and a folder goes only by pruning once it is empty: a recursive removal takes
whatever else lives there — a later upload named like the expansion directory lands inside it (the
api de-dupes files, not directories), and an upload racing the delete has its file on disk before
its row exists, where no list of rows can see it. That makes the rows the whole inventory, and two
rules keep them so: no attachment may take a name a generated file owns, and no expansion leaves a
placed file without its row (below). The first rule is ONE predicate
(`isReservedAttachmentName`, `@haive/shared/attachments`) that an upload, an archive member and an
expansion folder all apply: `_ATTACHMENTS.md` and `_PLAN_INPUTS.md` at the root, `*.extracted.md` at
any depth, since a delete unlinks a document's sidecar path whether or not the sidecar exists yet. A
file is probed onto the next free ` (n)` name; a FOLDER is renamed `<name> (2)` deterministically
(`reserveAttachmentDirs`), because a folder upload is one request per file and every file of it has
to land in the same place. Neither may be left: an orphaned tree stays bind-mounted, and an agent
would keep reading files the user believes they removed.
**One lock serialises every write that changes which files a task has.** `withTaskAttachmentsLock`
(`@haive/database`) is a transaction-scoped advisory lock keyed on the task — `plan/mirror.ts`'s
shape — that the api and the worker share: an upload CLAIMING its name, a delete from reading its
rows to rewriting the manifest, a sidecar stored only while its row exists, and every manifest
rewrite. A delete removes a sidecar path and its row in one section, so no sidecar can land between
the two; that replaced a two-sided check in which the api removed sidecars twice and the worker
looked for the row after writing. Three rules make it safe to hold. Sections are SHORT — never
across an upload stream or an archive extraction, because every waiter holds a pooled connection
and both pools are `max: 10`. Everything inside goes through `tx`, never the pool. And nothing that
takes the lock is called on the POOL from inside a section: handed `tx`, it nests as a savepoint,
where the lock is re-entrant. A wait past 30s is `55P03` (`isLockNotAvailable`): the api answers 503
having changed nothing, and the worker counts it as a per-item miss — a sidecar not stored, a
missing file still missing. `00-plan-inputs` re-checks a file it found gone under the lock before
calling it missing, since a delete removes files before rows and a deleted attachment is not a
missing one. The upload's INSERT stays outside the lock with its id chosen beforehand, so one whose
answer was lost can be told from one that failed: the file is taken back only when a second look
finds no row, since answering 500 for a stored upload sends the client's retry to store it twice.
**The expansion is the lock's longest section, and a crash anywhere in it leaves something the
next call can settle.** Each attempt extracts OUTSIDE the lock into its own
`.expanding-<archiveId>-<nonce>` staging dir and builds the finished tree there. The dir is 0700:
the extraction tool is handed its directory as a descriptor and never resolves it by name, so only
the worker enters it. The nonce is what stops a second call
moving the first's tree aside, which `extractArchive` does to any existing destination. One locked
section then re-checks that the archive is still unstamped, settles earlier attempts at it, claims
the first free folder by moving the tree WHOLE (`renameNoFollow` with `noReplace`), and writes every
member row and the stamp together — so two overlapping calls produce one tree. Before each move it
writes `placed-as`, naming the folder and its files, and the staging dir goes only after a confirmed
COMMIT, so an attempt that died after its move and before its rows is taken back by the next call
to hold the lock: whatever `placed-as` names that no row owns is removed. A tree still staged means
the move never happened, and then only the claimed folder goes, and only while it is an empty
directory. The intent is untrusted — the sandbox can write the uploads dir — so it may name one
folder and only names an expansion could have written: a sidecar has no row, and an intent naming
one would pass live extracted text off as an orphan. Nor more names than an archive may hold, since
each is a bind parameter of the settle's one query, and a list past Postgres' limit fails a delete's
section after its files have gone. A throw after the move takes the files back
INSIDE the callback, since postgres.js can reject the transaction while the callback still runs.
Every call first settles the attempts at archives deleted or stamped since, re-checked under the
lock, so an archive attached a moment ago keeps its attempt in flight. A DELETE settles the attempts
at the archives it removes in its own section (`settleExpansionAttempts`,
`@haive/shared/attachments-fs`), and a staging dir it then fails to remove is swept by the next
expansion call. The worker finds the uploads dir through a row, so once the last attachment is gone
it sweeps under the task repository's storage path, the one root an upload is ever written under,
and never in a read-only local repository. Settling
removes the intent inside the section, so no later settle acts on it twice — a name it freed is one
an upload can take, and that upload's file has no row until its bytes are in. An upload's claim
settles every attempt that wrote its intent before it takes a name (`settleExpansionIntents`): a
dead attempt's intent can still name a path free on disk, and settling it once the upload held that
name would take the upload's file before its row existed. Their staging dirs go after its section,
as a delete's do. The staging dirs
themselves are removed AFTER the section (`removeExpansionStagings`), never inside it: one can hold
a whole extracted archive, and removing it there would hold the lock and a pooled connection for as
long as that takes.
**`expansion_note` is the one durable account of what an archive lost, so it is read from the
column.** The expansion call reports only the archives THAT call expanded, so a later step, a
retried `00-plan-inputs`, and every workflow task (whose call site discards the result) used to
see nothing. `augmentPromptWithAttachments` now names each incomplete archive in an INCOMPLETE
ARCHIVES block, `00-plan-inputs` takes its `archiveNotes` from the column, and the attachments
panel shows it under the archive (`archiveExpansionBanner`, `web/src/lib/step-banners.ts`). All
three gate on `expanded_at`; the note is only the words. It carries archive member names (tar
keeps a name's bytes, newlines included) and an extractor's error line, so it is WRITTEN as one
bounded line (`expansionErrorLine`, `describePathDrops`) and collapsed and capped AGAIN wherever
it meets a prompt (`safeNote`), because rows written before that exist.
Three caps exist because a folder is not a handful of files. `augmentPromptWithAttachments`
rides every agent that works from the task's content: the LLM phase, each agent of a mining
fan-out (built once per fan-out, joined ahead of the ledger and terseness in `resolveLlmPhase`'s
order), and each DAG coder, reviewer, fix coder and issue advisor. The DAG replanner and merge-fix
agent go without it: the first is handed the issue graph and no specification, the second resolves
a git conflict. A RECOVERED mining agent, a wave agent `selectAgents` never authored, is sent the
prompt its step wrote (`task_step_agent_minings.dispatch_prompt`), which is augmented once with what
the task knows now and adapted once for the provider that takes it. A row written before that column
has only its last run's stored prompt, already the effective one, so it is replayed verbatim:
augmenting it again duplicated the terseness block and could duplicate the ledger. So past
`ATTACHMENT_PROMPT_FILE_LIMIT` (40)
the notice collapses to one
counted line per top-level folder and states the elision the way `changedFilesBlock` does —
under the limit the output is byte-identical to what it always was. `00-plan-inputs` bounds
sidecar EXTRACTIONS (a subprocess each) at 50, and a document past that is
`extractionSkipped`, deliberately NOT folded into `visualOnly`: that is a hard vision
requirement on the dispatch and must rest on a measurement, and "the extractor never ran" is
not one. `02-plan-coverage` caps displayed gap candidates at 60, since each carries its body
into the persisted detect payload and becomes a checkbox a person has to read; a re-run
filters out the handled ones, so the remainder is the next round's list. The on-disk
`_ATTACHMENTS.md` and `_PLAN_INPUTS.md` stay UNCAPPED — they are where the prompt sends a
reader for the names it could not fit.
**00-plan-inputs** (index -1, deterministic, no CLI) verifies every attachment row has its
file, refuses a greenfield build with neither a brief nor a file, and writes a readable form
of the kinds no CLI can open — `.docx`/`.xlsx` via `unzip -p` and a small in-repo OOXML reader
(`_plan-inputs.ts`; nothing in the pnpm store parses XML, and OOXML element names are
ISO-29500, so this matches an invariant not someone's formatting), `.pdf` via `pdftotext`.
Sidecars land beside the originals with the api's same chown-to-1000 dance and are indexed by
`_PLAN_INPUTS.md`, which the greenfield prompt names. An extraction that THROWS is reported
per-file and never fails the step — the original is still mounted — while one that yields
nothing is reported as empty, because "unreadable" and "says nothing" are different facts. An
extractor still running after 120 s is killed and reported the same way (`EXTRACT_TIMEOUT_MS`),
since a build's dispatch extracts too, and a hung `pdftotext` there would hold the whole wave.
`02-plan-coverage` reads that normalised set: it used to read the FIRST attachment as UTF-8
regardless of kind, so a PNG arrived as mojibake whose headings were whatever followed a
newline and a hash. Sections carry their `source`, so a gap names its file and two files'
line 12 stay two gaps.
**What a model must SEE is a hard `vision` requirement; a PDF that has text is a soft
preference.** A model with a learned `modelLimits.vision === false` is told by
`NO_VISION_BOUNDARY_PROMPT` not to open images at all, so handing it a wireframe produces a
confident plan that ignored the input. The dispatcher therefore EXCLUDES such a provider when
`vision` is in the capabilities and fails with a message naming what to change; the capability
rides `AgentMiningDispatch.capabilities` rather than the mining spec, because it depends on
the task's inputs and not on the step. `preferVision` is the soft half — it orders known-blind
providers last without excluding any, for an input that has both a visual and a textual form.
The set is `visualOnlyInputs`: images, PLUS any document that needed extraction and yielded no
text. A wireframe PDF is the second case and is why the split is not simply "images" — it is
large because of its pictures, `pdftotext` returns a handful of labels or nothing, and with no
sidecar to fall back on a blind model has nothing at all. That verdict is `ExtractionResult.
hasContent`, decided from the document's own content and NEVER from the rendered markdown:
this module adds a `---` between pages and a `## Sheet` heading per sheet, so `markdown.trim()`
is non-empty for a document that says nothing. MEASURED — a 31 MiB all-picture PDF extracted to
exactly `---`, was recorded as having text, and the requirement that exists to stop a blind
model planning around it did not fire. `PlanInputRow.hasText` carries it structurally beside
`note`, which is display copy nothing branches on.
`DEFAULT_TASK_ATTACHMENT_MAX_BYTES` is 256 MiB for the same reason: the files this feature
exists for are the large ones, and the upload is STREAMED with the cap as a running byte
count, so it bounds disk rather than memory. Raising a default is inert on an install that
already seeded the old one (`seedDefaults` is `setnx`), so `reconcileRaisedDefaults` lifts a
stored value that is still exactly the previous default, on every boot, idempotently — the
deploy path a config-only change would otherwise not have.
**Plan inputs follow the attachments, deletions and additions alike.** `00-plan-inputs` records
the set once, and 01's detect copies what it found into a PERSISTED payload, so a deleted picture
kept demanding `vision`, the index the root prompt reads FIRST kept naming a deleted file, and a
document attached after 00 ran was never extracted at all. `withLiveInputs` (`01-plan-build.ts`)
recomputes those fields at DISPATCH, the root and every wave, through `currentPlanInputs`
(`00-plan-inputs.ts`), and `02-plan-coverage` drafts its gate from the same function and hands
every agent it dispatches the same requirements. A deletion drops the input with every verdict it
carried, and a measured verdict never changes otherwise: a PDF that yielded no text stays
visual-only. An addition is prepared exactly as 00 prepares one (`preparePlanInput`), inside the
same 50-extraction budget, and
the note of an archive expanded since joins it. Preparing can take minutes, and the attachments
notice the dispatch builds afterwards names what is attached THEN, so a pass that prepared anything
reads the rows again and another pass catches up with any that changed, up to three in all.
Membership is by ROW, never by name: a file deleted
and re-uploaded under the same name is a different document, prepared on its own, and never
inherits the verdict of the one it replaced. When anything changed, the index is re-rendered, or
removed once nothing in it is left, and the result is written back over 00's output by
compare-and-set (`WHERE output = <what was read>`), so the next wave neither extracts the same
document again nor loses what it found, while a 00 retry, which resets that output, makes the write
match nothing. That record is also why nothing falls back to the detected fields: after it, a late
picture is a recorded input rather than an addition. A file attached since whose bytes cannot be
read is left unprepared and counts by KIND alone, since the attachments notice names every live
row: a picture requires `vision`, a PDF prefers it. A greenfield root with no brief refuses to
dispatch once nothing at all is attached (`assertSomethingToBuildFrom`), which is 00's own "a brief
or a file" rule re-checked against the live rows, before the inputs are prepared and again after,
since preparing a late document can take minutes and deleting the only one meanwhile must still
refuse. 02's manual repair drops a picked section whose document is gone, by the attachment ROW
the section was read from (`sourceId`): the gate carries the section's BODY, and by name a
same-named replacement recorded since would answer for it. A section's key is its row and line
rather than its name, so a repair of the deleted document does not mark the replacement's section
handled; a repair recorded before sections carried their row cannot say which one it covered and
keeps its name-only meaning. Every lookup fails open, onto the fields the build had before. A failed index REWRITE is
the exception: it drops the index from the prompt, and is recorded that way, since the file on disk
no longer lists what is attached.
**plan_chat** is one conversation on one card: a self-targeting `reviseLoop` re-parks the form every turn and the user ends it by submitting nothing. The transcript lives in `plan_node_messages` precisely because that revise resets the step row each cycle. The agent is handed the WHOLE plan (via `renderPlanMarkdown`, the same render committed as `.haive-data/plan.md` — one function so what the agent reads and what is committed cannot drift), so a request made while looking at one node can correctly patch another. **advisory** researches a non-code blocker and then STOPS: `02-advisory-decision` parks on a form and only the USER closes it — an agent concluding an unsigned contract is fine would turn a real blocker into a green tick.
**A sibling run too wide for one agent's reply is named, never sent.** The build-order step
(`03-plan-sequence`) asks one agent per undecided run for an upsert per child plus the links it
adds, and a patch holds `PLAN_PATCH_MAX_OPS` (500) ops, so a run past `SEQUENCE_MAX_RUN_CHILDREN`
(half of that) keeps its stored order and the finished step names it (`degradedNote`), and the
prompt states the op budget its reply has. `tooWideToSequence` is the one rule the fan-out and the
plan page's Order count (`computeSequenceProgress`) share, so the count never promises a pass that
would skip a group. At the cap the child list stays under `PLAN_EXPANSION_CONTEXT_MAX_CHARS`, since
`safeTitle` caps a title at 200 characters. MEASURED: no stored run came near it (widest 33).
**"Start next" is STRICTER than the gate that refuses a task, deliberately.** `computePlanReady`
(`shared/plan/ready.ts`) picks the lowest-numbered node that is startable now: `todo`, no unmet
`depends_on` of its own, **no ANCESTOR with one**, no open task on it, and a unit of work
(`taskable` for component/decision; a leaf for research/external, which carry `taskable` only
sometimes — MEASURED 10 of 16 and 4 of 12). The ancestor half is the point, and it is the half
`blockedById` deliberately omits: MEASURED, 1,286 of 7,079 taskable nodes in one plan carry no
prerequisite of their own and 23 also have a clean ancestor chain, and on another plan the
lowest-numbered node the DIRECT rule calls ready (#28) sits under two containers that are
themselves waiting, while the strict rule picks #65. Do NOT reconcile the two readings: a rule
strict enough to CHOOSE must not REFUSE, because choosing badly wastes a click while refusing
badly turns one wrong-direction edge (15 of 16 unsatisfiable deps were exactly that) into a
locked subtree. `POST /tasks` stays direct-only; the node panel's `ancestorBlockers` banner is
what explains the skip, and says out loud that it does not stop you. Cycles need no test of
their own — their members never lose their blockers, so rules 2 and 3 already exclude them and
everything beneath them. `nextUp` rides the plan overview (a property of the PLAN, like
`ordering` and `defects`); the SET behind the count is `GET /plan/ready`, served in the exact
shape `/plan/search` returns so the page filters it with the machinery it already has, capped
at 200 and REPORTING the cap.
**Impact answers "if I change this, what else must change?"** (`shared/plan/impact.ts`): an explicit BFS with a visited set, because the edge graph has cycles by construction and a recursive CTE without dedup would not terminate while one with dedup could not say where it stopped. Both caps are REPORTED, never applied silently. The mermaid source encodes nodes as a `pnode<32 hex>` token; the browser recovers the uuid from THAT, unanchored — mermaid prefixes rendered ids with its own render id, so a `^flowchart-` anchor binds zero handlers and fails silently.
The walk takes a SET of origins, because gate 1 asks the question of every component a spec named rather than of one of them. Two consequences. `depth` is then the distance to the NEAREST origin, which is the honest number — the per-origin walks it replaced recorded whichever origin reached a node first in list order, and MEASURED on a real task that reported 1 node at one hop where the multi-origin walk finds 72. And every origin is seeded as visited, so the walk never discovers an edge BETWEEN two origins: `renderImpactMermaid`'s `edges` option exists to draw those, and without it a multi-origin picture is a row of disconnected boxes. The diagram's `maxNodes` bounds HOPS only — origins are always drawn, so a caller with more origins than a picture can hold must decide not to draw one. **Code links** have one writer (the applier; the builder only links files it actually opened, with `evidence`) and rot is flagged, not guessed away: `11c-rag-reindex` marks links stale for the paths in `tasks.changedPaths`, and only re-assertion by an agent clears the flag — the difference between an impact view that is wrong and one that is merely old.
**Spec-writer integration:** when the repo has a plan, 04-phase-0b is handed a compact component index and must emit an `## Affected components` section naming `node:<uuid>` ids. `resolveAffectedComponents` (`workflow/_affected-components.ts`) parses those IDS — never the agent's prose, because name-matching picks the wrong node the first time two read alike — and walks the edge graph from all of them. It lives in its own module because two steps must agree on the answer: 04 stores it for the implementer's prompt (`_plan-impact.ts`, which states the invariant — agent and human reading different blast radii is worse than either reading none) and 06 renders it to the approver.
**Two things bound what the plan can do to that prompt, and both were measured on a 544-node plan.**
`recordTouchedPlanNodes` writes a `touched` link for every component the spec named, and `loadSeededPlanNodes` reads `implements` ONLY. Reading both made the step feed its own output back in: MEASURED on task 681f0f99, 5 links at task creation, +157 the moment 04 round 0 ended and +201 the moment round 1 ended, so 363 nodes were re-rendered IN FULL WITH BODIES — the seeded block went 9,493 to 232,365 chars and the whole spec prompt 219,412 to 441,506, against 43,641 for the same task with no plan. It compounds every round. That is the same role split `completePlanNodesForTask` relies on to avoid greening a node the task merely touched, so a reader that ignores the role is the bug, not the column.
`renderBoundedPlanIndex` then steps the index's DEPTH down until it fits `PLAN_INDEX_MAX_CHARS`, stating the reduction in the prompt. Depth, not a character slice: a slice ends mid-node and leaves a truncated `node:<uuid>` the writer can quote back as whole. The bound is a GUARD RAIL — 120k sits above every index measured on a run that went well, so nothing observed working is trimmed. Worth knowing before tuning it: the index is the DOMINANT term in every planned run, MEASURED at 93,035 of 124,215 chars (75%) for a 174-node plan, 115,620 of 133,331 (87%) for 193 nodes and 158,274 of 219,412 (72%) for 544. Whether a shallower index costs spec quality has NOT been measured; ~40k halves the spec prompt and is the experiment to run.
**The gate renders it as the plan canvas's own Impact view, not as prose.** The section carries `planImpact` on the form schema (`shared/schemas/form.ts`) and `PlanImpactSection` (`web/components/plan/`) draws depth groups → relation sub-groups → click-through rows, reusing the `PlanImpactList` the detail panel's Impact tab uses. It replaced one markdown list: MEASURED on a real task, 363 flat bullets in a `max-h-96` scroller holding 10,459px, with the distance an impact answer is ABOUT reduced to a `(2 hops)` suffix, nothing clickable, and a diagram seeded from `named[0]` ALONE that drew 19 of the 363 while reporting `mermaidOmitted: 0`. `body` is empty on such a section — one rendering, not two — and a section persisted before `planImpact` existed has neither and still renders as markdown, which matters because `task_steps.form_schema` is STORED and only rebuilt when it is null. 06's detect re-walks a payload that predates `mermaidDepth` rather than defaulting `reversed`, since a fabricated direction inverts every "Depends on" / "Depended on by" heading.
**The radius is a control, not a constant.** The gate carries the same reach buttons the Impact tab has, but over a FROZEN snapshot — so the resolver renders one bounded diagram per radius the walk reached and the buttons switch between them. The browser cannot draw its own: a hop in the payload carries no `viaNodeId`, and re-deriving a traversal there would put the cycle guard and the caps in two places. Only radii the walk actually reached get a button, because the panel's fixed 1-4 can afford one that returns nothing (it re-fetches) and a snapshot cannot.
`IMPACT_DIAGRAM_MAX_NODES` is the whole picture's budget, spent on ORIGINS first and on neighbours with what is left, so a wide named set draws origins alone. That is a real subgraph, not a row of loose boxes: MEASURED on the 161-named task, the first 40 named components carry 40 edges between them and only 4 of the 40 are isolated (217 edges among all 161) — which is why an earlier version that refused to draw past 40 origins was wrong. Origins are taken in SPEC order because which 40 got drawn has to be answerable, and "the ones the spec cites first" is checkable where a connectivity ranking is not. `namedOmitted` states the rest. A consequence worth knowing: nearest-first ordering means a wider radius only appends nodes that the budget then cuts, so once the picture is full it stops changing with the radius while the list keeps growing — the omission line says so. No count is ever subtracted from another: the diagram and the list walk different radii from different origin sets, so no arithmetic between them would be true.
The mirror (`worker/src/plan/mirror.ts`) writes `.haive-data/plan.json` + `plan.md` from every plan-step apply and from `12-post-onboarding`; `persistDetection` imports it on clone. `CONFIG_KEYS.PLAN_CANVAS_ENABLED` (admin toggle) off refuses new plan tasks and makes the onboarding step self-skip; existing plans stay readable and editable — hiding a plan someone made reads as data loss. Web: drill-down grid, not a graph engine (`components/plan/`); `plan-status.ts` is the single status→colour source; counts are server-computed. Live-Postgres coverage: `pnpm --filter @haive/worker smoke:plan-canvas` (107 checks).
## Review scope
Every reviewing step (07a, 07b, 08a, 08b, 08c, 08c2, 08d) is scoped by ONE collector,
`collectImplementationFiles` (`_impl-changes.ts`). It unions 07's agent-reported
`filesTouched`, the DAG issues' `filesModified`, and the dirty worktree, caps the list at 100
and REPORTS the cap (`changedFilesBlock`'s COVERAGE notice orders the agent to state what it
was not given) — a silent cap once had a reviewer approve 100 of 150 files as though it had
seen all of them. The list is not a convenience: `worktreeGitfileMask` bind-mounts an empty
file over the worktree's `.git` for every cli-exec invocation, so inside the sandbox there is
no `git status` and no `git diff`, and everything an agent knows about the change has to
arrive in its prompt.
Each path carries the LINES this change wrote (`lines 12-18, 45`, `new file`, `deleted`,
`no line changes (mode or rename only)`). Measured against the MERGE-BASE with the task's base
branch, not HEAD, because the two execution paths commit differently and only the fork point
covers both: single-agent work is still uncommitted at review time (the first commit is
`10-gate-3-commit`), while `dag-executor` commits every issue and merges it in, so a DAG task's
tree is CLEAN by 07b and `git diff HEAD` reports nothing. `git diff <ref>` compares the WORKING
TREE to that ref, so one call covers committed and uncommitted work together. Line numbers come
from the hunk headers' `+` side, which is the file as the agent will read it.
ABSENT is not "unchanged". A path git never saw (an agent-reported file, a diff that could not
be read) carries no note, and both the prompt legend and the scope fence say so explicitly:
the whole file is in scope. Narrowing on a measurement nobody made is the one direction this
must never fail in.
`SCOPE_BOUNDARY` (`_scope-fence.ts`, shared with the on-disk agent templates so the inline
persona and the file that OVERRIDES it cannot drift) is therefore keyed on lines, not files:
in scope = the lines the change wrote, the function or block each sits in — an edit's effect is
not confined to the edited line — plus blast radius (callers of a changed signature, consumers
of a changed schema, paths the change makes newly reachable). It used to be "the files this
change touched", which made a pre-existing defect anywhere in a 5,000-line file a blocking
finding against a three-line edit; MEASURED across 102 runs of one task, ~450 blocking findings
sat on legacy code the task never touched, each costing a capped fix round whose fixer then
REWROTE that legacy code (one worktree: 71 dirty files against a plan of 23). Out-of-scope
findings are never dropped, only re-dispositioned — `## INSIGHTS` for the peer/lens reviewers,
`in_scope: "no"` for security, the markdown report for 07b.
An EMPTY change set fails the step (`assertReviewableChange`), at the prompt-build boundary
rather than in detect() so a replayed `detect_output` is guarded too, and always before
dispatch. It used to render a fallback telling the agent to work the change out from the
workspace — which it cannot, so it guessed, and the verdict it returned covered nothing. A skip
would be worse than a failure: at gate 2 a review with no findings is indistinguishable from an
approval. The two ways to get an empty set are reported as different diagnoses, since
`dirtyWorktreeFiles` no longer swallows its own error — a failed scan names git's message, a
scan that ran and found nothing says the implementation wrote no files.
**Review is where a workflow run's time goes, and in DAG mode HALF of it is spent before the
merge.** MEASURED across five runs of one task, per-issue DAG reviewers against total
`06c-dag-execute` CLI time: 184 of 369 min (Opus 5 max), 59 of 109 (Opus 5 medium), 39 of 72
(Opus 5 medium, no plan), 131 of 349 (gpt-6-astra) — consistently ~50%, and the merged result is
then reviewed AGAIN by 07b, 08c, 08c2 and 08d. On the max-effort run that is ~487 of 1,096 CLI
minutes, 44% of the whole run. That per-issue layer is the `sprintReviewEnabled` checkbox on
06-run-config ("AI-review each issue before merge"), default ON.
**Do NOT reach for that checkbox as a cost lever — it is two features under one flag.**
`resolveReviewPhase` (the ~50%) reviews issues that SUCCEEDED, as the merge gate
`runLevelMerge` keys on. `resolveEscalationPhase` (issue-advisor -> replanner) handles issues
that FAILED, costs nothing when none did, and sits inside the same `if (plan.reviewEnabled)`.
So turning review off ALSO turns off failure recovery, and the `coderFailed.length > 0 &&
!plan.reviewEnabled` branch then fails the whole level on one failed coder. Both halves landed
the same day — `e37a39c7` (per-issue review gate) and `79e321ad` (escalation hierarchy) — which
is how they came to share a flag. Neither `dag-review-smoke` nor `dag-escalation-smoke` runs
with it false, so that combination is UNTESTED as well as unsafe.
Splitting them is the fix: escalation always available, per-issue review the toggle. Only then
is the default an empirical question — and what has NOT been measured either way is whether
07b/08c/08c2/08d catch what the per-issue reviewer catches. Retrofitting review onto a failure
is NOT the alternative: it gates SUCCEEDED issues before merge, and by the time a defect
surfaces at 07b, `cleanupLevelWorktrees` has destroyed the per-issue worktrees. The retroactive
path already exists and is the fix loop, in the integration worktree.
Two neighbouring numbers to read before concluding anything from a DAG run: `issueCount` and
`levelCount` (`06b-sprint-planning.output`). Levels are BARRIERS, so parallelism tracks the
level shape and not the issue count — MEASURED, 4 issues in 1 level ran 32 min wall against 72
CLI (2.25x), while 5 issues in 3 levels ran 111 against 109 (0.98x, i.e. the DAG bought no
concurrency at all and still paid per-issue worktrees and merges), and 13 issues in 5 levels ran
215 against 369 (1.72x). A single-issue level is a pure barrier; the leading one is the
deliberate convention-setter, the trailing ones are integrators. No rule here is safe to derive
from those five points — they vary by model and by effort — but a run whose wall time ~ its CLI
time is a DAG that serialised, and that is worth looking at before blaming the model.
**Similar code is reported, not changed.** Every implementing pass is told to leave the same
code or defect it finds outside the task alone and to list it instead: 07 as `similarSites`, the
DAG level coder and the DAG fix coder as `similar_sites`. `ingestReviewRun` reads the fix coder's
JSON for this field and its `concerns`, which reach the ledger. Gate 2 shows the union of every 07
round and every DAG issue as a status row after its checks, and the person acts on an entry by rejecting
with feedback that names it, which reaches 07 as a human directive. A site whose file a round after
its last report edited is MARKED, not dropped (`editedInRound`): that round may have edited the file
for something else, and dropping the site would then hide one still left unchanged. The mark reads
07's `filesTouched`, the agent's own and possibly incomplete account, so it is a hint and never
proof, and the row says the agent LEFT each site unchanged when it reported it rather than
asserting the site is unchanged now. A manual retry of 07 replaces that round's pass, its reports
included, exactly as it replaces the pass's summary and `filesTouched`: a retry means "redo this
step", the edits the pass left stay visible to every reviewer through the dirty-worktree scan, and
the rerun reads the same code. `quick_bugfix` runs no gate 2, so gate 3 shows the row whenever no
gate-2 decision exists.
The lists are display copy and are kept that way. They are sanitised on the way in and again on
read (`_similar-sites.ts`): a path must be single-line, fence-safe and inside the repository, and
a reason is collapsed to 200 chars. They never enter the task ledger, for the reason
`REPO_IS_DATA_ACTING_LINES` gives: an editing agent's prose relayed into later prompts is how
hostile repository text travels. `task_dag_issues.similar_sites` (migration 0165) is MERGED per
pass rather than written, because an advisor retry re-runs the coder into the same row. The DAG
schema reads a malformed list as `[]` (`.catch`), since a strict field there would turn a finished
coder into `failed_unrecoverable` over a list only a person reads.
**Out-of-scope findings reach gate 2 too.** 08e offers the `## INSIGHTS` lines agents wrote for
this task to act on, but auto-continue pre-answers 08e with an empty pick (`06-run-config`),
`plan_tasklist` runs gate 2 without 08e, and `quick_bugfix` runs neither. So gate 2 lists every
insight nobody picked at 08e in its own row after the similar-sites row (`_gate-insights.ts`), and
gate 3 shows it when no gate-2 decision exists. 08e's picks are subtracted by title and location
across every round, since its `i-N` ids are positions. The row is display copy under the same rules
as similar sites: each field collapsed to one line of at most 200 chars and escaped, the list cut
at 30 with the rest counted, and nothing of it entering a prompt.
## Review findings and waivers
`review_findings` is the durable record of what every reviewing step raised — 07b's validator, 08c/08c2, 08d's adversaries. Findings otherwise live only in `task_steps.output`, which a manual retry nulls, so nothing could say whether a reviewer change helped. `recordReviewFindings` (`_review-findings.ts`) is the only insert, and its two readers only inform: `loadFindingRecurrence` counts earlier rounds for `recurrence_count`, and `GET /stats/quality` aggregates it. No behaviour gates on it, which is why every write there is best-effort and never throws — telemetry must not fail the review that produced it.
`disposition` is the verdict on a finding. Every value that is written has exactly one writer:
- `open` — default. Nobody has ruled on it.
- `dismissed_refuted` — 08c's refuter panel disproved it. Written at INSERT, because the insert happens on the final apply() after refutation ran.
- `dismissed_human` — the developer picked a subset at gate 1.5 (`08d2`) and left this one out. Written by `dispositionReviewFindings`.
- `accepted_risk` — the fix loop hit its round cap (or oscillated), and the developer accepted the remainder at the escalation gate. Written by `acceptRemainingReviewFindings` from the queue's gate resolver, which is why that one takes a `db` rather than a `StepContext`.
- `fixed` / `recurred` — declared, SUPERSEDED, and must stay unwritten. Recurrence lives on `recurrence_count` instead (below); `fixed` should never be written at all, because absence of a finding is not evidence it was fixed — a reviewer that was skipped, budget-killed, or simply reworded it produces an identical absence. Do not read `open` as "still outstanding".
Both UPDATE paths are scoped to the CURRENT round and guarded on `disposition = 'open'`. The round scope matters because the same fingerprint recurs on its own row every round, so an unscoped update rewrites the history of rounds the developer never saw — and because with no `fixed` writer, an earlier round's rows are `open` whether they were fixed or recurred. The `open` guard keeps the more specific verdict: a refuter's disproof and a gate-1.5 waiver both outrank a blanket acceptance. Findings raised AFTER the acceptance stay `open` and are NOT swept into it — the developer accepted the remainder they had been shown, not findings nobody had seen yet. That is the normal case, not an edge one: acceptance stands down the FIX loop only, while `restartLoop` (a gate-2 reject) is suppression-immune, so a later round runs and records more. MEASURED on the dev install, every accepted task has findings past its accepted round (accepted 7 / findings to 9, 4 to 5, 5 to 7, 6 to 9). Those surface at gate 2, where the call is the developer's again.
A waiver is CONTEXT, never SUPPRESSION. Nothing reads `dismissed_human` back, and 08d2's detect deliberately does not fold on it. A verdict is about a MOMENT, not about the defect — the reason 08d gives where it carries its disproved set forward, and a human declining to spend a round on something is far weaker evidence than a PoC a verifier actually ran and could not reproduce. What a partial fix round leaves behind instead is a task-ledger entry (`formatWaiverLedgerEntry`) whose wording invites a re-raise and asks what changed; `augmentPromptWithLedger` already carries it into the next round's prompts, so it needs no plumbing of its own. Accept-as-is at gate 1.5 records NOTHING — those findings still surface advisorily at gate 2, so the call is deferred rather than made.
`recurrence_count` records how many EARLIER rounds of the task already saw that reviewer flag that FILE, stamped at insert by `recordReviewFindings` from `loadFindingRecurrence`. A counter beside the verdict rather than a disposition value, for two reasons: recurrence is orthogonal to a ruling (a finding can be both refuted and recurring, and `disposition` holds one value), and both disposition UPDATE paths are guarded on `disposition = 'open'`, so a `recurred` written there would hide exactly the most-repeated findings from a gate-1.5 waiver and from the escalation gate's acceptance. Surfaced two ways: `recurrenceTag` prefixes gate 2's finding lines with `[repeat xN]` plus a one-line legend, and `buildRecurringNote` (08c) hands the next fix round an "Already tried" block, since each round is a fresh `claude -p` and `loadPriorFixContext` dedupes prior diagnoses by prose.
The key is `(reviewerId, normalised path)` — deliberately NOT `fingerprint`. MEASURED across 9,750 findings in 68 multi-round tasks: the fingerprint matches across rounds 5 times (0.05%) because it hashes the issue TEXT and two rounds' descriptions of one defect are different sentences, while (reviewer, path) matches 1,408 of 3,294 (42.7%) and reading the rows confirms those are one complaint re-raised — one across 19 rounds, every row still `open`. The cost of the coarser key is that two DISTINCT defects in one file count as a repeat, which is why every surface says "this reviewer already flagged this file" and never "the same defect". Nothing branches on the count; it informs a human and a prompt.
`fingerprint` is `sha256(reviewerId, path, normalised issue)` — line numbers and ids stripped, path KEPT, since the same defect in two files is two findings. Rows are written PRE-merge, one per adversary that reported a finding, while every later step sees only the survivor of 08d's location merge and has no reviewer id to rebuild a key from. That is why 08d attaches the collapsed set to each surviving finding as `fingerprints[]` (same shape and reason as 08c's `RefutableFinding.fingerprints`), on the OUTPUT copy only so the merge's own objects and `raw` stay untouched.
## Review dimensions
`REVIEW_DIMENSIONS` (`packages/shared/src/review/dimensions.ts`) is the canonical set of 14 a
change is scored against. Before it, the same names were hardcoded prose in five files across
eleven sites, worded three ways (`Privacy/Compliance` / `Privacy / Compliance` / `Privacy`;
`Internationalization` / `i18n`). The per-dimension `criteria` are lifted VERBATIM from 07b's
Step 7 — the only copy that carried criteria rather than names — and stored as ALREADY-WRAPPED
lines so `numberedDimensionBlock` only re-numbers and re-indents. That is what makes the
full-set render byte-identical to the old literal, including the 3-space vs 4-space
continuation indent for items 1-9 vs 10-14; `dimensions.test.ts` asserts the whole 32-line
table against the original, because every install that has not touched the setting still
renders it and a diff there is a prompt regression for everyone.
NOT the same list as `QUALITY_DIMENSIONS` in `05-phase-0b5-spec-quality` (`goal_clarity` …
`documentation_updates`), which scores the SPEC. Both have 14 entries and share nothing else.
That constant is deliberately untouched by this feature.
**Two levels, and the split is about WHEN each step runs.** `repositories.review_dimensions`
is the policy; `tasks.review_dimensions` is a per-task override; resolution is
task ?? repo ?? all (`resolveTaskReviewDimensions`, `step-engine/review-dimension-context.ts`,
mirroring `guidance-context.ts`). Both columns are `text[]` NULL-means-all, following
`lspServers` rather than the `jsonb().notNull().default([])` shape, because `[]` (score
nothing) and "never chosen" must stay distinguishable. The override reaches only the REVIEW
steps: discovery is index 3 and the spec writer index 4, while the run-config form is 6.05, so
those two pass `scope: 'repo'` — reading the task value there would apply a narrowing the user
makes later to a spec that was already written. Failure is OPEN (every dimension) at every
reader, because failing closed would silently produce a review that scored nothing.
**The on-disk agent definition outranks the inline persona.** 04, 05 and 08c each wrap their
prompt in `agentDefinitionGuidance`, which says "if a `.claude/agents/<id>.md` exists in the
repo, follow it; otherwise follow the protocol below" — and `technical-spec-writer`,
`spec-quality-reviewer` and `peer-reviewer` all name the full 14 on disk. Editing only the
inline TypeScript persona narrows NOTHING for an onboarded repo. Making the templates
dimension-aware is not the alternative either: their bytes are hashed against `REFERENCE_CONTEXT`
(`template-manifest.ts`) and compared at `api/routes/upgrades.ts`, so a per-repo body reads as
drifted and is reverted on the next onboarding upgrade. So `dimensionScopeOverride` is APPENDED
AFTER the persona in those three, where it is the most recent instruction, and returns `''`
when nothing is excluded — a default run emits the prompt it always did. 07b needs no such
block and is the one site filtered at the source, because it carries no agent pointer.
**A skipped dimension is disclosed, never implied.** 07b records `excludedDimensions` on every
apply return (the parse-miss one included) and gate 2 renders `## Not reviewed`, because a
dimension nobody scored yields exactly the same empty finding list as one that passed. Same
rule as `fixed` never being written to `review_findings`. Both that field and gate-2's copy of
it are OPTIONAL: step outputs and gate detect payloads are PERSISTED, so a task already parked
at gate 2 replays an object written before the field existed and must still render.
**03 filters mining personas through an explicit pair list** (`DIMENSION_ONLY_AGENTS`:
`accessibility-specialist`, `security-auditor`), never by matching the persona's `field`
frontmatter — even though `field: accessibility` and `field: security` happen to equal their
dimension ids. `field` is a grouping label with its own vocabulary (`testing`, `review`,
`quality`, `api`) that only coincidentally overlaps: `testing` and `testability` already do
not match, so keying on it would filter some dimensions and silently miss others. An agent
belongs on that list only when dropping the dimension makes the agent pointless. The level-gated
08c lenses (operational/performance/simplicity) are NOT filtered — they describe their own
remit in their own words rather than naming the 14, and the QA level is a separate explicit
choice. 08d's six adversaries are attack roles, not dimensions.
Admin surfaces: a card on the repo tooling page (`/repos/<id>/tooling`) for the policy, and a
collapsed accordion in `06-run-config` for the per-task override — the one accordion allowed on
that form, whose controls are otherwise top-level by design (see `fddd12a5`). Neither can store
an empty set: the repository PATCH answers 400, and the task form resolves both an omitted
field and a submitted-but-empty one to the policy. That multi-select is deliberately NOT
`required` — in `validateFormValues` the flag rejects an ABSENT value, which is the legitimate
"leave the policy alone" case every programmatic submitter uses (it broke the workflow smoke),
while an explicitly empty array counts as present and skips the check, so the flag failed the
good input and guarded nothing. No global `CONFIG_KEYS` kill-switch, deliberately — the default
is already "all on", so a switch would gate nothing.
## Model pricing and spend
`cli_invocations.cost` records what a run COST, decided at completion beside `token_usage` and `model_identity` and then immutable. Distinct from `token_usage.costUsd`, which keeps its old meaning (what the CLI itself reported). The split exists because the reported number is real only where the CLI prices its own backend: the claude binary applies ANTHROPIC's table to every backend, so for zai/muse/openrouter/ollama it is fiction, and codex/gemini report no cost at all. MEASURED before the feature: of ~12,800 USD reported across 12,438 invocations, 31.68 USD was the only amount the product could count as real.
Precedence per invocation (`resolveCostDecision`): manual rate > CLI-reported where `costBasis` is `metered` > computed from a synced rate. Subscription auth short-circuits to non-billable regardless — a flat plan's per-token dollars are notional. `resolveCostBasis` demotes BOTH per-token bases under subscription auth, `metered` AND `estimate`, so a claude-binary wrapper on a flat plan (a GLM coding plan) is non-billable too; its counterfactual is then computed from Haive's own feed rate, never from the binary's Anthropic-table total, and an unpriced model there is `none` rather than fiction. `billable` is decided at WRITE time, where provider, auth mode, model and price source are all in hand, so reads honor a flag instead of re-deriving the rule (it had been duplicated across three SQL filters). A partial computation is recorded `source: 'none'`, never as a cost: a total missing one bucket looks like money and is not.
`ollama` is the one provider whose basis is per-MODEL rather than per-CLI, so `resolveCostBasis` takes an optional model key. One catalog entry covers the local daemon and Ollama Cloud alike, and the catalog says `local` because that is what `ollama` used to mean; a `-cloud`/`:cloud` model runs on ollama.com under a plan that is plan-included up to a limit and metered beyond it, which is neither free nor per-token-billed, so it resolves `subscription` (non-billable, priced only as the counterfactual). Keyed on the model TAG via `isOllamaCloudModel`, the same test the base-URL routing and the boot provisioner already use — NOT on the base URL, which a cloud provider row does not set at all because the local daemon proxies cloud models. Both tag forms matter: the common one is `<size>-cloud`, and every zero-rate cloud row the LiteLLM feed carried wore it, so a `:cloud`-only check would have missed all of them. A caller with no model in hand (the per-provider usage rollup groups by provider, not by model) keeps the catalog answer.
The non-billable half is not discarded, it is the SUBSCRIPTION COUNTERFACTUAL: `notionalCostUsdSql` (api `routes/tasks/_helpers.ts`) sums what those same tokens would have cost at list API rates, rendered grey and labelled on the task's provider card, never added to real spend. Two halves, each the exact complement of the real-spend rule: snapshot rows flagged non-billable minus `source: 'none'` (whose 0 means unpriced, not free), plus legacy `cost IS NULL` rows from a METERED provider on subscription auth. The claude-binary wrappers are excluded from the legacy half because their reported total is Anthropic-table fiction in either direction (5,968 USD of it against local ollama tokens), and codex/gemini report no cost at all, so both contribute 0 rather than a guess. MEASURED on the dev install: 29.63 USD real against 3,236 USD notional.
Rates live in `cli_model_prices`, effective-dated and append-only. A sync CLOSES the live row and inserts a replacement ONLY when a rate actually moved, so an unchanged 12-hourly tick writes nothing (verified: `inserted 0, closed 0, unchanged 651`). Three feeds, refreshed by the existing `REFRESH_VERSIONS` job (`cli-versions/model-prices.ts`), sequentially after `refreshOpenRouterModels` because it reads that cache rather than re-fetching 4 MB:
- **LiteLLM** `model_prices_and_context_window.json` for direct-vendor rates. The feed publishes one entry per (host, model), so `PROVIDER_LITELLM_VENDORS` maps each Haive provider to the vendors that ARE its backend and everything else is rejected — `azure_ai/claude-opus-5` and `vertex_ai/claude-opus-5` are different rows at different prices, and `azure/us/gpt-5.6-sol` really is 10% over list. MEASURED 2026-08-18: anthropic 26 chat models, openai 90, gemini 50 + vertex_ai-language-models 29, xai 44, zai 13, meta 3. `ollama` is deliberately NOT mapped here: all 21 of its rows are priced 0, which is the truth for a local model and a LIE for the four CLOUD models the feed also carries under that vendor (`deepseek-v3.1:671b-cloud`, `gpt-oss:20b-cloud`, `gpt-oss:120b-cloud`, `qwen3-coder:480b-cloud`). A stored 0 is not "unpriced", it is a PRICE — it makes an invocation record a real $0.00 — so those rows were retired by migration 0126 and the vendor mapping dropped.
- **OpenRouter** `/api/v1/models` for the gateway's OWN resale rates, provider-scoped to `openrouter` and never priced from a vendor rate — it bills its margin at the routed model's price, wrong by a different factor per model.
- **Ollama** the model pages themselves (`cli-versions/ollama-model-prices.ts` fetching, `cli-providers/ollama-pricing.ts` parsing), because Ollama publishes no price document and no pricing API — `ollama.com/pricing` lists plans and difficulty grades, not rates. Bounded to CONFIGURED CLOUD models, one page each per tick, never the whole library; cloud-only is a GUARD, since a stored rate on a `local`-basis run would be summed as real spend. MEASURED 2026-08-23 across the 10 configured cloud models: 1 (`kimi-k3`, $3.00 input / $0.30 cached / $15.00 output per 1M) publishes a cost block and 9 do not. Absence is a RESULT, not a failure — it is the common path and must never log an error or every tick cries wolf. A block that EXISTS and cannot be read fails loud and writes nothing. Amounts are paired with the label that follows them, never by position (an automated read of that page during diagnosis returned input $0.30 / cached $3.00, the two swapped), and `cached <= input` is asserted because a violation means the pairing is wrong. Selectors live in ONE constant marked volatile — this is HTML, it will reword.
Lookup is EXACT on a normalized model key (trim + lowercase, nothing else), deliberately NOT the longest-substring matcher `resolveContextWindow` uses: a wrong context window skews a cosmetic percentage, a wrong price is wrong money, and `claude` alone matches both opus and haiku (~15x apart). Variant markers are kept verbatim (`glm-5.3[1m]`, `deepseek-v4-pro:cloud`) because a context or hosting variant is usually its own SKU. An unmatched id is UNPRICED, never guessed — which is why the manual-override path is load-bearing rather than a nicety (`glm-5.3` is in no feed).
One arithmetic trap: Anthropic-shaped usage reports `input_tokens` EXCLUSIVE of the cache buckets, while codex and gemini report it INCLUSIVE of the cached prefix (`INPUT_INCLUDES_CACHE_PROVIDERS`). Pricing those two without subtracting charges cached tokens at the full input rate. Separately, cache-WRITE rates are frequently absent from the feeds, which costs nothing in practice: MEASURED, only claude-code (which uses its reported cost anyway) and amp (subscription) ever report cache-creation tokens — 0 across 6,204 invocations for every other provider.
Admin control is two-level and BOTH are enforced at lookup, not only at sync, so a change takes effect on the next invocation rather than at the next refresh: `CONFIG_KEYS.PRICING_AUTO_UPDATE_ENABLED` is the global kill-switch, and `cli_pricing_sync.auto_update_enabled` per CLI means "this provider's rates are admin-owned" — its feed rows are ignored and only manual rates apply. Manual rates are effective-dated the same way (a write closes the previous one), and retiring one closes it rather than deleting, since the rows that priced past invocations must stay readable. Admin UI: a card on `/admin` for the global switch plus display currency, and `/admin/pricing` for per-CLI toggles, the rate table and overrides.
Costs are stored canonically in USD. `fx_rates` holds daily ECB USD-per-unit rates and a task converts at the rate effective on ITS OWN date (`resolveCostDisplay`), so re-rendering a finished task yields the same figure; dated on the task rather than per invocation because FX drift within one task is far below the displayed precision. ECB publishes only the current day, so past rows cannot be re-fetched — do not truncate that table. A task older than FX collection converts at the earliest rate on record and is flagged `approximate`.
## Statistics
`/stats` (`packages/web/src/app/(app)/stats/page.tsx`) and the smaller `/dashboard` both read
`packages/api/src/routes/stats/`, which is seven endpoints sharing one query parser
(`_query.ts`: 30-day default, `MAX_RANGE_DAYS` 731, IANA zone validated against `Intl`, repo /
task-class / provider facets, and an `allUsers` flag each ROUTE re-checks against the caller's
role because the parser has no access to it). Every arithmetic answer is a pure function in
`@haive/shared/stats` — the api fetches and scopes, `computeBusySpan` / `buildTaskTimeBreakdown`
/ `buildEstimationAccuracy` do the maths, and none of them needs a database to test.
**ONE BUCKETER.** Day buckets are cut in JS with `Intl` (`dayKey`), never with `date_trunc` in
SQL, because the busy-span union already needs a bucketer in JS and two that must agree
eventually will not. The correct SQL form is recorded at `index.ts:476-489` for the day that
changes, along with the trap: the columns are `timestamp without time zone` holding UTC wall
clock, so a single `AT TIME ZONE $tz` reads them as already-local and MEASURED moves 5-10% of
rows into the wrong day. The consequence for new features is that anything day-shaped rides the
existing `/stats/timeline` payload rather than adding a day-aggregate endpoint.
**`/summary` + `/timeline` are fetched for every tab; the other five endpoints are lazy**, so an
unopened tab costs nothing. A new section belongs on a tab that already pays for its data —
`/stats/tasks` exists as its own endpoint for exactly this reason rather than as extra fields on
`/timeline`. Both pages resolve `timeZone` and the relative-preset clock on the CLIENT only and
leave them null until then, because `Intl` reports the container's zone during SSR and
`Date.now()` differs by milliseconds; both nulls GATE the first fetch. That mismatch, not a chart
bug, is what used to blank the charts.
**Under-sampling is disclosed, not smoothed.** `sampledRatio` carries `n` and `formatSampledRatio`
prints `n=3` INSTEAD of a percentage below `MIN_SAMPLES_FOR_TREND` (5); a figure that cannot be
computed renders as an em dash and never as zero. `computeDelta` returns a null `changeRatio`
against a zero baseline, which the UI shows as "new" — growth from nothing is neither infinite
percent nor 100%.
### The heatmap and the token bar
`ActivityHeatmap` and `StackedShareBar` (`components/stats/`) are plain DOM and STATICALLY
imported; only the three recharts charts are behind `next/dynamic`, and only because recharts is
a ~7 MB package with a d3/redux tree. Both render fields `/stats/timeline` and `/stats/summary`
had always computed and nothing had ever displayed — the per-day token series, `invocations`,
`tasksStarted`/`tasksCompleted`, and all four `tokens.*` buckets.
**Shading is by quantile of the WORKED days, not by a fraction of the maximum.** Nearest-rank
quartiles over the non-zero days, so every cut point is a value that actually occurred. MEASURED
on the real day series (agent-hours 44.76, 20.78, 16.36, 15.87, 13.19, 12.61, 8.35), dividing by
the maximum puts five of seven days under 0.37, and a fixed 0.3/0.6 threshold on that then
collapses them into one or two shades — a skewed week is the normal case here, since one long day
sets the maximum. Zero days are excluded from the quantiles and get their own colour and their own
legend swatch: "nothing ran" and "the quietest day that had work" are different claims.
`heat-scale.ts` also owns `localDayRange`, the INVERSE of the shared bucketer (a local day back to
an instant range) for the per-day drill-through — duplicated rather than imported because web must
not pull the `@haive/shared` barrel into the bundle, and tested against both DST days
(23h and 25h, which a single naive offset guess renders as 24).
**A tooltip's position cannot be inferred from an index.** The month blocks are `flex-wrap`, so
neither the month index nor the weekday column says where a block actually sits — MEASURED on a
one-year window, 15 blocks wrap and the LAST one starts a new row at the card's LEFT edge, where
an "it must be the rightmost block" rule hung the tooltip 105px outside the card. Tooltips
therefore open rightwards from the cell and the grid container RESERVES a right strip (`pr-44`,
against a measured 144px widest tooltip) so they fit by construction. Verified zero overflow past
the card and past the viewport at 366 cells / 15 blocks, at both 1600px and 900px wide.
**Chart colour is computed, not chosen**, with the `dataviz` skill's validator against the page
surface `#0a0a0a` (the app is dark-only: `globals.css` has one unconditional `:root`, zero `dark:`
variants, no `prefers-color-scheme`). Two sets, and both REJECTED alternatives are worth keeping:
- `HEAT_RAMP` is indigo-700/500/300/100, running deep to pale as magnitude RISES because on a dark
surface lightness is prominence. The obvious Tailwind run indigo-900/800/600/400/300 FAILS —
900↔800 are 0.04 apart in lightness and 800 sits at 1.73:1 against the surface, i.e. a busy day
rendering as background.
- `TOKEN_COLORS` is sky-600/orange-600/emerald-600/violet-500 (worst adjacent CVD ΔE 10.1,
normal-vision 28.8, all ≥ 3:1). Reusing the existing `tokens`/`cached`/`fresh` trio FAILS
outright — sky-300 vs cyan-300 measure CVD ΔE 4.6 and normal-vision ΔE 5.6, indistinguishable to
everyone. Those three were picked for the task page's total-time card, where each figure has its
own text label doing the identifying; adjacent fills in one bar have no such crutch, which is why
the task page keeps them unchanged.
Colour binds to the ENTITY and never to rank, so a filter that reorders segments repaints nothing.
**The token mix is a bar, not a donut,** and that is a measurement: cache reads are 73.4% of all
tokens on the dev install and output is 1.7%, which as a pie is a six-degree slice beside an arc
three-quarters of the way round. Segments carry no inline labels — a 1.7% segment cannot hold a
legible one, and a label clipped by its own segment is worse than one in the legend beneath. Note
the two different denominators on that tab: the bar's cache-read share is of all four buckets while
the "cached" tile is `cacheHitRatio`, of the prompt side alone, so the card says so.
### Ranked bar lists
`RankedBars` (`components/stats/ranked-bars.tsx`) replaces the two-column count tables on the
quality and reliability tabs, and `InlineBar` adds a scan aid to the per-task agent-hours column.
Tracks are sized against the LARGEST row, not the total: these lists answer "which of these is
biggest", which is a different question from `StackedShareBar`'s, and a share denominator
flattens every row once the list is long.
**One hue per list, and never one per rank.** These are magnitude comparisons, so colour carries
no identity; `COLORS[index % COLORS.length]` — the obvious shortcut, and what the upstream design
this borrows from does — repaints every surviving row whenever a filter changes the set. Where a
row's colour does mean something, the caller passes a function keyed on the ROW.
**Severity is a SCALE, so it is not sorted by count** (`order="given"` plus `SEVERITY_RANK`,
imposed on the page because the endpoint groups without an `ORDER BY`). Two consequences: the
colours are the reserved status steps, each read beside its own label rather than from hue alone;
and the rank number is SUPPRESSED whenever the order is not by value. MEASURED on the dev install,
numbering the scale reads "1 high, 2 medium, 3 low" against counts of 4, 35 and 33 — a rank
column on a non-ranked list states the opposite of the truth.
Provider bars were considered and left out: three providers on this install, and the table is
already ordered by tokens, so a second ranking on the same card would compete with it.
**Per-provider token bars are served normalised, and that is the whole reason they exist.**
`spend.byProvider` ships the RAW `inputTokens` its `TaskProviderUsage` shape has always carried,
so a bar built from that column would double-count cache reads for exactly the two providers the
normalisation exists for — MEASURED live, codex reads 46.5% cached raw against 86.9% normalised.
`/stats/summary` therefore attaches a `tokens` object per provider, and `toRawTotals` is SHARED
between that map and the window total, so Σ the rows equals `tokens` structurally rather than by
two call sites happening to agree (VERIFIED live: all five fields match exactly). It is attached
to the response and NOT folded into `TaskProviderUsage`, which the task detail page mirrors and
which has no business carrying a statistics concern. Doing it in the browser was never an option:
`inputIncludesCache` lives in `@haive/shared`, which web must not import.
Every bar in that card — the window total included — shares one left edge and one
`tokenSegments` definition. Each is normalised to its OWN total, so the card compares
COMPOSITION and never size (size is the figure beside the name), and a segment that changed
colour or position between the total and a provider row would make that reading wrong rather
than merely inconsistent. The legend is rendered once, on the total.
### Per-step spend and model identity
`GET /stats/steps` (lazy, behind its own tab) answers the question nothing else could: which
STEP the money and the hours go to. MEASURED on the dev install, `01-plan-build` alone is 38.3
of ~82 agent-hours.
Attribution is `coalesce(task_step_id, summary_for_step_id)` joined back to `task_steps` for the
human `step_id` — the same fold `enrichStepsWithCliStats` applies per task, so the window
reconciles with the per-step badges. VERIFIED live against `/summary` and `/tasks`: agent-ms
295,679,567, notional $1,237.56 and 1,718 invocations, identical across all three.
Three things this endpoint does differently from its neighbours, each for a reason:
- **`invocationAttributionFilter` APPLIES**, unlike on `/reliability`, which deliberately counts
superseded and unattributed rows because those rows ARE the waste it measures. A spend rollup
is the opposite case and has to reconcile.
- **agentMs is summed from the TIMESTAMPS, not from `duration_ms`.** That matches how every other
agent-hours figure here is defined, and MEASURED, 17 rows carry both timestamps and a null
`duration_ms` — the column form would silently drop them. Summed and never unioned: the
busy-span union is per task and does not decompose by step.
- **Step failure counts stay OUT.** `task_steps` is windowed on its own `created_at` (a step has
no `started_at` until it runs) while this rollup is windowed on the invocation clock, so one
row carrying both would report a "runs" and a "failed" describing different sets.
`/reliability` already ranks failing steps on the correct clock.
The model rollup groups on `model_identity ->> 'served'` — the same `->>` access `/reliability`
already uses, so no new pattern. A NULL `served` is rendered as **not recorded**, never folded
into a model or into a zero: codex and amp report no model at all and are permanently
`match: 'unknown'` by design. That bucket is not small and not a gap to chase — MEASURED, 162 of
1,718 invocations, and the LARGEST agent-hours of any row (34.3), because those are the long
codex runs. `billed` is not exposed at all: grok bills `grok-4.6-build` while serving
`grok-4.6`, so it is not an identity source. There is no index on `model_identity`, so this
group-by is a filtered scan narrowed by `cli_invocations_started_at_idx`; if that ever matters
the fix is an expression index on `(model_identity ->> 'served')`, not a denormalised column.
**Mining-agent ranking was evaluated and dropped.** `task_step_agent_minings` holds 1,486 rows
across 1,467 distinct `agent_id`s — 321 are generated `plan-expand-<nodeId>-p<N>` ids and the rest
are near-unique too — so a "most used agents" list is ~1,400 entries tied at n=1-2. Grouping on
`agent_title` is worse (1,456 distinct), because those are per-node prose rather than persona
names. The only signal in that table is failure rate, which belongs on the reliability tab.
### Agents, skills and MCP tools
`GET /stats/tool-usage` and `GET /tasks/:id/tool-usage` read `cli_invocations.tool_usage` through
ONE SQL rollup (`api/src/lib/tool-usage-rollup.ts`). The caller composes the predicate and the
rollup never decides which rows count; its LATERAL `jsonb_array_elements` / `jsonb_each_text`
reads are guarded INSIDE the function argument because both throw on a JSON null, which `loaded`
IS for every codex, amp and gemini row. `invocationAttributionFilter` APPLIES, as on `/steps`: a
reconciling rollup whose `total` must equal the runs the other tabs count.
**Denominators are stated, never implied.** `total` is the reconciliation figure; `unrecorded`
(a NULL column) and `unobservable` (`coverage: 'none'`) enter no "not used" sentence;
`observable` (full + partial, the partial ones flagged as floors) is the ONLY denominator such a
sentence may cite, and `share` is `sampledRatio(runs, observable)` so a three-run window renders
`n=3`. Assigned personas are counted on any RECORDED row — an assignment is a dispatch fact,
valid on a `none` row — and `assignedRecordedSince` says from when they exist at all, so the UI
says "not yet recorded" instead of rendering an empty list as "unused". The task endpoint groups
by the same `coalesce(task_step_id, summary_for_step_id)` fold as the step badges and sums its
per-step rows with `sumToolUsageSteps` (`@haive/shared/stats`), so the browser does no
arithmetic.
**The unused report compares what is installed with what the runs used, and it reads the
installed half from disk by NAME only.** `api/src/lib/tool-inventory.ts` walks the catalog's
`projectAgentsDir` / `projectSkillsDir` union through `@haive/shared/fs-safe` from the repository
root (`storagePath ?? localPath`, never a task worktree): a symlink anywhere is counted under
`skippedLinks` and never followed, no file content is opened, and an anchor that is not a
directory answers `unreadable` — never "nothing installed", which is what a lenient `readdir` on
a missing path would have said. `classifyInstalledItem` (`@haive/shared/stats`) decides from
four facts and nothing else: on disk plus a live `onboarding_artifacts` row on one of its paths
or an `agent.<id>` row in `template_manifest_cache` is `haive` (an upgrade writes it back, so
never a purge candidate); on disk and nobody's is `unmanaged`, the candidates; named in a run's
`loaded` inventory but not on disk and not a template is `cli-builtin` — claude's `Explore` and
`general-purpose`, or a file removed since the run — where the lever is a flag, not a deletion.
"Used" is judged over the WINDOW and `lastSeenAt` over the caller's whole history
(`toolUsageLastSeen`), because "not seen in this window" and "never seen since install" are
different claims and only the second supports deleting a file; a window with no observable run
reports NO rows, since nothing can then be called unused. MCP rows are offered-minus-called from
the runs' own `loaded.mcpServers`, never from `.claude/mcp_settings.json`, which is user-owned
and not read here. The three failures — `no-repository`, `no-path`, `unreadable` — are answers,
never a 404, so the rest of the tab still renders.
## Sandbox
`packages/worker/src/sandbox/clawker-client.ts` wraps the clawker binary. The worker container mounts `/var/run/docker.sock` and uses Docker-in-Docker to spawn per-task containers. Only the cloned repository is bind-mounted into the per-task container. The worker filesystem and the user home directory are never exposed. CLI authentication files are copied into a named volume per task at startup and the volume is destroyed at task end. At worker boot `reapOrphanedAuthVolumes` removes a task's copy whose task ended, and an isolated provider's or a user's auth volumes once that row is gone, so a deleted provider or user leaves no CLI credential on the host. A login session still holding such a volume is removed with it, since its row went with the owner's and nothing else would. The owner is read back from the volume's name (`cliAuthVolumeOwner`, beside the builders).
**The base image builds itself, and that is why there is no manual first-boot step.**
`haive-cli-sandbox:latest` is built from `packages/worker/sandbox-image/` and pushed to no
registry, so `docker image prune` — and a fresh checkout — leave docker resolving a
Docker Hub repository that does not exist, which is what "pull access denied" means here
and not an auth problem. `ensureSandboxCoreImage` (`sandbox/sandbox-core-image.ts`) is the
one builder: fire-and-forget at boot beside `ensureDdevCa` so a pruned host heals BEFORE
the first task, and again at every site that needs the image — the two build entry points
(`handleBuildSandboxImageJob`, `resolveBaseImageId`) plus the three that `docker run` the
base DIRECTLY (`runInSandbox`'s no-derived-image fallback, `ensureTaskAuthVolumes`,
`ensureIdeVolumes`). Deliberately NOT the three sites that swallow their own failure
(`usage-window/token-source.ts`, `npm-cache.ts`, the login chown helper) — a throwing call
inside a deliberately-silent path changes its contract, and they inherit the healed image
anyway. The in-flight promise is load-bearing rather than a boolean flag: cli-exec runs at
concurrency 5-7 and a flag set only on success lets every one of them start the same
multi-minute build.
Two consequences worth keeping. The ensure is a no-op whenever `SANDBOX_IMAGE` names
something other than the core tag, because building ours over an operator's pinned image
would silently replace it — the "missing image" failure is then correctly theirs to fix.
And `packages/worker/Dockerfile`'s production stage must keep copying `sandbox-image/` and
`docker/`: it otherwise ships only `dist`, which leaves both this self-heal and
`ensureDdevRunnerImage` with no build context on a deployed host.
**Builds of one image tag run once at a time.** Providers that resolve to one tag (the claude
family at one CLI version) each queued a `docker build` of it, and the builds ran side by side in
the light lane; each then removed the image the tag had named, which could be the one the other
had just built. `handleBuildSandboxImageJob` keeps the builds in flight by tag (`inFlightBuilds`):
a second provider joins the running build and does only its own bookkeeping, and the builder alone
removes the previous image. In-process is enough, since the queued jobs and the inline dispatch
build both run in the one worker compose pins, and a joiner holds its slot exactly as long as its
own build would have. Marking a shared tag ready skips every sibling still `building`
(`markProvidersReady`): that row has a build of its own queued or running, forced or not, and
reports its own result. A sibling's cache hit during a forced rebuild used to mark the rebuilding
provider ready and re-enable its Rebuild button while the build still ran. Joining the build in
flight could not close that, since the api sets `building` the moment Rebuild is clicked, before
any job exists to join.
**A provider's row names its image only once that image exists.** A build used to write its new tag
before building, so one that failed left the row naming an image that was never made, nothing named
the old one any more, and a later success removed the failed tag as "previous" while the old image
stayed for good. A build now writes only its status when it starts. On success one locked write
(`markProvidersReady`, its rows locked in id order) moves the provider, and on a shared tag every
sibling not still building, to the tag, each only while its config asks for that tag. It keys on
config, never on the tag a row names: a failed build leaves its row on the image it had, so a
sibling on this tag whose config has moved on keeps showing its own failure rather than being
marked ready on an image it no longer wants. It hands back the tags those rows named before, and
each goes through the removal check below. The claim is made in the
tag's turn (`withImageTagLock`) and only while the image stands, and every removal first waits out a
build of its tag running here: with the row no longer naming a tag while it is built, a provider
moving off that tag otherwise found it unnamed and removed it just before the build claimed it. A
build of a config the provider has since left moves nothing and records no failure on the row,
since the build of its new config owns it; its image goes unless a row names it, and its caller, an
inline dispatch included, hears the build as failed for that reason rather than running an image the
provider no longer asks for. Every write a build makes to its own row lands only while the
provider's config still asks for its tag (`writeWhileWanted`), its first included, so a build
already left when it starts builds nothing: an older build's `building` landing after a newer build
marked the row ready used to leave it `building` for good. A provider that stops needing an image has the one it named removed the
same way.
Deleting a provider removes the image it named. The api never touches Docker, so its DELETE queues
`REMOVE_SANDBOX_IMAGE` with the row's tag, and the worker removes the image through the same check a
rebuild's old image goes through (`removeOrphanedPreviousImage`): kept while any other provider
names the tag, and only once a build of that tag running here has ended, whatever it ended in, since
it may have been this provider's own. A build whose provider is gone once it ends, whatever it ended
in, removes both the tag it built and the one its provider named before, under the same check: a
provider deleted before its build registered left the removal nothing to wait for, and only the
build knew the tag it was replacing. That check and the removal take their
turn per tag with a build's cache hit (`withImageTagLock`): a removal that read no provider naming
the tag otherwise deleted the image a cache hit had just marked a provider ready on, and a rebuild
removing its old tag raced a sibling switching to that tag the same way. A removal that could not
be queued leaves the image, as every delete did before, and never fails the delete that already
happened.
`cli_providers.sandbox_image_build_status` is reconciled against the real images on every
boot (`clearPrunedSandboxImageState`, `data-migrations.ts`). Not cosmetic: a stale `ready`
clears `createSandboxLoginContainer`'s gate and turns "not built" into `No such image`,
and a stuck `building` — which nothing else anywhere resets — DISABLES the Rebuild button
that is the only way out of it. Resetting `building` is safe there and only there, because
`runDataMigrations` runs before any queue starts and compose pins the worker to one
instance.
Secret-file masking (default on, Tier 1): before each cli-exec invocation the worker hides files matching a secret deny-list from the AI CLI agent by bind-mounting empty read-only files over them inside the cli-exec sandbox (`packages/worker/src/queues/cli-exec/secret-mask.ts`, threaded via `resolveSecretMasks` in `exec-core.ts` for the `cli`/`agent_mining`/sub-agent kinds). The effective set is `DEFAULT_SECRET_DENY_GLOBS` (in `@haive/shared`) plus per-repo `secret_mask_deny_extend`, minus `DEFAULT_SECRET_CARVEOUTS` and per-repo `secret_mask_allow`. Untracked files only (`git ls-files` filter) — committed secrets are out of scope FOR MASKING, and are instead REPORTED once per repo by the `07_7-secret-sweep` onboarding step (which is why they are two features and not one: masking stops an agent reading a secret, the sweep tells the human one is already in their history). The tracked filter asks each linked worktree about its own paths, because `git ls-files` reports paths relative to the tree it runs in, so the repo root never lists `.haive/worktrees/<name>/x`. The app runtime (app-runner/ddev mount the same `haive_repos` subpath without masks) still sees the real files. Per-repo controls live on the tooling settings page (`secret_mask_enabled`/`secret_mask_allow`/`secret_mask_deny_extend`); `CONFIG_KEYS.SECRET_MASK_ENABLED` is the global kill-switch.
Masking fails closed. A scan that throws, a scan root that is not a readable directory, a match count over `SECRET_MASK_LIMIT`, or a task/repository row that cannot be resolved raises `SecretMaskError` instead of masking a partial set — a subset leaves the remainder readable, which is the one outcome the deny-list exists to prevent. "Masking is off" and "no secrets found" are only ever concluded from evidence that says so: the kill-switch, `secret_mask_enabled`, or a task with no repository (which mounts no tree). The repo root mirrors `resolveTaskRepoMount` exactly, so a repo with no `storage_path` is scanned at its named-volume subpath rather than skipped, and the root is `stat`ed before the scan — glob answers `[]` for a root that does not exist, which is byte-identical to a clean repo, while the sandbox mount binds the real tree regardless of what the worker can see. `handleCliExecJob` records it on the invocation (exit -1) and fails the step. The escape hatches are the repo's `secret_mask_allow` globs and the masking toggles; disabling masking skips the scan and never raises.
Git-data masking (always on): every agent prompt states that git is unavailable inside the sandbox and that the host stages and commits (`10-gate-3-commit`, `completeMergeHostSide`). That invariant is enforced, not incidental, and `repoGitDataBoundary` (`packages/worker/src/queues/cli-exec/gitfile-mask.ts`) is the one place that decides the shape it takes. It rides the same `SandboxExtraFile` mechanism as secret masking but is an integrity control, so `SECRET_MASK_ENABLED` never disables it and it has no switch of its own.
A WORKTREE invocation gets an empty read-only file over its `.git` gitfile. Without it an agent can repoint the gitfile at the container path (`printf 'gitdir: /haive/workdir/.git/worktrees/<name>' > .git`), which both grants itself a working git behind the commit gate and leaves host-side git fatally broken for every later step. `removeWorktreeDir` runs `git worktree repair` before removal so worktrees poisoned before this existed still clean up.
A REPOSITORY-ROOT invocation gets the real `.git` DIRECTORY mounted back over itself read-only, through the nested-subpath mount `resolveTaskUploadsMount` already uses. That is every invocation of a task with no worktree — every onboarding step, the steps before `01-worktree-setup`, every plan task, a `kb_author` task with a repository, every same-branch merge fixer — where the whole checkout is mounted read-write and `.git/hooks` and `.git/config` are uid 1000, the sandbox user. Three shapes from ONE no-follow lstat: a directory mounts, a gitfile or a link masks (a mount cannot express a link), and an ABSENT `.git` does NEITHER — docker refuses a subpath that does not exist, and a read-only tmpfs over a missing destination leaves a root-owned stub on the host that would break `12-post-onboarding`'s own `git init`. Applied to every cli-exec kind (both sub-agent kinds included), to `01b-install-plugins`, which reaches `runInSandbox` directly, to the task Terminal and to the browser IDE; the mount target is derived from the mount, so the same function serves `/haive/workdir` and the IDE's `/workspace`. The task shell's fresh-spawn `chown -R` walks AROUND it (`chownCmd`): a plain recursive chown exits non-zero there and that failure is only logged, so it would read as "writes may be denied" for the whole workspace.
The REPO-scope Terminal is the one surface that keeps a writable `.git`, because committing and pushing from there is what it is for (`repos/[id]/terminal`). The app runner and the DDEV runner keep their unmasked view of the tree so the running app's own tooling works — but each mounts only ONE repos-volume subpath (`RUNNER_SUBPATH_LABEL` records which, and a reuse whose resolved subpath differs recreates the container), where both used to mount the whole volume and so every user's repository. A worktree's gitfile names an absolute host path that exists in NO container, so git in a worktree cwd never worked in the terminal, the IDE or the runners; only the parent checkout's did.
Host-side git is the other half, and it is what makes an unwritable `.git` less load-bearing than it sounds: every host git argv is built by `hardenGitArgs` (`@haive/shared/git-args`, applied by `repo/git-exec.ts` and asserted by `git-exec-guard.test.ts`), which overrides every key git can be told to execute — `core.hooksPath`, `core.fsmonitor`, `credential.helper`, `core.sshCommand`, gpg signing, the pager — and adds the two flags `-c` cannot express: `--upload-pack`/`--receive-pack` (MEASURED on git 2.54.0: a repository's `remote.<name>.uploadpack` survives `-c`) and `--no-ext-diff --no-textconv` on the diff subcommands (`-c diff.external=` does not disable an external diff, it makes git run the empty string and exit 128). `GIT_CONFIG_NOSYSTEM` and `GIT_CONFIG_GLOBAL` are deliberately NOT used: they drop the images' `safe.directory=*` and every call then fails with dubious ownership. What has no lever is a `filter` or `merge` driver named by `.git/info/attributes`, which is exactly why that file is read-only wherever an agent runs.
Per-call agent isolation (default on, `CONFIG_KEYS.AGENT_ISOLATION_ENABLED`): an invocation that has no use for the repository's agent definitions is given EMPTY ones, and the persona it does need is pasted into its prompt instead of pointed at. Every CLI lists its agent directory eagerly — names and descriptions, on every call — so for a reviewer or a miner that listing is pure context cost, and the claude binary's own 8,000-char budget (bundled skills charged first) left 1 of 18 repo skills described. `agentIsolationApplies` (`orchestrator/dispatcher.ts`) is the whole rule, pure, and all seven must hold: the switch is on, `input.kind === 'prompt'` (the sub-agent kinds rebuild each sub-step's spec worker-side), no `file_write` (a tmpfs mask is WRITABLE, so an edit to a masked agent file would vanish with the container), no `subagents` (a dispatch that may spawn native sub-agents keeps the catalog it spawns from), the step declared no `agentPool: '*'` (`07_7-secret-sweep` does, because agent definitions are committed files and its whole contract is sweeping the committed tree), nothing names an agent directory or a file inside one — the prompt with Haive's own marker blocks stripped, every persona body it carries scanned VERBATIM, and the selected provider's instruction chain (`adapter.rulesFile`, following `@` imports only in `import` mode) — and the EXTERNAL text the dispatcher appends after the decision names none either. That last condition exists because `adaptPrompt` splices the MCP surface and the global-KB digest in AFTER the rule returns, and both carry strings Haive did not write: a repository's own `mcpServers` keys, taken verbatim from its `.claude/mcp_settings.json`, and author-written KB titles. Without it a repository could name a server `.claude/agents/foo` and reach an isolated invocation's final prompt with a path the mask then hid. It scans ALL user server keys rather than the ones a given run renders — `reachableUserServerNames` needs the emitted-server set, which depends on render options a pure rule does not have, and shadowing is run-time state, so a name shadowed today renders tomorrow and isolation keyed on it would flicker between runs of one repository. `withAppReach` is the third post-decision append and needs NO scan: it interpolates `reach.url` alone, which is ddev's `primary_url` (scheme://host[:port], no path component), and a hostname cannot contain a slash. The provider's agent rules block (Agent rules, under the CLI adapter system) is the fourth and last, and it IS scanned: its text is a person's to edit, so the final decision in `buildCliSidePlan` takes it as external text too. The async pre-check runs before a provider is known, so it scans no rules and can only ever be the more permissive of the two. `promptNamesAgentPath` (`@haive/shared`) matches on whole segments from the root, so `.claude/agents/x.md` counts while `docs/.claude/agents/x.md`, `.claude` alone and the `-legacy` quarantine sibling do not. The two scans fail in OPPOSITE directions on purpose: the instruction scan fails OPEN (a file it cannot read, or a chain past 5 imports or 1 MiB, ends isolation, because a truncated scan must never hide a referenced file), while the persona reader fails CLOSED (a secret-mask policy it cannot evaluate pastes nothing, since a body already in a prompt cannot be retracted). An `@` import is resolved where the CLI resolves it, in the sandbox: against the importing file's directory under `/haive/workdir`, followed while it lands inside the mount, and counted when its own path names an agent file (`@docs/../.claude/agents/x.md`). One that lands outside is not followed, since nothing there is the repository's; a worktree run mounts only the worktree, so its imports cannot reach the repository root. Bodies are read by FILENAME from that provider's own `projectAgentsDir`, never by the frontmatter `name` `loadAgentPersonas` keys on — the two can differ, and a name lookup would silently drop the customisation that outranks the inline persona — never follow a repository-controlled link (`readFileNoFollow` refuses a link in any component and verifies the descriptor it opened), and share one 64 KiB budget counted from each opened file's own `fstat` size, where a body that would exceed the remainder is never pasted and never truncated (a cut definition reads as a complete one) and records an `agent_persona.oversized` task event. Exec never reads the switch: the decision rides `CliCommandSpec.maskAgentDefinitions`, which is what keeps one invocation's prompt and its mounts on the same side of a flip, and `resolveAgentDefinitionMasks` (`queues/cli-exec/agent-definition-mask.ts`) then mounts a read-only empty tmpfs over every catalog agent directory that EXISTS — masking an absent one makes Docker materialise a root-owned stub that outlives the container, and on a read-only local-path mount it cannot be created at all. Unlike secret masking this FAILS OPEN throughout: a stat that throws masks nothing and logs, because it is a context control and the persona is already in the prompt. Secret and `#ddev-generated` file masks whose target falls under a masked directory are dropped (Docker cannot make a mountpoint inside a read-only tmpfs), and a stub Docker left behind is removed in a `finally` only when it did not exist before the run, is owned by the run's uid, and is still empty.
**An import that exits 0 is not an import that happened.** `ddev import-db` returns 0 for a dump that created nothing — wrong engine, truncated stream, an archive whose restore wrote no statements — so `01c-ddev-env` counts the tables afterwards (`ddevCountTables`, via `ddev psql` / `ddev mysql` so DDEV owns the credentials). MEASURED on task ef954a3d: the step recorded `"imported": true, "DDEV started; database dump imported"` against a database with ZERO tables; every request answered `relation "semaphore" does not exist`, and the run only found out ~20 hours and 5 fix rounds later at gate 2, where no code change could repair it because the failure was never in the code. Only a CONFIDENT zero blocks — a probe that could not be read returns null and warns, since refusing an import on a probe that failed to run would block projects whose database is fine.
**`06a-db-migrate` detects a framework by files it SHIPS, not by composer.json.** A `drupal/core` regex reports `unknown` for every Drupal 7 site (7.90 ships no composer.json at all), which left `migrationCommand` empty, silently skipped `drush updatedb`, and — because the D7 bootstrap pre-flight below it is gated on `framework === 'drupal'` — made the check that exists to catch an unusable database dead code on exactly the sites it was written for. Markers must be TRACKED files: the workspace is a git worktree, which materialises tracked files only, so `wp-config.php` (credentials + salts, universally gitignored) is the wrong way to find WordPress and `wp-includes/version.php` is the right one — the same failure class as a test fixture missing from a worktree. Root `includes/bootstrap.inc` is D7-only (D8+ puts it under `core/`) so it is checked first; both resolve to `drupal` because the command and the pre-flight are identical.
**A `ddev restart` RECREATES the web container, so nothing installed inside it survives one.**
Only `/mnt/ddev-global-cache` (a named volume) and the bind-mounted project outlive it;
`/home/ddev/.cache` is in the container's writable layer and is not a symlink into that cache.
That made two things wrong at once. `07c-ddev-reconcile` compared the on-disk `.ddev/` hash
against a baseline read from `01c-ddev-env`'s ROUND-0 output that nothing ever rewrote, so once
the implementation touched anything under `.ddev/` the hashes differed permanently and every
later fix round restarted DDEV again — MEASURED on task 681f0f99, rounds 1/2/3 all
`action: restart` with php unchanged 8.3 -> 8.3. A successful restart/migrate now stamps
`appliedBaseline` (the state it applied) and later rounds diff against that; the field is
OPTIONAL, so a payload written before it existed falls back to the boot baseline and behaves
exactly as it did. `loadAppliedBaseline` deliberately does not use `loadPreviousStepOutput`:
that returns the highest-round row, which during this step's own run is THIS row with a null
output, so the prior round's answer would never be seen — filtering on a written output also
makes a retry correct for free, since the reset nulls that column.
**Playwright's runtime is provisioned per task, and the two halves have different lifetimes.**
The DDEV web image ships neither the browser binaries nor the libraries they link against —
MEASURED on `ddev/ddev-webserver`: `ms-playwright` absent and 13 of the 15 libraries chromium
needs missing — and nothing in the sandbox can add them, since the CLI is given
`ddev_status`/`logs`/`restart` and no `ddev exec`. `ensureDdevPlaywrightBrowsers`
(`sandbox/ddev-playwright.ts`, called from 08b before a ddev playwright run, idempotent) puts
the BINARIES on the `ddev-global-cache` volume behind a symlink at the path Playwright already
looks in — so no `PLAYWRIGHT_BROWSERS_PATH` has to be threaded through the run command and a
human running `ddev exec npx playwright test` gets the same browsers — while the apt packages,
which cannot leave the container layer, are re-installed per container behind a `/tmp` marker,
the lifetime that matches. MEASURED: ~70s to re-provision after a restart, with no re-download.
`playwright install-deps` CANNOT be used, and that is measured rather than assumed: on Debian
trixie Playwright falls back to its ubuntu20.04 package set and asks apt for
`ttf-ubuntu-font-family`, which Debian does not have, and one unavailable name aborts the whole
`apt-get install` — so the command exits 0 having installed NOTHING. The package list therefore
comes from Playwright itself (`install-deps --dry-run`) and each name is filtered by `apt-get
install -s`, the authoritative resolver. That filter is load-bearing, not defensive:
`libasound2`, `libatk1.0-0` and `libfontconfig` are all `Candidate: (none)` in trixie and all
three are libraries chromium needs, so filtering on `apt-cache policy` instead would have
silently dropped three real dependencies and produced the broken browser this exists to
prevent; the simulator resolves them through their t64 providers and rejects only the package
Debian genuinely lacks. Reading that list is the one volatile dependency, so it FAILS LOUD with
its own exit code rather than installing an empty set and marking the container done. The
script ships base64-encoded because `ddevExec` interpolates its argument into a `bash -lc`
running in the RUNNER, where `$HOME` and `$(…)` would evaluate against the wrong container.
Chromium only — `playwright install` cannot be told what a config needs, and naming no browser
downloads all three — so a suite driving firefox or webkit reports the gap through 08b's
`degradedNote` instead. `CONFIG_KEYS.TEST_BROWSER_PROVISION_ENABLED` is the kill-switch.
The DDEV project's Mailpit is surfaced at every step that already shows the running app (08a, Gate 2, `99-run-app-ready`), and the URL is always what `ddev describe -j` REPORTS (`parseDdevMailpitUrls`), never one we compose. That is what makes a single string right in both viewing modes: the headed browser runs INSIDE the runner and dials the same `*.ddev.site` name the host does, so the VNC half needs nothing published — MEASURED, `curl http://<project>.ddev.site:8025` from inside a runner answers 200. The host half publishes slots 3/4 at runner create (the DB-port argument: `01c` boots before the browser choice is known) and pins `ddev config global --mailpit-{http,https}-port` to the same numbers, in the exec that already sets `--router-*-port`; global config is per-runner, so no repo file is touched and nothing lands in review scope. NOTHING gets a `localhost` twin: the runner's whole traefik config carries one rule, `HostRegexp(^<project>\.ddev\.site$)`, with no catch-all, so `Host: localhost` is answered 404 on every entrypoint while `Host: <project>.ddev.site:9999` answers 200 — the router strips the port and matches the NAME (all MEASURED), and nothing here registers `additional_hostnames`. The app had carried such an entry since `83897d67` without it ever routing; it was removed rather than fixed, since the two `*.ddev.site` URLs already cover the case. `kind: 'localhost'` stays correct for `appRunnerAccessUrls`, whose port is published straight off the container with no router in front. Routing itself survives a port mismatch (the router strips the port before matching, MEASURED at 200 on a deliberately wrong one); the pin exists so the printed URL is right, and `decideMailpitHostUrls` drops a host link whose reported port disagrees with the published one, since a project's own `.ddev/config.yaml` OVERRIDES the global pin. The in-app "Open Mailpit" button is its own `runtime-ensure` job name, NOT a field on ENSURE — every ENSURE enqueue shares `jobId: ensure-<taskId>` and BullMQ coalesces on it, dropping a second payload silently — and it carries no URL, which the worker resolves itself so the route cannot aim that browser anywhere else. It navigates to the http URL: the runner's Chromium keeps a trust store separate from the system one curl uses, and an interstitial would land the user on a warning page instead of their mailbox.
One browser per task, one TAB per agent. Every sandboxed CLI of a task gets the same `--browser-url` (`resolvers.ts` probes the runner once per invocation), and chrome-devtools-mcp selects `pages[0]` on connect — so N concurrent agents drive ONE tab unless told otherwise. They are told otherwise by `BROWSER_TAB_DISCIPLINE` (`sandbox/mcp-surface.ts`), which rides the MCP surface block and therefore reaches every browser-capable dispatch: llm, mining (`08d-adversarial-qa`, the only fan-out keeping the full surface) and `dag_parallel` coders. `isolatedContext` is banned there rather than unmentioned — it is the obvious-looking way to isolate and it starts a fresh cookie jar, discarding the one deterministic app login `_app-auth.ts` performs per task. Tab 0 is the human's view: `browser-probe-connect.js` and `browser-login.js` both reuse `pages[0]` and bring it to front. A per-agent headless browser is NOT the alternative — it cannot reach a `*.ddev.site` or app-runner URL from the cli-exec sandbox, and carries no login. Agents killed before their `close_page` are swept by `closeExtraBrowserTabs` (`browser-close-extra-tabs.js`) at the mining and DAG barriers ONLY, never at browser bring-up, which runs while a human may have tabs open. The tab it keeps is the one RECORDED as the human's (`browser-human-tab.js`, written by `browser-probe-connect.js`/`browser-login.js` at the moment they `bringToFront()` a tab, read back as a CDP `Target.getTargetInfo` id), never inferred from the tabs. Three inference attempts have been MEASURED lying: `browser.pages()` is not creation order (two probes minutes apart put the newest tab at index 0 and then the oldest, so an early keep-`pages[0]` version closed the app tab the human was watching), `/json/list` is a third order again, and `document.visibilityState` — which replaced them — reported `visible` for BOTH tabs of a two-tab window (identical `windowId` and bounds), along with `document.hidden` false, `hasFocus()` true, animation frames and screencast frames, so the sweep failed safe and closed nothing on exactly the runners that leak. No record, an unreadable one, or a recorded tab that is no longer open still closes NOTHING; a leaked tab costs memory, closing the wrong one costs work. The launch script also restores puppeteer's three anti-backgrounding switches, since with a tab per agent all but one are always in the background — the likeliest reason no page-side signal separates them any more.
The same barrier RESTORES THE WINDOW, and Gate 2's bring-up does it again before the human looks (`browser-restore-window.js`, `restoreRunnerBrowserWindow`/`restoreAppRunnerBrowserWindow`). `resize_page` is not viewport emulation: MEASURED in the shipped chrome-devtools-mcp 1.7.0 bundle it un-maximizes the OS window and then sets its bounds so the CONTENT matches, so 08a's deterministic 1280x800 outlives the MCP session — three live runners were sitting at 1280x887 on a 1920x1080 screen. The agents cannot undo it and are not asked to: `resize_page` takes a CONTENT size and there is no window-state tool, so the screen size overshoots the display, and one window serves every agent's tab, so a per-agent restore would resize a sibling mid-screenshot. Restore target is the screen read from the page, NOT the 1920x1080 literal in `start-browser-desktop.sh` and not the launch geometry (Chrome offsets that by 10,10, which would hang 20px off two edges). Chrome settles at 1919x1079 for a 1920x1080 request, so the skip test asks whether the window COVERS the screen rather than matching it, or every barrier would rewrite identical bounds.
## Build commands
- `pnpm install` installs all workspace dependencies.
- `pnpm build` runs `turbo run build` across the workspace; `@haive/shared` and `@haive/database` build first because all other packages depend on them.
- `pnpm typecheck` runs `tsc --noEmit` everywhere.
- `pnpm test` runs Vitest across the workspace.
- `pnpm test:e2e` runs Playwright against the dev compose stack.
- A smoke that puts jobs on a queue (`smoke:onboarding`, `smoke:workflow`, `smoke:fix-loop` and
their siblings) needs its OWN Redis and database, as CI gives it. Pointed at the dev stack's, the
dev worker takes its jobs off the shared queue and runs them with the dev stack's code rather than
the branch's. Start a throwaway `redis` container and a scratch database for it.
- `pnpm --filter @haive/database migrate` applies pending SQL migrations to `DATABASE_URL`. This is the applier: the `db-migrate` compose one-shot and CI both run it, and api/worker/web wait for it via `depends_on: service_completed_successfully`.
- `pnpm db:push` (`drizzle-kit push`, interactive) is the DEV escape hatch, not the applier. Use it to iterate on a schema shape, then hand-write the numbered migration before committing — the CI `schema-parity` job compares the two and goes red if they disagree. It will offer to DROP `schema_migrations`, because that table is deliberately absent from the Drizzle barrel; accepting is recoverable (the next `migrate` re-adopts) but never what you want.
- `pnpm docker:dev` (alias for `scripts/dev.sh up`) boots `docker-compose.yml` plus the dev override, GPU-aware. The script also exposes `rebuild`/`reset`/`restart`/`libs`/`logs`/`status` — run `pnpm docker help`.
- `pnpm docker:down` (alias for `scripts/dev.sh down`) stops everything; it keeps all data volumes (never `-v`).
### Cutting a release
The release event is a TAG, never a merge to main. Main takes dozens of commits a day and stays
CI-gated but publishes nothing; `git tag v0.2.0 && git push --tags` is the whole release, and
`.github/workflows/release.yml` does the rest. **The tag IS the version** — nothing in the tree
carries a release number (`APP_VERSION` is the dev sentinel `0.0.0-dev`), so there is no
version-bump commit and no drift across package.json files.
The workflow builds api/worker/web for amd64 and arm64 on NATIVE runners (the repo is public, so
GitHub's arm64 runners are free; macOS is a first-class RUN-IT target and QEMU would cost tens of
minutes), pushes them to `ghcr.io/<owner>/haive-*` by digest, joins each into a manifest list
tagged with the version plus `latest` — or `next` for a `-rc` tag — and attaches a `release.json`
to the GitHub Release. `haive upgrade` reads that manifest: it carries the image digests, the
`migrationHead`, and `contracts` (whether this release removes anything, which decides if a
rollback can simply re-pin the previous tag).
Pass `--min-from` to `scripts/build-release-manifest.mjs` only to declare a REQUIRED STOP ("you must
pass through v0.3 first"). It defaults permissive on purpose: a floor equal to the release blocks
every upgrade to it.
To run a published release instead of building from source:
```bash
HAIVE_VERSION=0.2.0 docker compose -f docker-compose.yml -f docker-compose.run.yml up -d
```
That overlay swaps `build:` for `image:` (via `!reset`, because compose MERGES and would otherwise
still build) and adds the `db-migrate` one-shot that `docker-compose.yml` lacks — the dev stack gets
it from the dev override, so a published-image install would otherwise boot against an unmigrated
database. `HAIVE_VERSION` has no default so the command fails rather than silently resolving
`latest`; which version an install runs is the question every upgrade and rollback turns on.
The api and worker packages depend at build time on `@haive/shared` and `@haive/database`. Always build those two first when running anything outside of turbo.
### Never drive compose directly
`scripts/dev.sh` is the only supported entry point for the stack LIFECYCLE — run it as
`pnpm docker <command>` (or `bash scripts/dev.sh <command>`). Do not run raw
`docker compose up/build/down` by hand: dev.sh layers `-f docker-compose.yml -f
docker-compose.dev.yml` and then adds an auto-detected `docker-compose.gpu.yml` (NVIDIA) or
`docker-compose.vulkan.yml` (Intel/AMD iGPU). A hand-run `docker compose up` silently omits
all of that, so it boots WITHOUT the dev override and pins Ollama to the CPU — a stack that
looks healthy and is merely slow, which is why the mistake surfaces late. Never add `-v`,
and never prune this project's volumes.
- `pnpm docker restart [service...]` recreates services and rebuilds shared libraries once.
- `pnpm docker rebuild [service...]` handles dependency or lockfile changes and recreates the appropriate dependency volumes. Use it without a service for root, shared, or database dependency changes.
- `pnpm docker reset` recovers stale or corrupt compiled output while preserving application data.
- `pnpm docker libs` rebuilds `@haive/database` and `@haive/shared` with one container writer.
Host-side `pnpm build`, `pnpm --filter ... build`, `pnpm typecheck` and `pnpm test` are
SUPPORTED and expected — turbo's `test` and `typecheck` tasks both declare
`dependsOn: ["^build"]`, so those two build `shared`/`database` on the host by design.
`packages/*/dist` is on the `.:/app` bind mount and therefore has two writers: the host
(uid 1000) and the root-running `dev-libs` container. `scripts/build-libs.sh` chowns both
dist dirs back to the repo owner after every build, failed ones included, precisely so the
host build always has a directory it can write — read its header before changing any of
this. MEASURED: a host `pnpm --filter @haive/shared build` exits 0 in ~5s, emits
byte-identical output to the container build, leaves dist uid-1000-owned, and api, worker
and web keep serving across it. If a host build ever fails with TS5033/EACCES under
`packages/*/dist`, run `pnpm docker libs` and retry; never chown by hand.
`pnpm install` is the one to prefer running through `pnpm docker rebuild`. api, worker and
web each mask `node_modules` with a named volume, so a host install cannot reach them — but
`db-migrate` is the single service with no such override and installs as root straight into
the bind-mounted host tree. A host install is not destructive, it just races that one
container. Two things follow from that shared tree, and BOTH are load-bearing in
`docker-compose.dev.yml`: db-migrate runs on `node:26-bookworm-slim` rather than alpine,
because pnpm resolves native optional deps for the libc it runs on and an alpine install
filled the host's store with musl builds (host `pnpm test` then died in rolldown with
"Cannot find native binding", whose named cause — a missing wasm fallback — is a red
herring: the dep resolved fine, for the wrong libc); and its command chowns
`/app/node_modules` back to the repo owner, the same dance `scripts/build-libs.sh` does for
`packages/*/dist`, since otherwise the tree it writes is root-owned and the host cannot
re-install to repair the store it just read.
For source-only changes to the api, worker, or web app, rely on the bind-mounted source and
the service's dev watcher first; confirm the loaded source and the logs before deciding a
restart or rebuild is necessary. The api and worker watchers POLL (`CHOKIDAR_USEPOLLING` in
`docker-compose.dev.yml`, read by the chokidar tsx bundles): watching by events, tsx stopped
seeing a file once git had replaced it on the bind mount, so a pull could leave the old code
running with no restart line. MEASURED in the worker container: the second `git checkout` of one
file and every write after it went unseen, all of them seen when polling, at ~1.6% of one core
for all 744 worker source and lib files polled once a second. A check with no repository wrapper runs inside the matching
existing service container — `docker exec` is fine for diagnostics and tests, but never to
install dependencies or rebuild runtime artifacts. Before restarting or rebuilding the
worker, inspect active tasks: recreating it can interrupt live CLI terminals and
in-progress task steps.
## Conventions
- All modules are `"type": "module"`. Use `.js` extensions in import paths even for TypeScript sources because of `NodeNext` module resolution.
- Zod is used for both validation and for generating `FormSchema` field metadata where possible.
- Logger is `pino` from `@haive/shared/logger`. Never `console.log` from server code. ONE exception, and it is structural rather than a lapse: `packages/database/src/migrate/` is a CLI and `@haive/shared` DEPENDS ON `@haive/database`, so importing the logger back would be a dependency cycle. It emits one JSON line per event through its own `emit()`.
- Host-side IO on a repository path goes through `@haive/shared/fs-safe`, never through a path-based `node:fs` call. The worker and the api run as root over trees that repositories and sandboxed agents write, so a path resolved by name is a path they can redirect; the primitives take `(anchor, rel)`, resolve `rel` one held directory descriptor at a time, refuse a link in any component, and verify the held inode through `/proc/self/fd` (Node-only, Linux-only, out of the root barrel and of every subpath web imports). Anchors are `<storage>/<userId>/<repoId>` or a repository's `localPath` — never `.haive`, a worktree or an uploads dir, since the sandbox mounts the whole repository root read-write. `packages/shared/test/fs-ratchet.test.ts` pins the remaining path-based calls per file (`fs-ratchet.json`, exact counts): a conversion lowers its entry, and a new call needs a reason in the PR. A child process follows the same rule: `runTool` (`worker/src/repo/tool-spawn.ts`) hands `tar`, `unzip` and `pdftotext` what they read and write as held descriptors, named `/proc/self/fd/N` in the child, never a path they would resolve by name, and runs them with PATH and LANG only, as uid 65534 under a root worker. A tool that re-opens its input (unzip seeks) needs that uid to be able to read it. Every upload is written 0644, and a file that is not is copied through the descriptor into a private, already unlinked file the uid can read (`toolReadable`), so the original's mode is never changed.
- postgres.js is patched (`patches/postgres@3.4.9.patch`, pnpm `patchedDependencies`; upstream
porsager/postgres#1209, unreleased) so a connection whose backend died rejects what it still owes
instead of writing to its closed socket. In 3.4.9 a transaction's ROLLBACK did that from a
`setImmediate`, and the TypeError killed the api or worker process. Every image copies `patches/`
before `pnpm install`, which refuses the lockfile without it, and a lockfile change reaches the dev
stack only through `pnpm docker rebuild`. A pool opened for one job ends with `end({ timeout: 5 })`:
after such a drop a bare `end()` never settles. `smoke:pg-drop` covers both, and is how to tell
whether a later release carries the fix and the patch can go. Dependabot leaves postgres to a
person (`.github/dependabot.yml`): pnpm refuses a lockfile whose patch matches no installed
version (`ERR_PNPM_UNUSED_PATCH`), so a bump in its weekly group would fail the whole group.
- Secrets are stored via envelope encryption: per-user DEK encrypts the secret, master KEK from `CONFIG_ENCRYPTION_KEY` encrypts the DEK. AES-256-GCM throughout.
- Drizzle schema lives in `packages/database/src/schema/`. Migrations are hand-written SQL in `packages/database/migrations/`, applied in filename order by the runner in `packages/database/src/migrate/`, one transaction per file with its `schema_migrations` row written inside it. **To add one:** iterate with `pnpm db:push`, then write `NNNN_name.sql` in that directory, keeping the guarded idempotent style (`ADD COLUMN IF NOT EXISTS`, `DO $$ … EXCEPTION WHEN duplicate_object`) — a developer's database is often legitimately AHEAD of the baseline and must survive your file. Declare a new column LAST in its Drizzle table: `ALTER TABLE ADD COLUMN` appends while `drizzle-kit push` builds the table in declaration order, so a column declared anywhere else makes the two schemas differ by column ORDER and turns `schema-parity` red — a failure no migration can fix. Never edit an applied migration: its sha256 is recorded, and a change hard-fails every install that ran it. `0000_baseline.sql` is generated and FROZEN; re-cutting the schema means a later numbered baseline, never an edit to that one. `pre-baseline/` is the record written before the directory was executable and is NEVER run — see its README: 19 of those files are unsafe to replay and one is destructive.
- Hono routes group by domain in `packages/api/src/routes/`. Auth middleware mounts globally.
- A request that changes state, and every WebSocket handshake, is refused when it comes from a page
other than the app's own (`isForeignOrigin`, `api/src/lib/request-origin.ts`). The session cookie
is `SameSite=Lax`, and every page on the same site still sends it: on localhost that is every
other port, the task's own app among them, so the cookie alone cannot say which page asked. The
api's own origin is allowed because the editor it proxies runs there. It is resolved as the
browser resolves it (`HAIVE_PUBLIC_API_URL`, else the app's host on `HAIVE_API_PORT`), not read
off `Host`, which a reverse proxy may rewrite to its upstream. A request with no `Origin` did not
come from a browser page. A GET must not start work either, since a link from any site carries
the cookie on a top-level GET; that is why the runtime access routes are POST. A new `'upgrade'`
listener fails `upgrade-origin.test.ts` until the test covers it.
- Forms are described by `FormSchema` from `@haive/shared` and rendered by `FormRenderer` in `@haive/web`. Do not write step-specific React components.
- EVERY prose body renders through `MarkdownView` / `.haive-md` (`packages/web/src/components/markdown/`). There is no plain-text branch. Underneath, the ONLY markdown renderer is `Markdown` (`components/markdown/markdown.ts`), react-markdown with `rehypeImageLinks` run last: an image in a body becomes a link to it and is never fetched, because every body is text an agent or a repository wrote and a fetch reaches the viewer's browser even inside a collapsed disclosure. `markdown.test.ts` fails on any other value import of react-markdown. Behind it, the page's Content-Security-Policy (`middleware.ts`, `lib/content-security-policy.ts`) admits images, media and fonts only from the page, `data:`, `blob:` and the api's host, resolved per request from the Host header because Next's `nextUrl` does not carry it, and named under both `http:` and `https:` because a TLS proxy hides the page's scheme from the server and Chrome does not let an `http:` source admit an `https:` URL on an explicit port; and mermaid (`mermaid-loader.ts`) draws labels as SVG text, locks its config against directives and frontmatter, and shows as code any source with a directive, frontmatter, `@{` shape metadata or a sequence `properties` line. PROSE is the word that scopes it: a form section carrying a structured payload instead of a body (`planImpact`, see the plan canvas notes) is not prose and does not belong in a markdown renderer — but such a section leaves `body` empty rather than shipping the same content twice, so there is still exactly one rendering of anything. `looksLikeMarkdown` still exists but decides ONE thing — the LINE-BREAK POLICY, not the style: a markdown body gets `remarkGfm` alone so a model's ~80-col hard wraps reflow, a plain body also gets `remarkSoftBreaks` so its newlines survive as `<br>`. Do NOT re-promote it to a renderer switch. It used to be one, and identical prose then jumped between mono-12px `<pre>` and sans-14px `.haive-md` on nothing more specific than whether it happened to contain one backtick. `remark-soft-breaks.ts` is a deliberate local stand-in for `remark-breaks`, not an oversight: the dep is not in the pnpm store and adding one costs a full stack rebuild. An option LABEL is markdown too, but only where a rich element renders it (multi-select checkboxes, both radio variants) — the two native `<select>` renderers keep theirs plain because an `<option>` holds text and nothing else, so a label written for a dropdown must read correctly unrendered. That was already the de-facto contract: `02-plan-coverage` and `00-plan-sequence` write inline code into labels and it rendered as literal backticks. MEASURED across all 803 stored option labels before the change, 3 contained markdown syntax and 0 of them were on a `select`. A generator emitting a label must now escape or fence what it interpolates — `11f-plan-reconcile`'s `code()` sizes the fence to the content, since a node title carrying one backtick would otherwise end the span early and spill the rest of the label as markup. A body nested INSIDE another styled block (form/field/option descriptions, note fields) uses `InlineMarkdown`, which takes its size and colour from a WRAPPER element via `.haive-md-inherit` — putting those classes on MarkdownView's own element does not work, because `font-size: inherit` resolves against the parent, so a `text-xs` there would be what it inherits FROM rather than what it applies. Inline code under that modifier is sized as a RATIO of the body (`0.9285em` = 0.8125/0.875), never the absolute rem the plain rule uses, with `pre code` reset to `inherit` so the same selector does not scale it twice. `globals.css` is outside the repo's prettier glob (`ts,tsx,js,jsx,json,md`) and is hand-formatted compactly — running `prettier --write` on it produces ~150 lines of unrelated churn.
- A fenced code block is found by one scanner, `scanFences` (`@haive/shared/markdown-fences`), and
never by a hand-rolled regex. It reads CommonMark's rules: a run of three or more backticks or
tildes behind at most three spaces, closed only by a run of the same character at least as long
with nothing after it, and an unclosed fence runs to the end of the text. A three-backtick regex
read a four-backtick fence's inner triple as its close, which lifted a quoted before/after pair
out as a side-by-side block and hid long blocks from the collapse toolbar. A line scanner cannot
see a fence inside a list item indented four or more spaces, or behind a blockquote's `>`. One
reader pairs differently on purpose: `11-final-review` unwraps a model's reply whose first line
opens a fence and whose last line closes it (`fenceOpener`, `closesFence`), and keeps the lines
between as they are, since a ` ```markdown ` wrapper holds ` ``` ` samples that close
it by CommonMark's rules.
- A message column is display copy, never state. `task_steps.status_message`, `task_steps.error_message` and `cli_invocations.status_message` all outlive the thing they describe: a park whose poll chain ended leaves its last line behind, a step that failed once and succeeded later keeps its error text, and an invocation picked up immediately can be labelled "queued" by a write that lost a race. Gate UI on the structural column that proves the state (`waiting_started_at` for a runtime park, `status` for a failure, `started_at` for a queued invocation) and use the message only as the words inside the banner. The rule lives in `packages/web/src/lib/step-banners.ts` (`parkBanner` / `failureBanner` / `invocationBanner`, unit-tested) — extend that module rather than re-deriving the condition at a new call site. Keying on copy presence produced three separate phantom states in one day: two live "waiting for a slot" banners on one task, a `done` step rendering "cli invocation failed" with a Retry button, and a running CLI advertising "Queued — machine at capacity".
- Do NOT enforce that rule as a CHECK constraint or a nulling trigger. `error_message` on a `done` row is legitimate for `fixLoopOnError` steps (`step-runner.ts`), which write `status: 'done'` together with the error as the diagnosis that routes the fix loop back to implementation — a blanket "done implies no error" would silently destroy it. Repair stale copy with a numbered data migration instead (see `0104`, `0105`, `0106`).
- `status = 'waiting_form'` alone does not mean a form is waiting for someone. The submit route clears `waiting_started_at` and leaves the status for the worker to move when it picks the job up, so a poller that stops on the status stops on a turn that was just sent. `isAwaitingFormInput` (`web/src/lib/submit-state.ts`) reads both columns, as the task page's step timer already does; the task queue stamps the column on every park it records (none of the 8 parked steps on the dev install lacked it).
## Where things live
- Step modules: `packages/worker/src/step-engine/steps/{onboarding,workflow,env-replicate}/`
- CLI adapters: `packages/worker/src/cli-adapters/`
- Sandbox wrapper: `packages/worker/src/sandbox/`
- Terminal proxy: `packages/api/src/routes/terminal.ts` plus `packages/web/src/components/terminal/`
- Orchestrator state machine: `packages/worker/src/orchestrator/state-machine.ts`
- Dispatcher priority chain: `packages/worker/src/orchestrator/dispatcher.ts`
## Phasing
Phase 0 scaffold is complete when `pnpm install` and `pnpm docker:dev` boot all services on a clean host with only Docker installed. Subsequent phases build the database schemas, auth, repository management, CLI adapters, sandbox, terminal proxy, step engine, sub-agent emulator, autonomous workflow, and environment replication in that order.
## Constraints
- WSL2 plus Docker is the only supported developer environment. No Windows-native installs.
- Concurrent per-task runtimes (DDEV/app runners) and CLI agents share ONE machine-aware RAM budget (`total - ~30% reserve`), not a fixed count. Two distinct numbers: the Docker `--memory`/CPU/pid **cap** each container runs under (a generous ceiling, swap disabled) and the **planning weight** admission budgets it at. Dividing the budget by the ceiling is what used to price a ~300 MB app-runner like a DDEV DinD. Weights are CALIBRATED absolutes, not fractions of the cap (a project's runner uses the same RAM on a 16 GB host as on a 64 GB one), measured from 1731 `docker stats` samples over 4 DDEV runners and 52 agent sandboxes: `ddevWeightMb` 1536 (peak 1036 incl. a full cold boot), `browserWeightMb` +1536 surcharge charged WHILE THE DESKTOP IS UP (browser-phase peak 2505), `agentWeightMb` 2048 (peak 1736), `appWeightMb` 1024 (the one unmeasured weight — strictly less machinery than a DDEV base). Each is clamped to the container cap, since a runner cannot occupy more than `--memory`. A cold `ddev start` is NOT the peak (image pulls are disk-bound); the peak is agent work during a verify/fix round, which is also when the browser desktop is up — so the browser flag is a proxy for that phase, not the cause. That surcharge is LIFECYCLE-SCOPED, not stamped into the weight label at container create: the desktop first starts at 07, so a task charged for it from `01c` paid for a Chromium that did not exist for hours — MEASURED, two of three live DDEV runners had zero Xvfb/x11vnc/chrome processes while each was charged 1536 MB for one, which is what parked other tasks against a budget nothing was spending. Labels are immutable after create, so the live half is a Redis SET of runner CONTAINER NAMES (`markBrowserDesktopUp`/`Down`, called inside `startBrowserDesktop`/`stopBrowserDesktop` so every step inherits it) folded into occupancy by `parseRunnerWeights`. It is clamped to the kind's class weight plus the surcharge, which is what stops a runner created BEFORE the move — whose label already contains it — being charged twice, while leaving a per-task `memoryLimitMb` pin (already above that cap) alone. Applied only to containers the same `docker ps` returned, so a stale entry is inert and no TTL or sweep exists. Agents take what the runtime pool leaves free, never below `agentFloor` (a runtime holder needs an agent to finish, so zero agents deadlocks) — but that subtraction is a difference of two peak-calibrated estimates and is wrong in BOTH directions at once, so `AGENT_POOL_MEASURED_ENABLED` (default on) lets the host's own `MemAvailable` answer instead. MEASURED: three DDEV runners charged 9216 MB were occupying 2551, flooring the pool at 2 while 7.9 GB sat free, and the dev base stack meanwhile overran its 30% reserve — a term the planned figure cannot see at all. `readHostAvailableMb` reads MemAvailable, NOT `os.freemem()` (that is MemFree: 750 MB against 7.9 GB available, which would zero the term), and prefers a cgroup limit when one is set below host total because `/proc/meminfo` reports the HOST inside a capped container. `agentWeightMb` stays the divisor — being coarse (19% of a 16 GB budget) is exactly why the planned quantisation could not express the difference. Only the RESERVATIONS half of runtime occupancy is subtracted from the measurement; a live runner's RSS is already missing from it. Growth BEYOND the planned figure is capped at `DEFAULT_AGENT_RAMP_STEP` (2) per 30s evaluation, each step re-measured with the previous one resident, so an illusory headroom stops itself; the plan's own width still opens at once (an idle host was never broken) and shrinking is never ramped, since a lower cap only stops NEW jobs. `AGENT_POOL_SAFETY_MB` (0 = a fifth of the host, clamped 2048..4096) is what the ramp holds back for the base stack's growth and for agents climbing from ~450 MB toward their 1736 MB peak; raise it if the host swaps as the pool ramps. Admin-tunable: RESOURCE_LIMITS_ENABLED / RUNTIME_MEMORY_MB / RUNTIME_CPUS / MAX_CONCURRENT_RUNTIMES=0 means no count cap / RUNTIME_DDEV_WEIGHT_MB / RUNTIME_APP_WEIGHT_MB / RUNTIME_BROWSER_WEIGHT_MB / AGENT_WEIGHT_MB / AGENT_FLOOR / AGENT_POOL_MEASURED_ENABLED / AGENT_POOL_SAFETY_MB / MAX_PARALLEL_AGENTS=0 means auto (a positive pin still short-circuits everything, measurement included) / RUNTIME_IDLE_REAP_MINUTES. On a DEV host the binding constraint is usually the base stack, not the weights: `next dev` alone holds ~1.6 GB and Ollama ~1 GB against the 30% reserve, so pin MAX_CONCURRENT_RUNTIMES or cap the dev web server's heap rather than raising weights. A step is gated only if it will actually spawn a runner (`needsRuntime: 'ddev' | 'app' | 'if-serving'`; `'if-serving'` resolves through `classifyRuntime`, so a code-only task never queues for a pool it does not use). A reaper reclaims leaked runners. See packages/worker/src/sandbox/{runtime-caps,runtime-admission,runtime-runner-reaper}.ts and `deriveRuntimeCaps`/`deriveAgentConcurrency` in @haive/shared.
- Runtime occupancy sizing the agent pool is only half the story: WHO gets those few slots matters too. A task holding NO live runtime runner yields its cli-exec slot to one that does (`enforceRuntimeHolderReserve`, `queues/cli-exec/agent-reserve.ts`), because a runner-holder's idle time is billed in committed RAM it can only release by FINISHING — while a runner-less task's slot is fungible. Observed without it: 3 tasks each holding a 3072 MB DDEV, two idle, an onboarding task holding one of the only 2 agent slots, and 4 more tasks parked behind capacity that could not free. Enforced at job PICKUP (a third sibling to `enforcePauseGate`/`enforceTaskAgentCap`, same `moveToDelayed` + `DelayedError`), never at enqueue — BullMQ freezes priority at `queue.add` while runner-holding is state that changes. Keyed on the docker-label invariant "holds a live runner", NOT on `tasks.type`, so a future runner-less task type inherits it. Demand is counted from the BullMQ WAITING list, not from queued `cli_invocations` rows: a DB count includes jobs that are delayed (paused task, per-task cap) and yielding to those idles the slot. Not preemption — a running agent keeps its slot. The one thing that outranks the reserve is an EXPLICIT vote: a runner-less task whose `tasks.vote_score` is STRICTLY above every holder's skips the yield (`voteScore`/`maxHolderVoteScore` on `agentReserveDecision`; both read live at pickup, never carried on the job, for the same reason the gate itself is at pickup). Strictly, because a tie states no preference and there the committed-RAM argument still decides — and at the default score 0 everywhere the term is inert, so an install where nobody votes behaves as it did before. A vote is the operator saying what to run first; holding a runner is this module inferring it from RAM, and the statement beats the inference. Without it a vote re-priced only the QUEUE while this gate handed the slot away anyway: MEASURED, +1 onboarding invocations sat 12.6/14.1/21.9 min behind unvoted DDEV holders that were at the head-adjacent bands. Admin-tunable: AGENT_RESERVE_ENABLED (off = first-come) / AGENT_RESERVE_MAX_HOLD_MINUTES (0 = strict, else release a held invocation after N minutes so runner-less work slows rather than stops; default 3, not 10, because the hold is only half the wait — a released job then waits for a slot to free, a term invisible in the knob).
- Queued cli-exec priority is `band = VOTE_BASE + rank - vote_score`, and BullMQ FREEZES it at `queue.add`. `rank` (the task's in-flight agent count) is therefore a SNAPSHOT, which turned a drained fan-out into a permanent sentence: MEASURED, an idle +1 task sat at band 9 (rank 5, from an `09_5` mining fan-out long since finished) behind a neutral task at band 6 that had two agents RUNNING, and led the queue on nothing but enqueue order. A vote cannot correct that on its own — it moves at most 5 bands and a 5-wide fan-out moves 4. `CliPriorityDecaySweeper` (`queues/cli-exec/priority-decay.ts`, 30s) rebuilds the band from live load: for `r` STARTED agents and `k` queued jobs in their current order the ranks are `r+1 .. r+k`, i.e. exactly what an incremental enqueue would produce right now — so the round-robin across tasks is preserved while the fan-out penalty decays. Order INSIDE a task never changes, and a settled queue produces zero writes. A sweeper rather than a hook on invocation-end because the count also drops on paths that never run that code (orphan sweep, preemption, cancel). It has no switch of its own: with `FAIR_SCHEDULING_ENABLED` off, jobs carry priority 0 and every one is skipped.
- The Docker socket mount in the worker container is effectively root on the host. Document this in the README and offer rootless Docker instructions in Phase 9 hardening.
- All step content lives in TypeScript modules. Do not pipe legacy markdown into a CLI prompt.
## Onboarding completion
**"Onboarded" is the RUN's verdict, not the artifacts'.** The four markers the API checks —
`.claude/agents`, `.claude/skills`, `.claude/workflow-config.json`, `KB_DIR` — are all written
by `07-generate-files`, the 8th of 27 registered onboarding steps, with the KB following at 08
and skills at 09_5. So from a third of the way in, a cancelled run and a run executing RIGHT
NOW leave a tree indistinguishable from a finished one: a task created against such a repo was
typed `workflow` and aimed at a knowledge base nobody finished building, and the repos list
offered "Create task" for a repository mid-onboarding.
`resolveOnboardingVerdict` (`api/src/lib/onboarding-state.ts`) is the one reader, shared by
`GET /repos` and `GET /repos/:id/onboarding-status`:
```
onboarded = markers present
AND no live onboarding task (created|queued|running|paused|waiting_user|waiting_pr)
AND (repositories.onboarded_at IS NOT NULL
OR a completed onboarding task exists
OR no onboarding task was ever started here)
```
`waiting_user` counts as LIVE deliberately — parked on a form is the normal state of
onboarding for most of its life, and it is the state the misreading happened in.
The last clause is what makes this deployable with NO backfill: a repo cloned in already
onboarded, and every repo onboarded before the column existed, keeps the verdict it had. It is
also why nothing re-derives the stamp at boot — a data migration that re-stamped from task
history would resurrect a repo whose artifacts were just reset.
`onboarded_at` is stamped by `stampRepositoryOnboarded` from the worker's `markTaskCompleted`
(`onboarding` type only), the same hook and the same reason as `completePlanNodesForTask`:
cancel and fail write through their own functions, so an abandoned run can never stamp a repo.
Cleared by `DELETE /repos/:id/onboarding-artifacts`, since the stamp must not outlive the files
it vouches for. `POST /repos/:id/mark-onboarded` is the manual route, for a run that did all
the work and then failed at a late step (13-onboarding-push against a repo with no remote); it
refuses when a marker is missing or a run is live.
**That reset takes back what onboarding wrote, and nothing else** (`resetOnboardingArtifacts`).
Its directories are DERIVED from the provider catalog, the same reason `getScaffoldEntries`
gives: the hand list it replaced was `['.claude', KB_DIR, LEARNINGS_DIR]` from when `.claude`
was the only CLI directory, so "start over" left the previous run's agents and skills on disk
for every other CLI and the next run wrote on top of them.
**The catalog is the CANDIDATE set, never the removal set.** 07 writes agents to the ENABLED
providers' dirs alone (`agentTargetsByDir`, from `providerRows.filter(p => p.enabled)`) and
`resolveSkillTargetDirs` does the same for skills, so on a repo where only claude is enabled a
`.codex/agents` or `.grok/skills` holds the user's own definitions and NOTHING of ours — and
this action is irreversible. `collectWrittenCliContent` answers it from the RUNS' own records,
never from the currently enabled providers: enablement is mutable global state that says nothing
about what THIS repo's onboarding did, so a CLI enabled afterwards would make its directory
eligible for a removal no run here ever wrote to, while one disabled since would strand the
skills it did write. A candidate outside that union is REPORTED, not removed.
**What it reads is what each step WROTE, not what it planned to.** 07's apply output carries
`wroteFiles`; its detect payload's `agentTargets` is the wrong source twice over, because with
the default `overwrite=false` `writeIfAllowed` SKIPS a pre-existing file — so a user's own
`code-reviewer.toml` is one a SUCCESSFUL apply deliberately left alone, and claiming it by
manifest id would exempt it from the quarantine and delete it with the directory — and because
the detect payload is persisted before the form is even shown, so a run cancelled while parked
there names directories nothing was written to (which is also why the query filters to
`status = 'done'`). `wroteFiles` additionally covers the two cases a target list misses: the
fallback write to `.claude/agents` when NO provider has an agents dir (amp alone, where
`agentTargets` is empty), and the LLM-discovered custom agents, which have no manifest id at
all. 09_5 contributes `written[].mirroredDirs`, each `written[].id` with the `SKILL.md` and
`sub-skills` inside it, and the `README.md` index it rebuilds every pass. `.claude` itself needs
no gate — it is Haive's own directory, and every `ONBOARDING_MARKERS` path lives in it.
Provenance is scoped to runs that started after `repositories.onboarding_reset_at` (migration
0161). A reset supersedes artifact rows but CANNOT touch `task_steps`, so an older run's
`wroteFiles` still names paths it wrote and the reset then DELETED — and if the user recreates
one of those names by hand and a later run SKIPS it under `overwrite=false`, that stale record
claims their new file. Reading only the NEWEST run does not fix it: with no re-onboarding since,
the newest run IS the pre-reset one. NULL there is every repo never reset, which reads exactly as
it always did, so the column needs no backfill.
`09_5b-skill-repair` is the third source and is read on its own terms: `repaired` (skill IDS)
with the target dirs in its DETECT payload. It CLEARS a failing skill's tree before rewriting
it, so 09_5's slug record for that skill is STALE — which is why the rows come back OLDEST
FIRST and a repair RETIRES the earlier claims under that skill before re-claiming it.
**A directory is claimed only when something inside it is named.** `hasDeeperClaims` is what
makes a claimed directory be WALKED, so claiming one with nothing named inside says the
opposite — that it is ours wholesale — and a file the user put there is deleted rather than
moved aside. `sub-skills` is therefore claimed only when the slugs in it were recorded, which
09_5b never does and 09_5 only does for outputs written since `subSkillSlugs` existed.
`11d-skill-sync` is a source ONLY for a task whose worktree was MERGED. It writes into the
task's WORKTREE (`resolveWorktree`), so until the merge its record describes a tree the reset is
not looking at, and claiming from it would delete an untouched ROOT copy of a skill it only ever
changed there. `12-worktree-cleanup` records that verdict as `merged`, set on the `merge_remove`
path and only when the merge actually ran, so the two rows are read together — which is why
`loadProvenanceSteps` also selects `task_id` and loads the cleanup step. Ignoring 11d wholesale
was NOT safe either, and that is the subtler half: an 11d removal leaves 09_5's claim for that
skill standing, so a file the user later recreates at the path is deleted as onboarding output.
A merged sync therefore RETIRES the claims of what it removed and re-claims what it generated.
It records no sub-skill slugs, so `sub-skills` under a skill it wrote stays unclaimed and is
moved aside.
**A claimed DIRECTORY is walked when anything claimed lives beneath it, and taken whole when
nothing does.** That is what separates a generated skill (`<skills>/<id>`, which holds Haive's
`SKILL.md` and `sub-skills` and may also hold a `NOTES.md` a person left there) from
`sub-skills` itself, which Haive renders wholesale. Without the walk the recursive removal took
the person's file along with ours; without the stop, every rendered sub-skill would be moved out
one by one and the directory would never empty. A descendant keeps its shape under the one
`-legacy` sibling (`<skills>-legacy/<id>/NOTES.md`) rather than growing a second quarantine
inside the tree.
**A row and a step hash are CO-EQUAL evidence, not a precedence chain.** `claimSatisfied` answers
a VERDICT — `ours` / `edited` / `unrecorded` — and either record matching the bytes on disk
settles it. It is not "the row wins", and the case that forbids that is live: 07 renders the
agents index through `resolveAgents`, which substitutes `stubCustomAgent` for an accepted id it
cannot resolve, while the manifest renders it through `resolveAgentsForIndex`, which DROPS that
id. Different table, so the `agents-index` row can NEVER match the README 07 actually wrote, and
a row-wins rule quarantines that file out of a directory Haive indisputably owns. The row is
still needed in the other direction: `02-upgrade-apply` rewrites a file and inserts a row
carrying the NEW hash while 07's step hash still names pre-upgrade bytes, so a step-hash-wins
rule would quarantine every upgraded repository's own files.
**`edited` is a claim about AUTHORSHIP and is only made where authorship is PROVEN.** A step
hash proves it alone, because one exists only for a path that step WROTE — `writeIfAllowed`
skips a pre-existing file and records nothing for it, and that absence is load-bearing rather
than an omission. A row does NOT prove it: `recordOnboardingArtifacts` inserts one per manifest
RENDERING, so a file apply skipped carries a row too, holding what Haive WOULD have written. A
row that no longer matches is therefore two different stories — "Haive wrote it and you changed
it" and "this was always yours" — and only the step record separates them. Getting this wrong
tells a person their own untouched file was "edited since Haive wrote it"; the skipped-file case
in `repo-artifact-reset.test.ts` exists for exactly that confusion and catches it.
Dispositions do NOT depend on which of the two refused: a file that fails the claim is
quarantined where it was quarantined and kept where it was kept. Only the REASON travels, using
the wording the settings pass has had since it shipped. That half is the point — with one flat
reason for both, a genuine user edit and a commit hook that reformatted what Haive wrote are
indistinguishable in production, and a gate nobody can measure is a gate nobody can tune.
07, `09_5-skill-generation` and `09_5b-skill-repair` all record hashes. `11d-skill-sync` does NOT,
and must not simply follow: it writes into the task's WORKTREE and those bytes reach the root
through a git merge with AI conflict resolution, so a mismatch there is evidence that a merge
happened, not that anyone edited anything. Its claims stay path-only.
**The skill tables are TOP-LEVEL and keyed by id, never carried on `written[]`.** 09_5's
`written[]` round-trips through `task_steps.iterations`, which is `jsonb`, and jsonb NORMALISES
object key order — so a hash nested in an element would land mid-object on every entry a later
pass carries forward, and could never be kept where the summariser's 4000-char slice cuts. Only
the top level of the returned object survives in declaration order, because that object reaches
`buildStepSummaryPrompt` in memory. `written` itself is declared LAST in 09_5 for the same
budget: at ~700 bytes per skill it alone fills that slice.
ONE hash covers every mirror directory in both steps, because `skillMd` and the sub-skill bodies
are rendered ABOVE the target-dir loop and neither renderer takes a directory. The README is the
exception and is keyed BY dir, since its render interpolates its own path — and in 09_5b it is
also rebuilt from that dir's own on-disk set.
**No claim is re-gated on a hash.** `sub-skills` still turns on the SLUGS, so an output recording
slugs but no hashes keeps exactly the claim it always had; re-gating would drop it for every
pre-existing output and change what happens to a file a person left in that directory.
Directories are claimed with `null` always, having no content to hash.
**A fixture for this is normalise-stable by default, so it pins nothing.** MEASURED twice in one
PR: removing `normalizeContent` from a SKILL.md hash left every step-side test passing, because
the fixture roughened only the sub-skill body. Roughen the OVERVIEW too — inside the text, since
`skillToMarkdown` trims its ends — and assert `normalizeContent(x) !== x` per file before
asserting what the hash is.
Two landmines to know about before extending this, both inert today:
`12-post-onboarding` inserts a row whose `diskPath` is `AGENTS.md` and whose `written_hash` is
the hash of the cli-rules BLOCK rather than the file — an entry in `writtenHashes` that can
never match its own path, harmless only because the sweep never visits a root file. And
`01b-install-plugins` runs CLI plugin commands against `.claude/plugins/drupal-php-lsp`, the
exact paths 07 wrote, so a CLI that ever normalises those JSONs makes them read as edited.
Rejected, so it is not re-litigated: having 07 insert artifact ROWS for the paths the manifest
does not cover would unify the two mechanisms, but a live row with no current rendering
classifies as `obsolete` in the upgrade plan and `02-upgrade-apply` then DELETES the file —
arming the upgrade path against the user's own custom agents. The step payload is the right
carrier.
**A partial reset therefore leaves a residue, and that is the accepted trade.** Where the walk
removed something but an I/O error skipped a generated directory, the epoch still advances —
`resetTouchedNothing` refuses only a run that touched NOTHING, since a second reset over an
already-clean tree legitimately removes nothing. `resolveKeptArtifactPaths` then keeps the live
artifact ROWS for the skipped paths, but the PATH-ONLY claims above are not rows: the epoch
excludes whole step rows by `ended_at`, so a retry reads those surviving files as unowned and
quarantines or keeps them instead of removing them. Both are reported. The two alternatives are
worse in the direction that loses data. Not advancing the epoch re-opens the hazard it exists for
— a stale `wroteFiles` claiming a file the user recreated by hand at a path the reset DELETED —
and exempting the skipped paths needs a new persistent per-path record, which is that same hazard
again with a longer life. So this is clutter, in the direction that loses nothing, the same
bargain `11d-skill-sync`'s exclusion already strikes.
**A live artifact row puts a directory in scope but never claims a file on its own.**
`recordOnboardingArtifacts` inserts one row per manifest RENDERING without consulting
`wroteFiles`, so the file apply SKIPPED has a row too, carrying the hash of what Haive WOULD
have written rather than what is on disk — claiming by row would hand the user's own definition
straight to the deletion the quarantine exists to prevent. The entry-level claim for a row is
therefore a hash check against the bytes on disk (`artifactMatchesDisk`), the same test the
settings files use and for the same reason. Rows still put the directory in scope, which is what
keeps an UPGRADED repo's dirs resettable: `02-upgrade-apply` writes through those rows and never
appears in any 07 `wroteFiles`.
**Inside a proven directory, what Haive cannot claim is MOVED, not deleted.** The quarantine
checkbox at 07 defaults OFF, on the stated grounds that an agent the user wrote by hand is
indistinguishable from one an older workflow left behind — so their own definitions legitimately
sit beside ours, and a recursive removal would take them. The reset therefore runs 07's own
mechanism first, to 07's own destination (`unmanagedAgentsDir`, the `-legacy` sibling a reset
then keeps), and removes the directory once only Haive's entries are left. `noReplace`, because
a name already quarantined is an earlier run's file and which of the two a person wants is not
ours to decide; an entry that could not be moved is reported and its DIRECTORY then survives,
with Haive's own entries removed individually around it. Agent ids for the claim come from the
template manifest rather than the step payload — `acceptedAgentIds` is the user's PICK, so an
agent they deselected would otherwise read as theirs and be quarantined out of its own
directory.
`.claude` is swept entry by entry and removes only what it can CLAIM — what 07's `wroteFiles`
names there (`workflow-config.json`, the slash commands, the Drupal LSP files) or what a live
row verifies by hash. Those claims keep their WHOLE path, never a head segment: the LSP plugin
is `.claude/plugins/drupal-php-lsp/<file>`, and collapsing that to `.claude/plugins` claims a
directory that also holds plugins the user installed. A `.claude` directory Haive wrote INTO but
does not own is WALKED on `hasDeeperClaims`, its claimed leaves removed, and dropped only once
nothing of the user's is left in it. A person's own `commands/`, `settings.local.json` or hooks live there too,
and the blanket removal this replaced took them. They are LEFT and reported rather than moved:
`.claude` survives the reset anyway, so there is nothing to move them out of the way OF, and a
`-legacy` sibling of it would be noise. Three of its entries are kept by name for their own
reasons:
`mcp_settings.json` (created once, never rewritten), any `*-legacy` quarantine (the user's own
agent definitions, which 07 MOVED there), and a `settings.json` whose bytes do not match the
live artifact row's `written_hash` — `writeIfAllowed` SKIPS an existing file, so that row is the
only evidence Haive wrote the one on disk. Each kept file is REPORTED through the same `skipped`
channel a refused link uses; the reset says what it left alone rather than passing it off as
reset. The repo's live `onboarding_artifacts` rows are superseded in the same request, after
those provenance reads: rows naming deleted files must not stay live, and `12-post-onboarding`
inserts without conflict handling, so a re-onboarding would otherwise collide with the
`(repository_id, disk_path) WHERE superseded_at IS NULL` unique index.
**The legacy RTK.md files are taken back on the same evidence.** From e4ea9a61 to bd9b94c0, 07
wrote the RTK body to `RTK.md`, `.gemini/RTK.md` and `.claude/RTK.md` and recorded no hash for
them. A claim for one therefore carries `LEGACY_RTK_MD_SHA256` (`@haive/shared`), the body as it
was written, frozen so a re-vendored `RTK_SLIM` cannot move it. The two outside `.claude` get a
pass of their own, since no directory pass reaches them. Content alone never claims: a file
holding the same bytes with no 07 record is kept and reported, and one edited since is kept as
edited. An edited `.claude/RTK.md` used to be deleted on the path-only claim.
**A claimed file is removed only while it still holds the bytes that made the claim.** The verdict
is read before the removal, so a save landing between the two was deleted as Haive's. Every file the
reset removes on its own, the settings pass included, goes through `removeFileIfNoFollow` against
the row's or the step's hash, as an upgrade's delete does; one that no longer matches is kept and
reported as edited, and its directory stays around it. A file past the read cap never matches: the
part read can normalise to a render the whole file is not. A file a save put at the name while the
old one was judged is kept too, though the old one goes. One whose name a save took while it was judged and
refused cannot go back, so it stays under its private name (`ParkedFileError`, carrying the errno)
and the reset reports where it is and carries on. That is no failure to read the tree, so it never
counts toward the floor below that refuses a walk which touched nothing. A claim with no hash behind
it is still taken by its path, and so is everything in a directory the reset removes whole: a save
into one of those at that moment can still be lost.
Two consequences are refusals, and only two. A SECOND onboarding task on a repo that has a live
one is a 409 at `POST /tasks` — two runs write the same `.claude/` files, the same KB and the
same scope list, so it is a corruption path rather than a queue. Everything else stays the
caller's call: a `workflow` or `run_app` task on a mid-onboarding repo is allowed by the API,
and only the New Task form declines to submit one (its inferred type there is `onboarding`).
Same stance as `computePlanReady` — a rule strict enough to CHOOSE must not REFUSE.
**FOUR writers can touch a repository root, and the live-task read guards one of them.** The
survey that established this is worth not repeating:
1. **A revived task job** writing while the walk runs. Still open, deliberately. Two gate designs
were built and abandoned: one on the task queue (`task-queue.ts` is concurrency 5 for the whole
INSTALLATION, `worker/src/index.ts` force-closes with `close(true)`, and a gate throw inside the
processor's `try` reaches `markTaskFailed`), one on the repo queue (below). Neither buys much —
between job pickup and `markTaskRunningWithStep` every branch ends in a task-status write and
none writes a path the reset deletes, so the window carries no data loss.
2. **The repo queue**, which `rm -rf`s the root in THREE handlers (`clone.ts` copy, init and
clone), reached only when a repository is created or its creation is retried: `refresh-tree`
fast-forwards instead (below). Closed by the claim below.
3. **`stampRepositoryOnboarded`**, which no lock can reach: it runs from `markTaskCompleted` AFTER
`completed` is committed, so every live-task guard has already stopped seeing that run.
4. **`PUT /tasks/:id/files/content`**, whose `EDITABLE_PREFIXES` are `KB_DIR` and `LEARNINGS_DIR`
— exactly what the reset removes recursively — with no task-status check at all, because it
exists to review knowledge gates on failed and completed tasks.
**`repositories.root_claimed_at` is a CLAIM, not a lock, and it is RECIPROCAL.** Both the reset
and the three repo-queue handlers that `rm -rf` the root take the same claim
(`claimRepositoryRoot`, `'reset' | 'rebuild'`), so whichever arrives second is refused in either
direction; `refresh-tree` and the knowledge-file editor refuse while it is held. The handlers HOLD
it across their destructive work rather than checking once before it — a one-directional check
leaves the window where a handler has already passed it and the reset claims the row before
`rm(dest)` runs, and both then walk the same tree. `root_claim_kind` exists only so a refusal can
name what it is waiting for; nothing branches on it.
`DELETE /repos/:id` refuses while a claim is live, reading it on the repository row it locks: a
delete during a clone used to let the resource cleanup it queues run before the clone finished
writing, which left the checkout on disk with no row to find it by. A writer that claims after the
lock waits for the delete's commit and then finds no row, so it writes nothing.
**It is a LEASE, not a deadline on the work.** `acquireRootClaim` renews while its caller works
(`ROOT_CLAIM_RENEW_MS`, a third of the window, on an `unref`ed timer so a held claim never keeps a
process alive). A fixed expiry cannot tell a dead holder from a slow one, and both exist here —
`gitClone` has no timeout and `copyTree` is unbounded by repository size — so expiring a live
holder re-admits the concurrent `rm -rf` the claim exists to exclude, i.e. the race returning at
the fifteen-minute mark. With renewal, expiry means "the holder stopped renewing", which is what
abandonment actually is. Use `acquireRootClaim`, not the bare `claimRepositoryRoot`, for anything
that is not certainly shorter than the window.
**Renewal ends at `release()` and NOWHERE else.** An elapsed-time cap was added here and reverted,
so do not re-add it: any deadline eventually expires a holder that is still rewriting the tree,
which is the catastrophe above with extra steps. The worry it answered — a handle never released,
renewing forever — is the lesser failure by every measure: it needs a caller to skip the `finally`
all of them have, it is visible because the repository refuses and names what holds it, and it
ends at the next process restart since the timer is `unref`ed. `root-claim-lease.test.ts` fails if
a cap comes back.
The claim is released with the STAMP it took, and renewal is conditional on it too: a holder whose
lease DID expire and was taken over learns it lost rather than clawing it back or clearing its
successor's claim — silently, and exactly on the slowest trees, which are the ones that reach the
window at all.
The knowledge-file editor (`PUT /tasks/:id/files/content`) HOLDS one for its write, as kind
`edit`. Checking is not enough there: the SELECT can finish just before a reset claims, and the
write then lands in a tree being recursively removed — resurrected as an orphan, or written to an
already-unlinked inode, which returns 200 and silently loses the edit.
It is taken only for a write that lands in the ROOT — `root === anchor`, i.e. the task has no
worktree. Most knowledge edits are on a workflow task, which does have one, and the reset never
touches `.haive/worktrees/`: its targets are the catalog directories, `KB_DIR`, `LEARNINGS_DIR`
and `.haive/install.json`. Claiming for those would refuse an ordinary edit whenever a reset ran,
and refuse a reset whenever anyone was editing, over a collision that cannot happen.
Where it IS taken the claim is per-REPOSITORY, so two root-level knowledge saves in one repository
at the same instant refuse one another with "try again in a moment". Accepted deliberately: the
hold is one file write long, and a per-file lock would not exclude the reset, which is repo-wide
by nature. A task with no repository takes nothing, since there is no root to protect.
A lock must be HELD across the work it protects, and both places to hold one cost more than the
race: the repo worker and the task worker run in ONE process on ONE `max: 10` pool, so a repo job
holding a connection across `rm -rf` + `copyTree` at concurrency 5 deadlocks the pool; and
`repo-queue.ts` keeps BullMQ's 30s `lockDuration`, so a handler blocked past it can be failed as
STALLED without running its catch, stranding `status = 'cloning'` until the boot reconciler in
`data-migrations.ts` releases it (`maxStalledCount` 10 makes that rarer, not impossible). A
`repo_status` enum value
expresses the same claim and was refused because Postgres cannot drop an enum value: an
irreversible migration for a reversible problem. The cost of a row over a lock is that a writer
killed mid-job leaves it set, which `ROOT_CLAIM_STALE_MS` bounds — generously, since expiring
early re-admits the `rm -rf` the claim exists to exclude.
**Absorbing an error means no count taken afterwards is authoritative, and that is where the two
worst bugs on this branch lived.** `guard` reports WHAT it absorbed, and two decisions read it.
`mayRemoveSweptDirWhole`: a top-level directory may go whole unless the sweep hit an `io` failure
— its `left` is then still the initial zero, and removing on that deletes the definitions the
quarantine exists to move aside. `sweepSurvivors`: a NESTED sweep that did not complete reports a
survivor, or the outer `.claude` sweep adds its unfinished zero to its own total and removes
`.claude` whole, taking the user's plugins. The two differ on `refused` deliberately — at the top
level the refused path IS the removal target and removing the LINK is right, while nested it is a
child whose parent would be taken with it. Both are pure, exported and mutation-checked, because
neither difference is reachable from a fixture: a containment refusal is handled inside the sweep,
and a genuine `EIO` cannot be provoked from a temp directory. The other five guarded sites ignore
the result safely — the settings pass leaves its file on disk, so the later sweep counts it as
kept, and the rest only report or remove.
**A filesystem error inside the walk is a per-item outcome, not a route failure.** It used to
rethrow, which aborted the route BEFORE the supersede and BEFORE the epoch stamp — files deleted,
rows live, no epoch. A bad disk reaches that, and so does writer 4: `removeChild`'s final rmdir
rethrows everything but `ENOENT`/`ENOTDIR`, so a file created under a directory being removed
raises `ENOTEMPTY`. Errors now become `skipped` entries carrying their errno; anything that is not
a filesystem error still propagates, because a `TypeError` is a bug. The one tail that needs its
own rule is a walk that did NOTHING and hit an IO failure: superseding and stamping over an intact
tree is UNRECOVERABLE, since the epoch excludes every prior step row from `loadProvenanceSteps`
permanently and the next reset can then claim nothing, remove nothing and keep everything while
the repo still reads onboarded. That case is reported as a failure and touches no rows. The
predicate is "an error occurred", never "nothing was removed" — a second reset over a clean tree
also removes nothing and stays a no-op.
**Every piece of "onboarded" evidence is read against the epoch, not just the column.** Guarding
`onboarded_at` alone was cosmetic: the completed onboarding task row lives forever, so
`hasCompleted` kept answering yes across a reset. `resolveOnboardingVerdict` therefore requires a
completion NEWER than `onboarding_reset_at`, and "no run was ever started here" stops counting
once a reset exists — a reset IS a run started and taken back. `mark-onboarded` gets its own rule
rather than that comparison (it has no task to compare against) and refuses while a reset is
unanswered, because it is the documented escape hatch for a repo whose markers are on disk, which
is precisely what a reset that could not read the tree leaves. Its UPDATE also CARRIES the epoch
it validated (`IS NOT DISTINCT FROM`, so the null case is covered) and refuses under a live root
claim: validating and then writing unconditionally is the same read-then-act shape the claim
exists to remove, and a reset landing in between would have its work undone by hand. With no
epoch every term is byte-identical to what it was, which is why none of this needed a backfill.
Left alone deliberately: `landPlanMerge`'s `git merge --ff-only` in the root, which refuses rather
than overwriting local changes and only touches paths differing between HEAD and target.
**A committed MCP server list is not consent on another install.** `12-post-onboarding` commits
`.haive-data/tooling.json`, and `loadUserMcpServers` hands its `mcpSettingsJson` to every CLI, so a
clone used to run whatever servers the repository named, past the explicit opt-in 04 asks for
(`keepRepoMcpServers`). 04 records the list it writes, the servers that opt-in keeps included,
since the runtime and 07's rewrite read the record and never the file, and refuses the opt-in
once the file no longer holds the definitions its form named (`mcpServersFingerprint`).
`importHaiveDataMirror` now moves a list holding anything but Haive's own
definitions to `importedMcpSettingsJson`, which nothing reads, and the tooling page asks: accept
puts it back, discard drops it. A server counts as Haive's only when its definition EQUALS Haive's
under that name (`mcpServersNeedingConsent`), since a file can put any command under
`chrome-devtools`. The three keys (`ONBOARDING_TOOLING_CONSENT_KEYS`) are this install's decision:
stripped from an incoming mirror and never committed. Acceptance is an HMAC of the list keyed with
the install's encryption key (`mcpAcceptanceMark`), never a plain hash: an older release's import
stored the committed tooling verbatim, so a file could carry a matching hash of its own list.
`holdImportedMcpServerLists` holds such rows at boot. A record this install's own 04 wrote equals
that run's `output.tooling`, so only one matching no local 04 run is held, and the mark keeps an
accepted list from being held again.
**Refresh brings a checkout up to its source and never deletes one.** `POST
/repos/:id/refresh-tree`, which the repos page offers as Retry on a repository in `error`, used to
enqueue the clone or copy job, and both remove the root before anything else: MEASURED, a clone
holding one unpushed commit lost it, and every task worktree under `.haive/worktrees` went with the
tree. `handleRefresh` (`repo/refresh.ts`, claim kind `refresh`) fetches the checkout's branch from
its source and runs `merge --ff-only`. The source is `origin` with the stored credential or, for a
writable folder import, the host folder itself (the user's call: that copy carries Haive's own
onboarding output, and the folder often has no remote). It refuses, saying why beside a `ready`
repository, while a task holds the checkout (`CHECKOUT_HOLDING_TASK_STATUSES`, shared with plan
Pull), when the checkout has commits the source lacks or no history in common, and on a detached
HEAD; git itself refuses a fast-forward that would overwrite a local change. The task check is read
once, under the claim, and nothing that starts a task reads the claim, so a task that starts during
a refresh is not fenced out: the window writer 1 above leaves open for the reset. It loses nothing
here either, since the fetch writes no working tree, the fast-forward stops at a change git sees and
the rebuild renames; what it can cost is a git lock collision that fails one side, which a Retry
clears. Only a checkout that is not usable (no `HEAD` at its own top level) is rebuilt, and whatever
stood there is renamed to a `<repoId>.aside-<time>` sibling first, never removed. A read-only folder
import is re-scanned as before, and a repository with no remote and no folder answers 409 without
touching its row.
## Onboarding template versioning
Deterministic onboarding artifacts (agent specs, slash commands, `workflow-config.json`, Drupal LSP plugin files, the `agents/README.md` index) are registered as `TemplateItem`s in `packages/worker/src/step-engine/template-manifest.ts`. Every item has:
- `id`: stable slug (e.g. `agent.code-reviewer`, `command.review`, `workflow-config`).
- `schemaVersion`: integer, bumped only on shape-breaking changes (filename change, new required frontmatter field).
- `contentHash`: sha256 over the rendered reference output, computed on worker boot from `REFERENCE_CONTEXT` and cached per-manifest.
- `render(ctx)`: invokes the existing generator and returns `TemplateRendering[]`; multiple renderings per item when an agent fans out across CLI target dirs.
On worker boot, `syncTemplateManifestCache(db)` upserts the manifest into Postgres (`template_manifest_cache`) so the API can compute the current set hash without importing worker-side generators. Per-repo install state lives in `onboarding_artifacts`, one live row per `(repository_id, disk_path)`, soft-deleted via `superseded_at`.
**An item `REFERENCE_CONTEXT` renders empty carries its own `referenceCtx`.** A hash of nothing
never changes, so the banner could never see such an item's body change: the three PHP LSP plugin
files and the two RTK settings files, gated on PHP LSP and on RTK, hashed `sha256('')` from the day
they shipped. Each now renders in a context that selects it. `agents-index` stays empty on purpose,
since its body is per repository and the upgrade plan compares it per repository. Rows recorded while
an item hashed empty still carry that hash, so `syncTemplateManifestCache` converges them in the
transaction that moves the cache (`convergeReferenceHashes`): a row whose written hash is today's
body takes the new hash and reads as current, and one holding an older body keeps the empty one, so
the banner offers the newer.
### When changing a template
1. **Body-only change (rewording an agent prompt, fixing a typo, updating a command example):** edit the generator in `_agent-templates.ts` / `07-generate-files.ts`. The manifest's `contentHash` recomputes on worker boot and the upgrade-status endpoint starts reporting the template as changed. **Do not bump `schemaVersion`.**
2. **Shape change (rename agent id, change `workflow-config.json` schema, change a command's disk path):** bump the item's `schemaVersion` in `template-manifest.ts`. Rollback across a `schemaVersion` bump restores the prior artifact's stored bytes (migration 0013) and reverts correctly; only legacy rows written before stored content existed refuse to revert across a bump (they have no prior bytes and re-rendering a changed shape is unsafe).
3. **New template:** add the generator, append to `buildTemplateItems()` in `template-manifest.ts`. First upgrade per-repo will surface it in the `new_artifact` bucket.
4. **Removed template:** delete the `TemplateItem`. First upgrade per-repo will surface existing artifacts in the `obsolete` bucket.
Out of scope of onboarding-upgrades: `.claude/skills/` and agents written from LLM discovery (06_5) without a bundle source, `.haive-data/knowledge_base/`, `.claude/mcp_settings.json`, `.claude/onboarding-review.md`. **Skills and agents installed from custom bundles are tracked and upgradable** (each bundle item lands as an `onboarding_artifacts` row with `templateId = "custom.<bundleId>.<itemId>"`; the `bundle_resync` step before `01-upgrade-plan` refreshes git bundles and the upgrade-plan/apply path treats `custom.*` rows the same as Haive templates). LLM-generated skills and KB content are refreshed by the `/workflow` code-change phase. Project knowledge lives under `.haive-data/` — `knowledge_base/` and `learnings/` — not under a vendor dir, because Haive is model-agnostic; the canonical paths are `KB_DIR`/`LEARNINGS_DIR` in `packages/shared/src/knowledge-paths.ts`, and `stripManagedKnowledgeGlobs` makes them impossible to scope-exclude from the folder pickers (06_7/09_7, the api exclusions endpoint, and the repos-page editor). `mcp_settings.json` is user-owned — created on first onboarding, never rewritten.
**An upgrade also restores the rules delivery 07 set up outside the manifest.** `02-upgrade-apply`
re-creates each currently enabled import-mode provider's `@AGENTS.md` stub (`CLAUDE.md`,
`GEMINI.md`) through `ensureRulesImportStub` (`steps/onboarding/_rules-files.ts`, the helper 07
itself calls), and records a refused link per file instead of throwing, since a throw there would
skip the applicable-template and install-manifest writes after it. Without the stub a
claude-family CLI never loads AGENTS.md, rules block included, and nothing else ever wrote it back.
`GET /repos/:id/upgrade-status` checks the same files through the same module
(`@haive/shared/rules-files`, with each provider's `rulesFile` in the catalog), so a repository
whose only gap is a missing import still gets the upgrade banner. A rules file linked anywhere but
AGENTS.md is reported apart (`linkedRulesFiles`) and never offers an upgrade, since the restore
refuses to write through a link, and a repository root the api cannot read claims nothing.
`03-upgrade-commit` stages the paths 02 reports writing (`writtenPaths`) beside its base list,
because a workflow task checks out HEAD: a refreshed block left uncommitted never reaches one. It
also stages each stub 02 left holding the import whose HEAD copy lacks the line
(`headLacksImport`), and a `CLAUDE.md -> AGENTS.md` link 02 left alone. It stages AGENTS.md itself when
its cli-rules region on disk is missing from HEAD or differs from HEAD's (`agentsRulesVerdict`):
only the region is compared, so an edit elsewhere in the file never triggers it, and a file it
cannot compare (a link, one past 1 MiB) is reported rather than guessed. A write list alone misses
two cases: 02 reports `unchanged` for a stub an earlier attempt of the step wrote before failing,
and for one onboarding wrote with its commit off, so both stayed out of HEAD for good. The HEAD
check greps the blob rather than reading it back, since a read-back is capped by the child
process's output buffer and a file past it would read as "missing"; a check that fails for any
reason other than a missing HEAD stages nothing. It never stages rules files by name beyond that, so a person's uncommitted edits
elsewhere stay out of the upgrade commit; a stub that IS staged goes in whole, the same file-level
granularity AGENTS.md already had. A link is staged through its own check (`isLinkToAgentsMd`),
since `hasWorkspaceEntry` refuses every link. A rules file or an RTK settings file
(`RTK_SETTINGS_PATHS`) git ignores is left out on every route, AGENTS.md included, and reported
(`dropIgnoredRulesFiles`), and `12-post-onboarding` applies the same filter to what it stages: the
stage runs `git add -f` for `.haive/install.json`'s sake, which would otherwise commit a personal
CLAUDE.md or settings file the repository deliberately keeps out of history. 12 stages a settings
file only through its live artifact row, never by name, so one Haive did not record stays out. The
check runs after any `git init`, since before it there is no repository to ask.
**The cli-rules row records the region on disk, not the render.** 07 leaves an existing region
alone unless its overwrite flag is on, so `12-post-onboarding` reads the region back and records
it through `cliRulesRegionRecord` (`steps/onboarding/_rules-files.ts`): the region's bytes and their
hash as `templateContentHash`, so a stale region shows as an upgrade and a rollback restores bytes
that were really there. The row claims those bytes as Haive's (`writtenHash` equal to that hash)
only when they are a render, this one or one an earlier `onboarding` or `upgrade` row of the
repository recorded; a `backfill` or `rollback` row does not count, since its hash is whatever was
on disk. Anything else keeps the render's hash, so the upgrade plan offers the region as a
conflict, whose default is skip. No region on disk means no row, and a live one is retired, since
it would read as the person's deletion. `01-upgrade-plan`'s backfill applies the same rule to the
region: it used to store the whole file, which a rollback could paste into the region.
**Every backfill row claims only a render, whatever its kind.** The backfill used to record the
disk's hash, so an edited file became its own baseline and the next template change pre-selected
overwriting it as a `clean_update`. A path with no row whose bytes no render accounts for (this
render, or for the rules region one recorded earlier) is now a `conflict`, not a pre-selected
`new_artifact`, and on a repository's first upgrade every path has no row. 01's backfill records
nothing for such a path: nothing there is Haive's until 02 writes it, and a row would belong to the
upgrade with no prior, which a rollback reads as a file the upgrade introduced and deletes. 02
records the path only when it replaces the file, keeping what it held as a superseded baseline (the
rules region through `cliRulesRegionRecord`, any other file through `backfillRecord`). It keeps one
for a live row too when the bytes on disk are not what that row records, or a rollback of an
Overwrite would restore Haive's old bytes over the person's edits; 02 retires the row and the
baseline in one instant, and the baseline, written after the row, wins 04's `generated_at` tie. Both apply
one rule to bytes that are not a render: the bytes are `writtenContent`, so a rollback restores
them; the render's hash is `writtenHash`, so they are never taken as Haive's; and their own hash is
`templateContentHash`, so the template reads as not installed. A rollback copies both hashes, so a
restored file is offered again at the next upgrade rather than classified `unchanged`.
`GET /repos/:id/upgrade-status` keeps one row per template, and a rendering that is not current
stands for it, so such a file keeps the banner up beside current siblings. A path with no row at all
is invisible there once a sibling rendering has one; the next upgrade still offers it.
**A file already holding the new render is recorded, never asked about.** A live row whose
template moved while its path already holds the current render (one another install upgraded and
git brought here, or one a person brought up to date by hand) is `adopt`. It used to read as a
`conflict`, since the bytes matched neither the row's baseline nor its render, and asked the person
to choose between two identical files. 02 retires the row and inserts a `backfill` row claiming the
render, writing nothing, and outside `writtenPaths`, `createdPaths` and `retiredRowIds`: 03 stages
nothing for it, and a rollback, which restores only what the upgrade replaced, leaves it. The bytes
are read again at apply, since the form parks between the plan and the apply, and a file changed
meanwhile is left for the next upgrade; a retry finds the row already retired and records nothing
more. A form shown for anything else names the adopted paths in a note, and an upgrade with nothing
else to ask runs without one.
**An upgrade or a rollback deletes a file only while it holds what Haive wrote there, judged on
the bytes it deletes.** `removeIfHaives` (02, shared by 04) hashes them the way the plan compares
them (the whole file, or the rules region alone) against the row's `writtenHash`, at APPLY time,
because the form parks between the plan and the apply. A file is judged on the inode it takes:
`removeFileIfNoFollow` (`@haive/shared/fs-safe`) moves it to a private name in its directory first,
so a save landing meanwhile makes a new file the delete never touches, and puts a refused one back
without replacing a file written since. The rules region is stripped the same way
(`rewriteFileIfNoFollow`): rewritten on the parked inode and put back, since a strip over the file at
its path overwrote a save that landed between its read and its write. A file with no region, or no
AGENTS.md at all, is left as it is; the strip used to write that absence as an empty AGENTS.md,
which 03 then committed. Only a writer that already held the file open can still land between the
judge and the act, and nothing short of a lock it takes too can exclude one. A link or a directory
standing there is kept the same way, since neither is what Haive
wrote; only a read that could not run fails, which leaves the row for the next attempt. A row
alone proves nothing:
`12-post-onboarding` records one even for a file 07 skipped. 02's obsolete removal keeps such a file,
warns, and leaves its row live. 04's undo of a file the upgrade introduced keeps one edited or replaced
since, warns, and retires the upgrade's row all the same. 04 also reads only the rows the upgrade WROTE
(`source = 'upgrade'`). 01's backfill rows belong to the same task, but they record what was already
there, and read as new files they were deleted: on a first upgrade's rollback, that took every
adopted file the person had declined.
**Every upgrade read stops at one cap, and a path it did not read is never taken for nothing
there.** `readUpgradeFile` (`@haive/shared/rules-files`) reads no further than
`RULES_FILE_READ_CAP` (1 MiB) and answers `absent` only for a missing path: past the cap, a link,
anything but a regular file and a read that fails are `unread`. The plan used to read a link as
absent, offer a new file there pre-selected, and the write then threw and failed the whole apply.
01 now marks such an entry `unread` under `UNREAD_HASH`, which matches no record and reads neither
as deleted nor as new, and 02 offers no choice on it: the form names it, with why, in one note, and
the apply skips it whatever the form sent. The form parks between the two, so 02's writes,
removals and hook strips check the cap again on the bytes they touch, and a removal it could not
compare says so rather than that the file was edited. 04 leaves a path it cannot read where a
removed file or region would go back, and says so; a link in the way there used to throw and fail
the rollback. upgrade-status reads the RTK settings files, and 01's rules-import note names only
the stubs the apply can add a line to, the same way, so neither offers what the apply refuses.
**A rollback puts back what stood before, absence included, from what 02 recorded.** Reading it off
the rows failed three ways: a file missing before the upgrade came back from the row that recorded it,
or from a backfill row written while it was missing; a row an old reset superseded was restored over
bytes it never held; and a file already holding the render with no row was deleted as new. So 02
records every path it wrote into nothing (`createdPaths`: no file, or for the rules region no region,
with the live row it retired there) and every row it retired or kept as a baseline (`retiredRowIds`).
04 removes a created path under the check above, an AGENTS.md it created for the region whole while
nothing else was written to it, and puts the retired row back live, so a reinstated file reads as
deleted again. Any other path restores the newest row the upgrade retired there, and a path with none
held the bytes the upgrade wrote, so only its row is retired. 01's backfill records no row for a file
missing from disk. An output from before the record falls back to reading the rows, and only those
retired no later than that upgrade completed: a later upgrade that failed part-way leaves its
records at the same paths.
**02 records what stood at a path before it writes there, so a retried apply records what the first
attempt did.** The record is its own statement ahead of the write, never part of the transaction
that retires the rows, since an apply that fails after writing leaves paths that now hold the
render and show nothing of what stood there. A retry finding the render takes the newest record an
earlier attempt of the step made for that path. Every write gets one: the bytes that stood, recorded
as the row that records them when they match it (its hashes, snapshot and schema version, so
restoring it restores what restoring that row did), or a marker where nothing stood, which carries no
observed hash and, for the rules region, `''` where the file stood without the region. Only a path
with no row that already holds the render records nothing. A marker names its path in
`createdPaths` and is never in `retiredRowIds`. It takes the instant the transaction retires a row
at its path, which is how a retry tells the live row the failed attempt retired where nothing stood
from one something else retired since. An earlier attempt's own rows hold what it wrote, so a retry
drops them from what it reports retired.
A restore replaces only what the upgrade left there (`restoreIfUpgrades`): the file, or the rules
region, is judged on the bytes it replaces against its upgrade row's `lastObservedDiskHash`, else
its `writtenHash` (a hook strip claims none of what it left), read by id at apply so a payload
detected earlier still judges. A file or region edited or removed since is the person's and stays,
reported, and one that cannot be compared (a link, past 1 MiB) stays too. Its ledger reverts all the
same, a `rollback` copy of the row before marked `userModified` with what the disk held, so the next
upgrade judges the file against the version before; it is not counted as reverted. One already
holding the restore is an earlier attempt's, and counts. A save that takes the name while it is
judged keeps it, and the warning names where the parked bytes went.
A path the upgrade REMOVED gets its bytes back the same way. The upgrade wrote no row there, so 04
used to visit only the rows it wrote and never put back an obsolete file it deleted, and a file no
row recorded had nothing to restore from. 02 now keeps what each removal took (the file, or the
rules region with its markers) as a baseline and names the path in `removedPaths`, and 04 puts it
back only where nothing stands now: a file created at the path since, a region written into
AGENTS.md since, or an AGENTS.md removed since is what someone did after the upgrade, and stays.
That baseline is written before the removal it records, from the bytes the plan saw there, because
an absent path says nothing about who removed it: an apply retried after its removal ran finds the
file gone, and so does one whose file a person deleted while the form was parked. Only the first
finds an earlier attempt's baseline, and records the removal; the second drops its own, so a
rollback leaves the file deleted. A removal that keeps the file drops its baseline too. What
already holds the restore is an earlier attempt of the same rollback, which a retry takes as put
back. Such an upgrade, like one that only took RTK blocks out, can leave no live row at all, so
upgrade-status offers its rollback from what 02 recorded (`lastUpgradeRemovedContent`) until a
rollback completes after it. It answers as
onboarded for any repository an upgrade was started on, whether that upgrade is running, failed or
finished, since POST /tasks starts one only on an onboarded repository and the banner shows nothing
for any other. What a rollback puts back stays in the repository's
applicable set whatever its snapshot renders, so a file restored after RTK went off is offered for
removal again rather than dropping out of the banner's view.
**One upgrade or rollback runs at a time, and a rollback undoes the newest upgrade once.** Both are
`onboarding_upgrade` tasks, and two side by side apply and revert the same files: an upgrade parked
on 02's form applied over a rollback that ran meanwhile, and a second Roll back click queued a
second revert of the same upgrade. POST /tasks and the rollback route insert either through
`insertUpgradeTask` (`routes/upgrades.ts`), which refuses with 409, naming the live task, while
another one of the repository is live, under a per-repository advisory lock so two clicks cannot
both pass the check. The rollback step reverts the newest completed upgrade, so the route accepts a
rollback only while that upgrade is the newest completed task of the two kinds
(`latestUpgradeToRollBack`): after a rollback nothing is left to undo until another upgrade
completes, and upgrade-status offers no rollback either. The banner offers the live task
(`inProgressUpgradeTaskId`) whether or not drift is left to review, where "Continue upgrade" used to
start another, and hides Roll back while one runs. A check at creation cannot see a Retry of one
that failed, which revived it beside the next, so the rule is also an index
(`tasks_one_live_upgrade_per_repo_idx`, migration 0168): one live `onboarding_upgrade` task per
repository, whoever writes the status. The api answers its violation 409 in `errorHandler`,
whichever route revived the task. The worker revives one whose parked form is answered, and a
refusal there would drop the answer, so the submit route refuses such an answer before storing it; a
revival the worker still loses to a race reads as the task not pointed and records
`upgrade.revive_refused`. Its statuses are the api's `LIVE_TASK_STATUSES`, pinned by a test. Both
create routes write the task, its event and its move to `queued` in one transaction, since no sweep
starts or ends a `created` task, and one left by a failure part-way would block every later upgrade
and rollback. The migration fails the ones an older release left that way before it builds the
index, and then keeps each repository's newest live one.
**Switching RTK off reaches the upgrade.** 01's render context takes the repository's live
`rtk_enabled` wherever the context recorded a choice. One from before RTK recorded none and stays
off, since the column defaults on. The RTK settings files (kind `rtk-config`) then read as
`obsolete` and go only under the rule above, and 03 records each removal (`deletedPaths`, `git rm
--cached` while the path is still absent), or HEAD would keep the hook and every worktree checked
out from it would restore it. Nothing records the ones a blank scaffold seeds (a row needs a task,
and INIT has none), nor any of a repository with no rows at all, so for those 01 renders the RTK
templates as if RTK were on, for every CLI in the catalog since the scaffold seeded them for the
CLIs enabled then, and offers a file still holding that render, or its hook, as `obsolete` against
the render's hash. A blank repository is probed whatever other rows it has, skipping only the
paths a live row records: onboarding records one for a seeded file only when it renders that
template itself, with RTK on and the file's CLIs still enabled. Any other repository's rows are
taken as the whole record. One someone edited is kept by that rule, so the form offers instead to
take the hook out of it, unticked (`withoutRtkHookEntry`): only hook items whose command is exactly
RTK's go, with the entries and lists they leave empty, and the file keeps its own indent and line
endings. A file that parsing and writing back would change anywhere else (an integer past 2^53,
`1.0`, an escape, a repeated key, spacing of its own), or one nested too deep to write back at
all, is not offered. One that is not valid UTF-8 is kept at apply and says so, since writing its
text back would put U+FFFD where the bytes it could not decode were. What it held becomes a superseded baseline and the new live row claims none of it
(`writtenHash` stays the render's), so a rollback puts the hook back and no later upgrade or reset
takes the file as Haive's. A retry after an attempt that took the hook out and failed before
recording it finds the file holding what the plan's bytes strip to, and records that edit with the
plan's bytes as what it held. A plan whose RTK choice was the repository's live one
(`rtkFollowsLive`) is refused at apply once RTK is switched again, since the form parks between the
two, and retrying the plan step plans it afresh. The "off" synthesized for a context from before
RTK was nobody's choice and is never compared. The RTK block is no manifest item, so 02 takes it
out of AGENTS.md, CLAUDE.md and GEMINI.md by its markers, as a reset does, with the newline 07 wrote
after it (`stripRtkBlocks`). A link is refused and reported, as is a file past the 1 MiB cap the
plan reads with or one that is not valid UTF-8, and a `CLAUDE.md -> AGENTS.md` link is left to
AGENTS.md's own pass. Each strip records what the file held before it changes (`rtkBlockStrips`),
in a superseded row kept apart from the rows a rollback restores (`RTK_BLOCK_RECORD`), and a retry
that finds the block gone takes the record an earlier attempt made while the file still holds what
that strip left: a save that took the name while the strip was parked is not its doing. A rollback puts the blocks back
ahead of the rules region, whose restore would move the file off what the strip left, and only
while no RTK block stands in the file: all of what the file held while it still holds what the
strip left, else the blocks appended the way 07 appends one. The same blocks standing there count
as put back; another block, or a file removed since, is kept and named. The `@AGENTS.md` stubs 02
restores stay, since without them a claude-family CLI never loads AGENTS.md. 03 keeps a stripped
file git ignores out of the commit, whichever provider it belongs to. 01 names the files holding a
block, so the form says what will change. `GET /repos/:id/upgrade-status` reads no RTK settings template as current for such a
repository and reports the files still holding a block (`rtkBlockLeftovers`), and the settings
files no row records that 01 would offer for removal (`rtkSettingsLeftovers`): it looks where 01
looks, reads RTK's choice through the same snapshots, and judges each file by the predicate 01
uses (`holdsRtkSettings`), so the banner offers the upgrade, and never one whose plan offers
nothing. A repository never onboarded gets neither, since POST /tasks refuses it an upgrade.
Switching RTK back on offers the settings files again, and the banner says so: the
upgrade that switched it off left their ids out of `applicable_template_ids`, so upgrade-status
counts an RTK template as applicable again while RTK is on and the providers of a snapshot that
recorded the choice read its file (`RTK_SETTINGS_READERS`, `@haive/shared`, the list 07 renders
from too). 01 renders from such a snapshot ahead of one from before RTK (`pickRenderSnapshot`), and
both take the newest (`newestArtifactsFirst`, `@haive/shared`), since an upgrade replaces only the
paths it writes and rows from several runs stay live side by side, so the banner and the plan read
the same one, and a repository whose snapshots all predate RTK stays quiet. Only a re-onboarding writes the block.
**"Keep my edits" is a decision; Skip is not.** Both leave the file alone. Keep also records the
version declined, so the next upgrade offers only a newer one. On a live row it moves
`templateContentHash` in place. An untracked path gets a `backfill` row whose `writtenHash` is the
render, so it claims nothing, and a rollback, which reads only `upgrade` rows, never takes the file
for one it introduced. Skip records nothing, so the path is offered again.
`unclaimBackfilledEdits` (`data-migrations.ts`) brings the rows written before into that shape: only
such a row has `user_modified` with `written_hash` equal to `last_observed_disk_hash`, and it and its
rollback copies get the two hashes swapped.