# Haive Deterministic multi-CLI orchestration and AI agentic workflow utility. Reimplements a legacy markdown-driven Claude Code workflow, the autonomous `/workflow` implementation loop, and a sandboxed local environment replication step set as a deterministic web project. Agentic CLI invocations only happen for parts that genuinely need reasoning. Everything else runs as TypeScript step modules with web forms. The legacy markdown step content has been ported into TypeScript step modules under `packages/worker/src/step-engine/steps/`. The original source archive is no longer vendored in the repo. ## Stack - pnpm workspace monorepo, turborepo - Node 26, TypeScript 5.7, ES2024 target, NodeNext modules - Hono 4 REST API on port 3001 - Next.js 16 + React 19 + Tailwind 4 web UI on port 3000 - Drizzle ORM on PostgreSQL 18, postgres.js driver - BullMQ on Redis 8 (noeviction policy is required) - Uploaded repo archives live in the `haive_repos` named volume (shared by api and worker); Mailpit for dev SMTP - Docker Compose orchestrates everything; the dev stack is driven by `scripts/dev.sh` (aliased `pnpm docker:dev`), which wraps `docker compose up` with the dev override and GPU layering - clawker (Apache 2.0 Go binary) wrapped via child_process in `sandbox/clawker-client.ts`. The binary is installed only when the worker image is built with `CLAWKER_RELEASE_URL` plus a matching `CLAWKER_SHA256`; nothing in this repo sets either, so the binary is absent from the shipped image and the wrapper is inert until someone wires it ## Monorepo layout ``` haive/ package.json pnpm workspace root pnpm-workspace.yaml turbo.json tsconfig.base.json docker-compose.yml postgres, redis, mailpit, api, worker, web docker-compose.dev.yml dev override: port exposure, hot reload, db-migrate one-shot packages/ shared/ @haive/shared types, schemas, crypto, config, logger database/ @haive/database Drizzle schema and migrations api/ @haive/api Hono REST + WebSocket terminal proxy worker/ @haive/worker BullMQ workers, step engine, sandbox manager, CLI adapters web/ @haive/web Next.js Conductor-style UI tests/ e2e/ Playwright specs fixtures/ ``` ## Package boundaries - `@haive/shared` holds zod schemas, types, crypto, logger, ConfigService, SecretsService, and UserSecretsService. May be imported by every other server-side package. May import `@haive/database` for the Drizzle schema namespace used by the secrets services; the actual db client is injected at runtime via `initialize(db)` so shared never instantiates a connection. - `@haive/database` exports the Drizzle schema and a `createDatabase(url)` factory. Instantiated only by the api and worker packages. - `@haive/api` is HTTP + WebSocket only. It must never spawn child processes for CLI execution; that responsibility lives in the worker. - `@haive/worker` owns BullMQ queues, the step engine, the CLI adapter registry, the sandbox/clawker wrapper, and the dispatcher priority chain. It holds no HTTP routes. - `@haive/web` consumes only the public REST API of `@haive/api`. It must not import from `@haive/database` or `@haive/worker`. ## Architecture summary Three queues on Redis: - `task-queue` runs the orchestrator. One job per task. Owns the step machine and persists every transition to Postgres. - `cli-exec-queue` runs the sandbox worker. One job per CLI invocation. Spawns a per-task Docker sandbox via `docker run`/`create` (`sandbox/docker-runner.ts`) — NOT clawker (clawker backs the persistent terminal/login containers in `sandbox/container-manager.ts`); captures piped stdout/stderr and streams to a Redis Stream (`cli-stream:`). Steerable Claude-family runs keep stdin open (`-i`) so a user steer reaches the CLI mid-run. - `env-replicate-queue` runs Dockerfile builds for environment replication. State source of truth is Postgres. Every step transition, every CLI invocation, every form submission is a row. Crash recovery reads the last row. ## Step engine Every legacy markdown step becomes a `StepDefinition` with four phases: 1. `detect` runs first. Pure or shells out via clawker. No LLM. Always runs. 2. `form` returns a `FormSchema`. The web UI renders the schema and the user submits values. 3. `llm` is optional. Spawns a CLI invocation through the dispatcher. For CLIs without native sub-agents the splitter emits a sequential script. 4. `apply` runs last. Writes outputs to `task_steps.output` and to actual files under the workspace. Step lifecycle: `pending` to `running(detect)` to `waiting_form` to `running(apply)` to optional `waiting_cli` then back to `running(apply)` to `done` or `failed` or `skipped`. ## CLI adapter system `packages/worker/src/cli-adapters/base-adapter.ts` defines `BaseCliAdapter`. Implemented adapters: `claude-code`, `codex`, `gemini`, `amp`, `zai`, `antigravity`, `ollama`, `muse`, `grok`, `openrouter`. Each declares `supportsSubagents`, `supportsCliAuth`, `supportsMcp`, `supportsPlugins`, `defaultAuthMode` (`subscription` or `api_key`), and `apiKeyEnvName`. `supportsSteering` defaults to false; only the Claude-family adapters (`claude-code`, `zai`, `ollama`, `muse`, `openrouter`) override it to true. Four adapters are the same trick: the stock `claude` binary pointed at a non-Anthropic Anthropic-wire endpoint via `ANTHROPIC_BASE_URL` — `zai` (api.z.ai), `ollama` (in-stack daemon or ollama.com), `muse` (api.meta.ai), `openrouter` (openrouter.ai/api). All four piggyback the `claude-code` install, so they build no new sandbox image. Their per-endpoint quirks are MEASURED against the live API and recorded in the adapter, not taken from vendor docs — `muse` exists as its own adapter purely because its effort scale rejects `max`, and `openrouter` records the opposite finding (its effort enum is validated globally at the gateway, so every level is safe on every model). Re-probe before "correcting" any of those comments. `ollama` records the third effort finding, and it is the only scale in the codebase whose default is not its top level: `low|medium|high|max` with default `high`. The level arrives as Anthropic `output_config.effort`, which Ollama's compat layer reads only because the claude binary sends `thinking:{"type":"adaptive"}` — the layer honors `thinking.type` for the literals `enabled`/`disabled` and falls through to effort for anything else. An out-of-range level never 400s; it leaves the think value unset, which Ollama then forces on for a thinking-capable model. `max` is exposed but not the default because gpt-oss does not recognise it as a harmony level and reasons LESS at `max` than at `high`, and deepseek-v4-pro at `max` spent an entire output budget in the thinking channel and returned empty text — the same failure the disable-thinking proxy exists for. `high` is additionally what the binary already sent before the scale existed, so the default changed nothing. Orthogonally, the level is on/off only for most LOCAL models: every built-in renderer in ollama 0.24.0 reads `think.Bool()` and none reads `ThinkLevel`, while Ollama Cloud models are rendered upstream and honor all four. Do not "correct" this scale to match zai's — re-probe. `openrouter` traffic goes through the `openrouter-compat-proxy` sidecar (`docker/openrouter-compat-proxy/`), not straight to openrouter.ai. The claude binary appends a trailing `role:"system"` message (the Agent tool's agent-type listing) on top of the top-level `system` field; OpenRouter passes Anthropic models through natively, but every other vendor needs an Anthropic→OpenAI translation in which that message keeps its position, and vLLM-style backends answer `400 "System message must be at the beginning."` (measured on `qwen/qwen3.8-27b` across all three of its upstreams; `--disallowedTools Agent` does not stop the binary emitting it). The proxy hoists it into `system`. Unconditional rather than toggled — the rewrite is a no-op when no such message is present and was verified not to change already-working models — so unlike ollama's thinking proxy there is no per-provider switch. `resolveOpenRouterBaseUrl` is the single place that endpoint is chosen, shared by the adapter and the Test-connection probe; a provider that sets `ANTHROPIC_BASE_URL` by hand bypasses the proxy and the probe then says so. `openrouter` fronts 400+ models, so the provider form picks from a cached catalog rather than a free-text slug: the worker's `REFRESH_VERSIONS` job also refreshes `openrouter_model_cache` (`cli-versions/openrouter-models.ts`), the API serves it at `GET /cli-providers/openrouter/models`, and the picker refuses models whose `supported_parameters` lacks `tools` because Claude Code cannot run a step without native tool use. The cache is never a gate: an empty or errored catalog degrades to a free-text model field. `model_identity` (`queues/cli-exec/model-identity.ts`) records which model ANSWERED, not which one was configured. Two DISTINCT channels, not two names for one value: the claude-family stream-json `system`/`init` event carries what the binary ASKED for, each `assistant` event's `message.model` carries what the endpoint SERVED, and they disagree — api.z.ai answers a `glm-5.3[1m]` request as `glm-5.3`, and recorded streams show the same provider served `glm-5.2` on 2026-07-24 and `glm-5.3` on 2026-08-18, i.e. an endpoint can swap models with no config change here. Coverage is MEASURED per provider: `claude-code`/`zai`/`ollama`/`muse`/`grok`/`openrouter` report both channels; `gemini` names its models only as the keys of `stats.models`; `antigravity` names one only in its `--log-file` and only as a human LABEL (`Gemini 3.7 Flash (High)`), parsed from ONE constant marked volatile that returns null on a reword; `codex` and `amp` report NOTHING. codex's `exec --json` carries no model on any typed event (verified against a complete 3.4 MB SUCCESSFUL run — `model_provider` strings found in such a stream come from old `~/.codex/sessions` rollout files an agent happened to read, not from live events), and amp emits `agent_mode` instead because it abstracts the model away. Those two are permanently `match: 'unknown'`, which is why `unknown` never fails a run. Two traps in that data. `result.modelUsage` keys are recorded as `billed` and are NOT an identity source: `grok` serves `grok-4.6` while billing `grok-4.6-build`, and `claude-code` bills a `claude-haiku-*` call for its own session titling. And an `assistant` event with `model:""` is a message the BINARY authored (an API error), not a model reply — filtered on the angle-bracket convention rather than the literal word, so a future sentinel is excluded too. `requested`/`served` are stored verbatim; only `match` is lenient, and only for one case: an endpoint that DROPS a trailing variant tag while naming the same model (`glm-5.3[1m]` to `glm-5.3`) is `exact`. That keys on the tag's PRESENCE differing, not on tag-stripped equality, so a version swap (`glm-5.2[1m]` to `glm-5.3`) and a different variant (`[1m]` to `[200k]`) both stay `differs`, and ollama's colon marker (`glm-5.2:cloud`) is untouched. A mismatch warns and never blocks — claude-code legitimately resolves an alias to a dated snapshot — unless `CONFIG_KEYS.MODEL_IDENTITY_STRICT` (default false, admin toggle) is on. Capture rides the same parse as token usage, so it costs no extra call, prompt or tokens; the canary (`00-model-health`) then copies the task-default provider's identity onto `tasks.model_identity`, while per-invocation truth stays on `cli_invocations.model_identity` because per-step CLI preferences let one task run several models. Re-measure with `packages/worker/test/model-report-discover.ts` before "correcting" any of this — it runs each adapter's real invocation and feeds the live output through the shipped parser. The dispatcher (`resolveDispatch`) filters to enabled providers, orders the resolved preferred provider first, and picks the first whose adapter is registered and has `supportsCliAuth` — plus `supportsSubagents` when the step declares the `subagents` capability. If none matches, the step is skipped. Every plan it emits is a CLI invocation; there is no API-mode branch. Auth mode selects which credentials the CLI is given, not whether the dispatcher bypasses the CLI. The sub-agent emulator splits a single sub-agent specification into either a native `Task()` call (Claude Code) or a sequential prompt script (everything else). A sequential script runs inside a single `cli-exec-queue` job — the runner is an in-memory for-loop over the sub-steps, with no per-sub-step DB writes. A crash mid-script therefore fails the whole invocation; restart re-runs from sub-step 0. (Mid-script resume would require persisting each sub-step's parsed output to `cli_invocations` before moving on — not implemented.) ## Plan canvas `plan_nodes` and friends (`packages/database/src/schema/plan.ts`, migrations 0130/0131) are the durable, per-repo tree of what a project is MEANT to be — the intentional counterpart to the KB, which only describes how the code currently is. One plan per repo, enforced by a partial unique index on the root node rather than a `plans` table. A repo with no remote and no local tree is a first-class `source: 'blank'` repository: the repo-queue INIT job creates the storage dir, `git init`s it and lands one commit, so worktrees, task attachments and the `.haive-data/` mirror all work on a project that does not exist yet. **One writer.** Every write — LLM turn, UI edit, markdown import — goes through `applyPlanPatch` in `@haive/shared/plan` (NOT the worker: the api writes plans too and cannot import the worker; `global-kb/task-context.ts` is the precedent for a shared package taking an injected `db`). It owns the materialised ancestry (`path`, self-inclusive and slash-terminated so one `LIKE 'prefix%'` selects a subtree), the `version` optimistic-concurrency check (a stale `expectedVersion` is a 409 the UI must SHOW, never a silent refetch), and subtree moves, which rewrite every descendant in one prefix-substitution UPDATE. The SQL offset must be cast — `substring(path from $n::int)` — because an untyped bind resolves to Postgres' POSIX-REGEX `substring(text, text)` overload and returns NULL. `importPlanMirror` is the ONE deliberate bypass: a mirror restore recreates nodes with their original ids verbatim (the spec writer and plan.md quote those ids), which the patch contract deliberately cannot express; it runs only when the repo has zero plan rows and refuses whole rather than half-import when the ids collide. **Plan task types are not in `workflowTypeSchema`.** That zod enum is the create-task REQUEST validator; `plan_build` / `plan_chat` / `advisory` need a node id the generic form has no field for, so they are spawned by their own endpoints (`POST /repositories/:id/plan/...`) — the same arrangement `kb_author` already has. They live in the DB enum and in web's local `WorkflowType` union only. **Completion greens the node** from the worker's `markTaskCompleted` (the only completion write; cancel and fail have their own functions, which is the point). Only `todo`/`in_progress` advance — `blocked_human` and `not_applicable` are verdicts a person entered, and a task finishing is weaker evidence than that. Status roll-up is DERIVED at read time (`rollUpStatus` in shared, one source for every view) and never stored: a `blocked_human` descendant makes every ancestor render blocked; green requires every descendant `done` or `not_applicable` — `not_applicable` must not prevent green, but must never render green itself either. **Two triggers, one builder.** `createPlanBuildStep` (steps/plan/01-plan-build.ts) serves both the standalone `plan_build` task and the onboarding wrapper `10_8-plan-build` (index 14.5, after the KB and RAG exist, before the mirror is staged), because `onboarding_upgrade` reconciles template artifacts only and never re-runs onboarding steps — an already-onboarded repo can reach the builder no other way. The build is LEVEL-BY-LEVEL (pass 0 drafts root + level 1, each later pass fans out one mining agent per frontier node) because one pass cannot emit 400 nodes without truncating. The frontier is recomputed from the DB every pass, and excludes `research`/`external` nodes (a blocker waiting on a person is not a system to decompose) and `taskable` leaves. **plan_chat** is one conversation on one card: a self-targeting `reviseLoop` re-parks the form every turn and the user ends it by submitting nothing. The transcript lives in `plan_node_messages` precisely because that revise resets the step row each cycle. The agent is handed the WHOLE plan (via `renderPlanMarkdown`, the same render committed as `.haive-data/plan.md` — one function so what the agent reads and what is committed cannot drift), so a request made while looking at one node can correctly patch another. **advisory** researches a non-code blocker and then STOPS: `02-advisory-decision` parks on a form and only the USER closes it — an agent concluding an unsigned contract is fine would turn a real blocker into a green tick. **Impact answers "if I change this, what else must change?"** (`shared/plan/impact.ts`): an explicit BFS with a visited set, because the edge graph has cycles by construction and a recursive CTE without dedup would not terminate while one with dedup could not say where it stopped. Both caps are REPORTED, never applied silently. The mermaid source encodes nodes as a `pnode<32 hex>` token; the browser recovers the uuid from THAT, unanchored — mermaid prefixes rendered ids with its own render id, so a `^flowchart-` anchor binds zero handlers and fails silently. **Code links** have one writer (the applier; the builder only links files it actually opened, with `evidence`) and rot is flagged, not guessed away: `11c-rag-reindex` marks links stale for the paths in `tasks.changedPaths`, and only re-assertion by an agent clears the flag — the difference between an impact view that is wrong and one that is merely old. **Spec-writer integration:** when the repo has a plan, 04-phase-0b is handed a compact component index and must emit an `## Affected components` section naming `node:` ids. Its apply parses those IDS — never the agent's prose, because name-matching picks the wrong node the first time two read alike — resolves them through the edge graph, and gate 1 renders the set with a mermaid diagram, stating the traversal cap when one was hit. The mirror (`worker/src/plan/mirror.ts`) writes `.haive-data/plan.json` + `plan.md` from every plan-step apply and from `12-post-onboarding`; `persistDetection` imports it on clone. `CONFIG_KEYS.PLAN_CANVAS_ENABLED` (admin toggle) off refuses new plan tasks and makes the onboarding step self-skip; existing plans stay readable and editable — hiding a plan someone made reads as data loss. Web: drill-down grid, not a graph engine (`components/plan/`); `plan-status.ts` is the single status→colour source; counts are server-computed. Live-Postgres coverage: `pnpm --filter @haive/worker smoke:plan-canvas` (41 checks). ## Review findings and waivers `review_findings` is the durable record of what every reviewing step raised — 07b's validator, 08c/08c2, 08d's adversaries. Findings otherwise live only in `task_steps.output`, which a manual retry nulls, so nothing could say whether a reviewer change helped. It is WRITE-ONLY on purpose: `recordReviewFindings` (`_review-findings.ts`) is the only insert and nothing in worker or api SELECTs the table. No behaviour gates on it, which is why every write there is best-effort and never throws — telemetry must not fail the review that produced it. `disposition` is the verdict on a finding. Every value that is written has exactly one writer: - `open` — default. Nobody has ruled on it. - `dismissed_refuted` — 08c's refuter panel disproved it. Written at INSERT, because the insert happens on the final apply() after refutation ran. - `dismissed_human` — the developer picked a subset at gate 1.5 (`08d2`) and left this one out. Written by `dispositionReviewFindings`. - `accepted_risk` — the fix loop hit its round cap (or oscillated), and the developer accepted the remainder at the escalation gate. Written by `acceptRemainingReviewFindings` from the queue's gate resolver, which is why that one takes a `db` rather than a `StepContext`. - `fixed` / `recurred` — declared, SUPERSEDED, and must stay unwritten. Recurrence lives on `recurrence_count` instead (below); `fixed` should never be written at all, because absence of a finding is not evidence it was fixed — a reviewer that was skipped, budget-killed, or simply reworded it produces an identical absence. Do not read `open` as "still outstanding". Both UPDATE paths are scoped to the CURRENT round and guarded on `disposition = 'open'`. The round scope matters because the same fingerprint recurs on its own row every round, so an unscoped update rewrites the history of rounds the developer never saw — and because with no `fixed` writer, an earlier round's rows are `open` whether they were fixed or recurred. The `open` guard keeps the more specific verdict: a refuter's disproof and a gate-1.5 waiver both outrank a blanket acceptance. Findings raised AFTER the acceptance stay `open` and are NOT swept into it — the developer accepted the remainder they had been shown, not findings nobody had seen yet. That is the normal case, not an edge one: acceptance stands down the FIX loop only, while `restartLoop` (a gate-2 reject) is suppression-immune, so a later round runs and records more. MEASURED on the dev install, every accepted task has findings past its accepted round (accepted 7 / findings to 9, 4 to 5, 5 to 7, 6 to 9). Those surface at gate 2, where the call is the developer's again. A waiver is CONTEXT, never SUPPRESSION. Nothing reads `dismissed_human` back, and 08d2's detect deliberately does not fold on it. A verdict is about a MOMENT, not about the defect — the reason 08d gives where it carries its disproved set forward, and a human declining to spend a round on something is far weaker evidence than a PoC a verifier actually ran and could not reproduce. What a partial fix round leaves behind instead is a task-ledger entry (`formatWaiverLedgerEntry`) whose wording invites a re-raise and asks what changed; `augmentPromptWithLedger` already carries it into the next round's prompts, so it needs no plumbing of its own. Accept-as-is at gate 1.5 records NOTHING — those findings still surface advisorily at gate 2, so the call is deferred rather than made. `recurrence_count` records how many EARLIER rounds of the task already saw that reviewer flag that FILE, stamped at insert by `recordReviewFindings` from `loadFindingRecurrence`. A counter beside the verdict rather than a disposition value, for two reasons: recurrence is orthogonal to a ruling (a finding can be both refuted and recurring, and `disposition` holds one value), and both disposition UPDATE paths are guarded on `disposition = 'open'`, so a `recurred` written there would hide exactly the most-repeated findings from a gate-1.5 waiver and from the escalation gate's acceptance. Surfaced two ways: `recurrenceTag` prefixes gate 2's finding lines with `[repeat xN]` plus a one-line legend, and `buildRecurringNote` (08c) hands the next fix round an "Already tried" block, since each round is a fresh `claude -p` and `loadPriorFixContext` dedupes prior diagnoses by prose. The key is `(reviewerId, normalised path)` — deliberately NOT `fingerprint`. MEASURED across 9,750 findings in 68 multi-round tasks: the fingerprint matches across rounds 5 times (0.05%) because it hashes the issue TEXT and two rounds' descriptions of one defect are different sentences, while (reviewer, path) matches 1,408 of 3,294 (42.7%) and reading the rows confirms those are one complaint re-raised — one across 19 rounds, every row still `open`. The cost of the coarser key is that two DISTINCT defects in one file count as a repeat, which is why every surface says "this reviewer already flagged this file" and never "the same defect". Nothing branches on the count; it informs a human and a prompt. `fingerprint` is `sha256(reviewerId, path, normalised issue)` — line numbers and ids stripped, path KEPT, since the same defect in two files is two findings. Rows are written PRE-merge, one per adversary that reported a finding, while every later step sees only the survivor of 08d's location merge and has no reviewer id to rebuild a key from. That is why 08d attaches the collapsed set to each surviving finding as `fingerprints[]` (same shape and reason as 08c's `RefutableFinding.fingerprints`), on the OUTPUT copy only so the merge's own objects and `raw` stay untouched. ## Model pricing and spend `cli_invocations.cost` records what a run COST, decided at completion beside `token_usage` and `model_identity` and then immutable. Distinct from `token_usage.costUsd`, which keeps its old meaning (what the CLI itself reported). The split exists because the reported number is real only where the CLI prices its own backend: the claude binary applies ANTHROPIC's table to every backend, so for zai/muse/openrouter/ollama it is fiction, and codex/gemini report no cost at all. MEASURED before the feature: of ~12,800 USD reported across 12,438 invocations, 31.68 USD was the only amount the product could count as real. Precedence per invocation (`resolveCostDecision`): manual rate > CLI-reported where `costBasis` is `metered` > computed from a synced rate. Subscription auth short-circuits to non-billable regardless — a flat plan's per-token dollars are notional. `resolveCostBasis` demotes BOTH per-token bases under subscription auth, `metered` AND `estimate`, so a claude-binary wrapper on a flat plan (a GLM coding plan) is non-billable too; its counterfactual is then computed from Haive's own feed rate, never from the binary's Anthropic-table total, and an unpriced model there is `none` rather than fiction. `billable` is decided at WRITE time, where provider, auth mode, model and price source are all in hand, so reads honor a flag instead of re-deriving the rule (it had been duplicated across three SQL filters). A partial computation is recorded `source: 'none'`, never as a cost: a total missing one bucket looks like money and is not. `ollama` is the one provider whose basis is per-MODEL rather than per-CLI, so `resolveCostBasis` takes an optional model key. One catalog entry covers the local daemon and Ollama Cloud alike, and the catalog says `local` because that is what `ollama` used to mean; a `-cloud`/`:cloud` model runs on ollama.com under a plan that is plan-included up to a limit and metered beyond it, which is neither free nor per-token-billed, so it resolves `subscription` (non-billable, priced only as the counterfactual). Keyed on the model TAG via `isOllamaCloudModel`, the same test the base-URL routing and the boot provisioner already use — NOT on the base URL, which a cloud provider row does not set at all because the local daemon proxies cloud models. Both tag forms matter: the common one is `-cloud`, and every zero-rate cloud row the LiteLLM feed carried wore it, so a `:cloud`-only check would have missed all of them. A caller with no model in hand (the per-provider usage rollup groups by provider, not by model) keeps the catalog answer. The non-billable half is not discarded, it is the SUBSCRIPTION COUNTERFACTUAL: `notionalCostUsdSql` (api `routes/tasks/_helpers.ts`) sums what those same tokens would have cost at list API rates, rendered grey and labelled on the task's provider card, never added to real spend. Two halves, each the exact complement of the real-spend rule: snapshot rows flagged non-billable minus `source: 'none'` (whose 0 means unpriced, not free), plus legacy `cost IS NULL` rows from a METERED provider on subscription auth. The claude-binary wrappers are excluded from the legacy half because their reported total is Anthropic-table fiction in either direction (5,968 USD of it against local ollama tokens), and codex/gemini report no cost at all, so both contribute 0 rather than a guess. MEASURED on the dev install: 29.63 USD real against 3,236 USD notional. Rates live in `cli_model_prices`, effective-dated and append-only. A sync CLOSES the live row and inserts a replacement ONLY when a rate actually moved, so an unchanged 12-hourly tick writes nothing (verified: `inserted 0, closed 0, unchanged 651`). Three feeds, refreshed by the existing `REFRESH_VERSIONS` job (`cli-versions/model-prices.ts`), sequentially after `refreshOpenRouterModels` because it reads that cache rather than re-fetching 4 MB: - **LiteLLM** `model_prices_and_context_window.json` for direct-vendor rates. The feed publishes one entry per (host, model), so `PROVIDER_LITELLM_VENDORS` maps each Haive provider to the vendors that ARE its backend and everything else is rejected — `azure_ai/claude-opus-5` and `vertex_ai/claude-opus-5` are different rows at different prices, and `azure/us/gpt-5.6-sol` really is 10% over list. MEASURED 2026-08-18: anthropic 26 chat models, openai 90, gemini 50 + vertex_ai-language-models 29, xai 44, zai 13, meta 3. `ollama` is deliberately NOT mapped here: all 21 of its rows are priced 0, which is the truth for a local model and a LIE for the four CLOUD models the feed also carries under that vendor (`deepseek-v3.1:671b-cloud`, `gpt-oss:20b-cloud`, `gpt-oss:120b-cloud`, `qwen3-coder:480b-cloud`). A stored 0 is not "unpriced", it is a PRICE — it makes an invocation record a real $0.00 — so those rows were retired by migration 0126 and the vendor mapping dropped. - **OpenRouter** `/api/v1/models` for the gateway's OWN resale rates, provider-scoped to `openrouter` and never priced from a vendor rate — it bills its margin at the routed model's price, wrong by a different factor per model. - **Ollama** the model pages themselves (`cli-versions/ollama-model-prices.ts` fetching, `cli-providers/ollama-pricing.ts` parsing), because Ollama publishes no price document and no pricing API — `ollama.com/pricing` lists plans and difficulty grades, not rates. Bounded to CONFIGURED CLOUD models, one page each per tick, never the whole library; cloud-only is a GUARD, since a stored rate on a `local`-basis run would be summed as real spend. MEASURED 2026-08-23 across the 10 configured cloud models: 1 (`kimi-k3`, $3.00 input / $0.30 cached / $15.00 output per 1M) publishes a cost block and 9 do not. Absence is a RESULT, not a failure — it is the common path and must never log an error or every tick cries wolf. A block that EXISTS and cannot be read fails loud and writes nothing. Amounts are paired with the label that follows them, never by position (an automated read of that page during diagnosis returned input $0.30 / cached $3.00, the two swapped), and `cached <= input` is asserted because a violation means the pairing is wrong. Selectors live in ONE constant marked volatile — this is HTML, it will reword. Lookup is EXACT on a normalized model key (trim + lowercase, nothing else), deliberately NOT the longest-substring matcher `resolveContextWindow` uses: a wrong context window skews a cosmetic percentage, a wrong price is wrong money, and `claude` alone matches both opus and haiku (~15x apart). Variant markers are kept verbatim (`glm-5.3[1m]`, `deepseek-v4-pro:cloud`) because a context or hosting variant is usually its own SKU. An unmatched id is UNPRICED, never guessed — which is why the manual-override path is load-bearing rather than a nicety (`glm-5.3` is in no feed). One arithmetic trap: Anthropic-shaped usage reports `input_tokens` EXCLUSIVE of the cache buckets, while codex and gemini report it INCLUSIVE of the cached prefix (`INPUT_INCLUDES_CACHE_PROVIDERS`). Pricing those two without subtracting charges cached tokens at the full input rate. Separately, cache-WRITE rates are frequently absent from the feeds, which costs nothing in practice: MEASURED, only claude-code (which uses its reported cost anyway) and amp (subscription) ever report cache-creation tokens — 0 across 6,204 invocations for every other provider. Admin control is two-level and BOTH are enforced at lookup, not only at sync, so a change takes effect on the next invocation rather than at the next refresh: `CONFIG_KEYS.PRICING_AUTO_UPDATE_ENABLED` is the global kill-switch, and `cli_pricing_sync.auto_update_enabled` per CLI means "this provider's rates are admin-owned" — its feed rows are ignored and only manual rates apply. Manual rates are effective-dated the same way (a write closes the previous one), and retiring one closes it rather than deleting, since the rows that priced past invocations must stay readable. Admin UI: a card on `/admin` for the global switch plus display currency, and `/admin/pricing` for per-CLI toggles, the rate table and overrides. Costs are stored canonically in USD. `fx_rates` holds daily ECB USD-per-unit rates and a task converts at the rate effective on ITS OWN date (`resolveCostDisplay`), so re-rendering a finished task yields the same figure; dated on the task rather than per invocation because FX drift within one task is far below the displayed precision. ECB publishes only the current day, so past rows cannot be re-fetched — do not truncate that table. A task older than FX collection converts at the earliest rate on record and is flagged `approximate`. ## Sandbox `packages/worker/src/sandbox/clawker-client.ts` wraps the clawker binary. The worker container mounts `/var/run/docker.sock` and uses Docker-in-Docker to spawn per-task containers. Only the cloned repository is bind-mounted into the per-task container. The worker filesystem and the user home directory are never exposed. CLI authentication files are copied into a named volume per task at startup and the volume is destroyed at task end. Secret-file masking (default on, Tier 1): before each cli-exec invocation the worker hides files matching a secret deny-list from the AI CLI agent by bind-mounting empty read-only files over them inside the cli-exec sandbox (`packages/worker/src/queues/cli-exec/secret-mask.ts`, threaded via `resolveSecretMasks` in `exec-core.ts` for the `cli`/`agent_mining`/sub-agent kinds). The effective set is `DEFAULT_SECRET_DENY_GLOBS` (in `@haive/shared`) plus per-repo `secret_mask_deny_extend`, minus `DEFAULT_SECRET_CARVEOUTS` and per-repo `secret_mask_allow`. Untracked files only (`git ls-files` filter) — committed secrets are out of scope FOR MASKING, and are instead REPORTED once per repo by the `07_7-secret-sweep` onboarding step (which is why they are two features and not one: masking stops an agent reading a secret, the sweep tells the human one is already in their history). The tracked filter asks each linked worktree about its own paths, because `git ls-files` reports paths relative to the tree it runs in, so the repo root never lists `.haive/worktrees//x`. The app runtime (app-runner/ddev mount the same `haive_repos` subpath without masks) still sees the real files. Per-repo controls live on the tooling settings page (`secret_mask_enabled`/`secret_mask_allow`/`secret_mask_deny_extend`); `CONFIG_KEYS.SECRET_MASK_ENABLED` is the global kill-switch. Masking fails closed. A scan that throws, a scan root that is not a readable directory, a match count over `SECRET_MASK_LIMIT`, or a task/repository row that cannot be resolved raises `SecretMaskError` instead of masking a partial set — a subset leaves the remainder readable, which is the one outcome the deny-list exists to prevent. "Masking is off" and "no secrets found" are only ever concluded from evidence that says so: the kill-switch, `secret_mask_enabled`, or a task with no repository (which mounts no tree). The repo root mirrors `resolveTaskRepoMount` exactly, so a repo with no `storage_path` is scanned at its named-volume subpath rather than skipped, and the root is `stat`ed before the scan — glob answers `[]` for a root that does not exist, which is byte-identical to a clean repo, while the sandbox mount binds the real tree regardless of what the worker can see. `handleCliExecJob` records it on the invocation (exit -1) and fails the step. The escape hatches are the repo's `secret_mask_allow` globs and the masking toggles; disabling masking skips the scan and never raises. Worktree gitfile masking (always on): every agent prompt states that git is unavailable inside the sandbox and that the host stages and commits (`10-gate-3-commit`, `completeMergeHostSide`). That invariant is enforced, not incidental — `worktreeGitfileMask` (`packages/worker/src/queues/cli-exec/gitfile-mask.ts`) bind-mounts an empty read-only file over the worktree's `.git` gitfile for every cli-exec invocation. Without it an agent can repoint the gitfile at the container path (`printf 'gitdir: /haive/workdir/.git/worktrees/' > .git`), which both grants itself a working git behind the commit gate and leaves host-side git fatally broken for every later step. It rides the same `SandboxExtraFile` mechanism as secret masking but is an integrity control, so `SECRET_MASK_ENABLED` never disables it. Never masked at the repo root (there `.git` is a directory), and never applied to the terminal, IDE, app-runner or ddev containers, whose git must keep working. `removeWorktreeDir` runs `git worktree repair` before removal so worktrees poisoned before this existed still clean up. One browser per task, one TAB per agent. Every sandboxed CLI of a task gets the same `--browser-url` (`resolvers.ts` probes the runner once per invocation), and chrome-devtools-mcp selects `pages[0]` on connect — so N concurrent agents drive ONE tab unless told otherwise. They are told otherwise by `BROWSER_TAB_DISCIPLINE` (`sandbox/mcp-surface.ts`), which rides the MCP surface block and therefore reaches every browser-capable dispatch: llm, mining (`08d-adversarial-qa`, the only fan-out keeping the full surface) and `dag_parallel` coders. `isolatedContext` is banned there rather than unmentioned — it is the obvious-looking way to isolate and it starts a fresh cookie jar, discarding the one deterministic app login `_app-auth.ts` performs per task. Tab 0 is the human's view: `browser-probe-connect.js` and `browser-login.js` both reuse `pages[0]` and bring it to front. A per-agent headless browser is NOT the alternative — it cannot reach a `*.ddev.site` or app-runner URL from the cli-exec sandbox, and carries no login. Agents killed before their `close_page` are swept by `closeExtraBrowserTabs` (`browser-close-extra-tabs.js`) at the mining and DAG barriers ONLY, never at browser bring-up, which runs while a human may have tabs open. The tab it keeps is the one RECORDED as the human's (`browser-human-tab.js`, written by `browser-probe-connect.js`/`browser-login.js` at the moment they `bringToFront()` a tab, read back as a CDP `Target.getTargetInfo` id), never inferred from the tabs. Three inference attempts have been MEASURED lying: `browser.pages()` is not creation order (two probes minutes apart put the newest tab at index 0 and then the oldest, so an early keep-`pages[0]` version closed the app tab the human was watching), `/json/list` is a third order again, and `document.visibilityState` — which replaced them — reported `visible` for BOTH tabs of a two-tab window (identical `windowId` and bounds), along with `document.hidden` false, `hasFocus()` true, animation frames and screencast frames, so the sweep failed safe and closed nothing on exactly the runners that leak. No record, an unreadable one, or a recorded tab that is no longer open still closes NOTHING; a leaked tab costs memory, closing the wrong one costs work. The launch script also restores puppeteer's three anti-backgrounding switches, since with a tab per agent all but one are always in the background — the likeliest reason no page-side signal separates them any more. The same barrier RESTORES THE WINDOW, and Gate 2's bring-up does it again before the human looks (`browser-restore-window.js`, `restoreRunnerBrowserWindow`/`restoreAppRunnerBrowserWindow`). `resize_page` is not viewport emulation: MEASURED in the shipped chrome-devtools-mcp 1.7.0 bundle it un-maximizes the OS window and then sets its bounds so the CONTENT matches, so 08a's deterministic 1280x800 outlives the MCP session — three live runners were sitting at 1280x887 on a 1920x1080 screen. The agents cannot undo it and are not asked to: `resize_page` takes a CONTENT size and there is no window-state tool, so the screen size overshoots the display, and one window serves every agent's tab, so a per-agent restore would resize a sibling mid-screenshot. Restore target is the screen read from the page, NOT the 1920x1080 literal in `start-browser-desktop.sh` and not the launch geometry (Chrome offsets that by 10,10, which would hang 20px off two edges). Chrome settles at 1919x1079 for a 1920x1080 request, so the skip test asks whether the window COVERS the screen rather than matching it, or every barrier would rewrite identical bounds. ## Build commands - `pnpm install` installs all workspace dependencies. - `pnpm build` runs `turbo run build` across the workspace; `@haive/shared` and `@haive/database` build first because all other packages depend on them. - `pnpm typecheck` runs `tsc --noEmit` everywhere. - `pnpm test` runs Vitest across the workspace. - `pnpm test:e2e` runs Playwright against the dev compose stack. - `pnpm db:push` runs `drizzle-kit push` against the database in `DATABASE_URL`. - `pnpm docker:dev` (alias for `scripts/dev.sh up`) boots `docker-compose.yml` plus the dev override, GPU-aware. The script also exposes `rebuild`/`reset`/`restart`/`libs`/`logs`/`status` — run `pnpm docker help`. - `pnpm docker:down` (alias for `scripts/dev.sh down`) stops everything; it keeps all data volumes (never `-v`). The api and worker packages depend at build time on `@haive/shared` and `@haive/database`. Always build those two first when running anything outside of turbo. ## Conventions - All modules are `"type": "module"`. Use `.js` extensions in import paths even for TypeScript sources because of `NodeNext` module resolution. - Zod is used for both validation and for generating `FormSchema` field metadata where possible. - Logger is `pino` from `@haive/shared/logger`. Never `console.log` from server code. - Secrets are stored via envelope encryption: per-user DEK encrypts the secret, master KEK from `CONFIG_ENCRYPTION_KEY` encrypts the DEK. AES-256-GCM throughout. - Drizzle schema lives in `packages/database/src/schema/`. Migrations in `packages/database/src/migrations/` are generated via `drizzle-kit generate` and applied via `drizzle-kit push`. - Hono routes group by domain in `packages/api/src/routes/`. Auth middleware mounts globally. - Forms are described by `FormSchema` from `@haive/shared` and rendered by `FormRenderer` in `@haive/web`. Do not write step-specific React components. - EVERY prose body renders through `MarkdownView` / `.haive-md` (`packages/web/src/components/markdown/`). There is no plain-text branch. `looksLikeMarkdown` still exists but decides ONE thing — the LINE-BREAK POLICY, not the style: a markdown body gets `remarkGfm` alone so a model's ~80-col hard wraps reflow, a plain body also gets `remarkSoftBreaks` so its newlines survive as `
`. Do NOT re-promote it to a renderer switch. It used to be one, and identical prose then jumped between mono-12px `
` and sans-14px `.haive-md` on nothing more specific than whether it happened to contain one backtick. `remark-soft-breaks.ts` is a deliberate local stand-in for `remark-breaks`, not an oversight: the dep is not in the pnpm store and adding one costs a full stack rebuild. A body nested INSIDE another styled block (form/field/option descriptions, note fields) uses `InlineMarkdown`, which takes its size and colour from a WRAPPER element via `.haive-md-inherit` — putting those classes on MarkdownView's own element does not work, because `font-size: inherit` resolves against the parent, so a `text-xs` there would be what it inherits FROM rather than what it applies. Inline code under that modifier is sized as a RATIO of the body (`0.9285em` = 0.8125/0.875), never the absolute rem the plain rule uses, with `pre code` reset to `inherit` so the same selector does not scale it twice. `globals.css` is outside the repo's prettier glob (`ts,tsx,js,jsx,json,md`) and is hand-formatted compactly — running `prettier --write` on it produces ~150 lines of unrelated churn.
- A message column is display copy, never state. `task_steps.status_message`, `task_steps.error_message` and `cli_invocations.status_message` all outlive the thing they describe: a park whose poll chain ended leaves its last line behind, a step that failed once and succeeded later keeps its error text, and an invocation picked up immediately can be labelled "queued" by a write that lost a race. Gate UI on the structural column that proves the state (`waiting_started_at` for a runtime park, `status` for a failure, `started_at` for a queued invocation) and use the message only as the words inside the banner. The rule lives in `packages/web/src/lib/step-banners.ts` (`parkBanner` / `failureBanner` / `invocationBanner`, unit-tested) — extend that module rather than re-deriving the condition at a new call site. Keying on copy presence produced three separate phantom states in one day: two live "waiting for a slot" banners on one task, a `done` step rendering "cli invocation failed" with a Retry button, and a running CLI advertising "Queued — machine at capacity".
- Do NOT enforce that rule as a CHECK constraint or a nulling trigger. `error_message` on a `done` row is legitimate for `fixLoopOnError` steps (`step-runner.ts`), which write `status: 'done'` together with the error as the diagnosis that routes the fix loop back to implementation — a blanket "done implies no error" would silently destroy it. Repair stale copy with a numbered data migration instead (see `0104`, `0105`, `0106`).

## Where things live

- Step modules: `packages/worker/src/step-engine/steps/{onboarding,workflow,env-replicate}/`
- CLI adapters: `packages/worker/src/cli-adapters/`
- Sandbox wrapper: `packages/worker/src/sandbox/`
- Terminal proxy: `packages/api/src/routes/terminal.ts` plus `packages/web/src/components/terminal/`
- Orchestrator state machine: `packages/worker/src/orchestrator/state-machine.ts`
- Dispatcher priority chain: `packages/worker/src/orchestrator/dispatcher.ts`

## Phasing

Phase 0 scaffold is complete when `pnpm install` and `pnpm docker:dev` boot all services on a clean host with only Docker installed. Subsequent phases build the database schemas, auth, repository management, CLI adapters, sandbox, terminal proxy, step engine, sub-agent emulator, autonomous workflow, and environment replication in that order.

## Constraints

- WSL2 plus Docker is the only supported developer environment. No Windows-native installs.
- Concurrent per-task runtimes (DDEV/app runners) and CLI agents share ONE machine-aware RAM budget (`total - ~30% reserve`), not a fixed count. Two distinct numbers: the Docker `--memory`/CPU/pid **cap** each container runs under (a generous ceiling, swap disabled) and the **planning weight** admission budgets it at. Dividing the budget by the ceiling is what used to price a ~300 MB app-runner like a DDEV DinD. Weights are CALIBRATED absolutes, not fractions of the cap (a project's runner uses the same RAM on a 16 GB host as on a 64 GB one), measured from 1731 `docker stats` samples over 4 DDEV runners and 52 agent sandboxes: `ddevWeightMb` 1536 (peak 1036 incl. a full cold boot), `browserWeightMb` +1536 surcharge charged WHILE THE DESKTOP IS UP (browser-phase peak 2505), `agentWeightMb` 2048 (peak 1736), `appWeightMb` 1024 (the one unmeasured weight — strictly less machinery than a DDEV base). Each is clamped to the container cap, since a runner cannot occupy more than `--memory`. A cold `ddev start` is NOT the peak (image pulls are disk-bound); the peak is agent work during a verify/fix round, which is also when the browser desktop is up — so the browser flag is a proxy for that phase, not the cause. That surcharge is LIFECYCLE-SCOPED, not stamped into the weight label at container create: the desktop first starts at 07, so a task charged for it from `01c` paid for a Chromium that did not exist for hours — MEASURED, two of three live DDEV runners had zero Xvfb/x11vnc/chrome processes while each was charged 1536 MB for one, which is what parked other tasks against a budget nothing was spending. Labels are immutable after create, so the live half is a Redis SET of runner CONTAINER NAMES (`markBrowserDesktopUp`/`Down`, called inside `startBrowserDesktop`/`stopBrowserDesktop` so every step inherits it) folded into occupancy by `parseRunnerWeights`. It is clamped to the kind's class weight plus the surcharge, which is what stops a runner created BEFORE the move — whose label already contains it — being charged twice, while leaving a per-task `memoryLimitMb` pin (already above that cap) alone. Applied only to containers the same `docker ps` returned, so a stale entry is inert and no TTL or sweep exists. Agents take what the runtime pool leaves free, never below `agentFloor` (a runtime holder needs an agent to finish, so zero agents deadlocks) — but that subtraction is a difference of two peak-calibrated estimates and is wrong in BOTH directions at once, so `AGENT_POOL_MEASURED_ENABLED` (default on) lets the host's own `MemAvailable` answer instead. MEASURED: three DDEV runners charged 9216 MB were occupying 2551, flooring the pool at 2 while 7.9 GB sat free, and the dev base stack meanwhile overran its 30% reserve — a term the planned figure cannot see at all. `readHostAvailableMb` reads MemAvailable, NOT `os.freemem()` (that is MemFree: 750 MB against 7.9 GB available, which would zero the term), and prefers a cgroup limit when one is set below host total because `/proc/meminfo` reports the HOST inside a capped container. `agentWeightMb` stays the divisor — being coarse (19% of a 16 GB budget) is exactly why the planned quantisation could not express the difference. Only the RESERVATIONS half of runtime occupancy is subtracted from the measurement; a live runner's RSS is already missing from it. Growth BEYOND the planned figure is capped at `DEFAULT_AGENT_RAMP_STEP` (2) per 30s evaluation, each step re-measured with the previous one resident, so an illusory headroom stops itself; the plan's own width still opens at once (an idle host was never broken) and shrinking is never ramped, since a lower cap only stops NEW jobs. `AGENT_POOL_SAFETY_MB` (0 = a fifth of the host, clamped 2048..4096) is what the ramp holds back for the base stack's growth and for agents climbing from ~450 MB toward their 1736 MB peak; raise it if the host swaps as the pool ramps. Admin-tunable: RESOURCE_LIMITS_ENABLED / RUNTIME_MEMORY_MB / RUNTIME_CPUS / MAX_CONCURRENT_RUNTIMES=0 means no count cap / RUNTIME_DDEV_WEIGHT_MB / RUNTIME_APP_WEIGHT_MB / RUNTIME_BROWSER_WEIGHT_MB / AGENT_WEIGHT_MB / AGENT_FLOOR / AGENT_POOL_MEASURED_ENABLED / AGENT_POOL_SAFETY_MB / MAX_PARALLEL_AGENTS=0 means auto (a positive pin still short-circuits everything, measurement included) / RUNTIME_IDLE_REAP_MINUTES. On a DEV host the binding constraint is usually the base stack, not the weights: `next dev` alone holds ~1.6 GB and Ollama ~1 GB against the 30% reserve, so pin MAX_CONCURRENT_RUNTIMES or cap the dev web server's heap rather than raising weights. A step is gated only if it will actually spawn a runner (`needsRuntime: 'ddev' | 'app' | 'if-serving'`; `'if-serving'` resolves through `classifyRuntime`, so a code-only task never queues for a pool it does not use). A reaper reclaims leaked runners. See packages/worker/src/sandbox/{runtime-caps,runtime-admission,runtime-runner-reaper}.ts and `deriveRuntimeCaps`/`deriveAgentConcurrency` in @haive/shared.
- Runtime occupancy sizing the agent pool is only half the story: WHO gets those few slots matters too. A task holding NO live runtime runner yields its cli-exec slot to one that does (`enforceRuntimeHolderReserve`, `queues/cli-exec/agent-reserve.ts`), because a runner-holder's idle time is billed in committed RAM it can only release by FINISHING — while a runner-less task's slot is fungible. Observed without it: 3 tasks each holding a 3072 MB DDEV, two idle, an onboarding task holding one of the only 2 agent slots, and 4 more tasks parked behind capacity that could not free. Enforced at job PICKUP (a third sibling to `enforcePauseGate`/`enforceTaskAgentCap`, same `moveToDelayed` + `DelayedError`), never at enqueue — BullMQ freezes priority at `queue.add` while runner-holding is state that changes. Keyed on the docker-label invariant "holds a live runner", NOT on `tasks.type`, so a future runner-less task type inherits it. Demand is counted from the BullMQ WAITING list, not from queued `cli_invocations` rows: a DB count includes jobs that are delayed (paused task, per-task cap) and yielding to those idles the slot. Not preemption — a running agent keeps its slot. The one thing that outranks the reserve is an EXPLICIT vote: a runner-less task whose `tasks.vote_score` is STRICTLY above every holder's skips the yield (`voteScore`/`maxHolderVoteScore` on `agentReserveDecision`; both read live at pickup, never carried on the job, for the same reason the gate itself is at pickup). Strictly, because a tie states no preference and there the committed-RAM argument still decides — and at the default score 0 everywhere the term is inert, so an install where nobody votes behaves as it did before. A vote is the operator saying what to run first; holding a runner is this module inferring it from RAM, and the statement beats the inference. Without it a vote re-priced only the QUEUE while this gate handed the slot away anyway: MEASURED, +1 onboarding invocations sat 12.6/14.1/21.9 min behind unvoted DDEV holders that were at the head-adjacent bands. Admin-tunable: AGENT_RESERVE_ENABLED (off = first-come) / AGENT_RESERVE_MAX_HOLD_MINUTES (0 = strict, else release a held invocation after N minutes so runner-less work slows rather than stops; default 3, not 10, because the hold is only half the wait — a released job then waits for a slot to free, a term invisible in the knob).
- Queued cli-exec priority is `band = VOTE_BASE + rank - vote_score`, and BullMQ FREEZES it at `queue.add`. `rank` (the task's in-flight agent count) is therefore a SNAPSHOT, which turned a drained fan-out into a permanent sentence: MEASURED, an idle +1 task sat at band 9 (rank 5, from an `09_5` mining fan-out long since finished) behind a neutral task at band 6 that had two agents RUNNING, and led the queue on nothing but enqueue order. A vote cannot correct that on its own — it moves at most 5 bands and a 5-wide fan-out moves 4. `CliPriorityDecaySweeper` (`queues/cli-exec/priority-decay.ts`, 30s) rebuilds the band from live load: for `r` STARTED agents and `k` queued jobs in their current order the ranks are `r+1 .. r+k`, i.e. exactly what an incremental enqueue would produce right now — so the round-robin across tasks is preserved while the fan-out penalty decays. Order INSIDE a task never changes, and a settled queue produces zero writes. A sweeper rather than a hook on invocation-end because the count also drops on paths that never run that code (orphan sweep, preemption, cancel). It has no switch of its own: with `FAIR_SCHEDULING_ENABLED` off, jobs carry priority 0 and every one is skipped.
- The Docker socket mount in the worker container is effectively root on the host. Document this in the README and offer rootless Docker instructions in Phase 9 hardening.
- All step content lives in TypeScript modules. Do not pipe legacy markdown into a CLI prompt.

## Onboarding template versioning

Deterministic onboarding artifacts (agent specs, slash commands, `workflow-config.json`, Drupal LSP plugin files, the `agents/README.md` index) are registered as `TemplateItem`s in `packages/worker/src/step-engine/template-manifest.ts`. Every item has:

- `id`: stable slug (e.g. `agent.code-reviewer`, `command.review`, `workflow-config`).
- `schemaVersion`: integer, bumped only on shape-breaking changes (filename change, new required frontmatter field).
- `contentHash`: sha256 over the rendered reference output, computed on worker boot from `REFERENCE_CONTEXT` and cached per-manifest.
- `render(ctx)`: invokes the existing generator and returns `TemplateRendering[]`; multiple renderings per item when an agent fans out across CLI target dirs.

On worker boot, `syncTemplateManifestCache(db)` upserts the manifest into Postgres (`template_manifest_cache`) so the API can compute the current set hash without importing worker-side generators. Per-repo install state lives in `onboarding_artifacts`, one live row per `(repository_id, disk_path)`, soft-deleted via `superseded_at`.

### When changing a template

1. **Body-only change (rewording an agent prompt, fixing a typo, updating a command example):** edit the generator in `_agent-templates.ts` / `07-generate-files.ts`. The manifest's `contentHash` recomputes on worker boot and the upgrade-status endpoint starts reporting the template as changed. **Do not bump `schemaVersion`.**
2. **Shape change (rename agent id, change `workflow-config.json` schema, change a command's disk path):** bump the item's `schemaVersion` in `template-manifest.ts`. Rollback across a `schemaVersion` bump restores the prior artifact's stored bytes (migration 0013) and reverts correctly; only legacy rows written before stored content existed refuse to revert across a bump (they have no prior bytes and re-rendering a changed shape is unsafe).
3. **New template:** add the generator, append to `buildTemplateItems()` in `template-manifest.ts`. First upgrade per-repo will surface it in the `new_artifact` bucket.
4. **Removed template:** delete the `TemplateItem`. First upgrade per-repo will surface existing artifacts in the `obsolete` bucket.

Out of scope of onboarding-upgrades: `.claude/skills/` and agents written from LLM discovery (06_5) without a bundle source, `.haive-data/knowledge_base/`, `.claude/mcp_settings.json`, `.claude/onboarding-review.md`. **Skills and agents installed from custom bundles are tracked and upgradable** (each bundle item lands as an `onboarding_artifacts` row with `templateId = "custom.."`; the `bundle_resync` step before `01-upgrade-plan` refreshes git bundles and the upgrade-plan/apply path treats `custom.*` rows the same as Haive templates). LLM-generated skills and KB content are refreshed by the `/workflow` code-change phase. Project knowledge lives under `.haive-data/` — `knowledge_base/` and `learnings/` — not under a vendor dir, because Haive is model-agnostic; the canonical paths are `KB_DIR`/`LEARNINGS_DIR` in `packages/shared/src/knowledge-paths.ts`, and `stripManagedKnowledgeGlobs` makes them impossible to scope-exclude from the folder pickers (06_7/09_7, the api exclusions endpoint, and the repos-page editor). `mcp_settings.json` is user-owned — created on first onboarding, never rewritten. The review file is one-shot; each upgrade writes a new `.claude/upgrade-reviews/.md`.