git:20261018.f07081a to git:20261018.d6ca502

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

Vizier is a Rust-based AI agent framework: a single binary that runs multiple concurrent AI agents, each with its own tools, memory, and provider, exposed over Discord, Telegram, and HTTP (REST + WebSocket + a bundled React WebUI). Storage is embedded SQLite — no external DB service required. Agent memory itself lives as human-readable markdown documents on disk (organized into per-agent **bundles**, see below), read/written through a pluggable `DocumentStore` abstraction; every other entity lives in SQLite.

## Commands

```sh

just install # cargo fetch + npm i in webui/

just dev # cargo watch -s "just run" (hot-reload)

just run # cargo run -- run --config dev.vizier.yaml

just run-d # same, detached (-d)

just shutdown # cargo run -- shutdown --config dev.vizier.yaml

just build # cd webui && npm run build

just release # cargo build --release

just catalogue-update # refresh assets/catalogue/ (models.dev seed + logos) — needs network

just jaeger # local Jaeger in Docker (UI :16686, OTLP :4318); jaeger-down removes it

just run-otel # just run, exporting traces to that Jaeger with content capture on (dev-otel: hot reload)

just docker # docker-compose down && docker-compose up -d

cargo test # run tests (very few exist today — see below)

cargo clippy # lint

```

There is no `just test` or `just lint` target — use `cargo test` / `cargo clippy` directly. Tests are sparse (`src/agents/agent/model/registry.rs`, `src/storage/memory.rs`, `src/skill/context.rs`); most correctness is exercised by running the binary, not by a test suite. For end-to-end checks, run the binary with an agent on the offline `dummyplug` provider (no keys needed): send `tools`, then a tool name for a sample request, then the JSON request to run the tool through the real loop (and `context` to echo the per-request context block the agent received, §6) — see `specs/008-dummyplug-provider/contracts/dummyplug-protocol.md`. The constitution requires this for any agent-observable change.

WebUI typecheck: `cd webui && npm run typecheck` (runs `react-router typegen && tsc`).

### Build gotcha: `build.rs` runs the WebUI build

`build.rs` shells out to `npm run build` in `webui/` on every `cargo build` **if** `webui/node_modules/` exists. If `node_modules/` is absent and `webui/build/client/` doesn't already exist, the build panics. Run `just install` first, or make sure `webui/build/client/` is pre-populated (this is how crates.io publishes work — see `include` in `Cargo.toml`).

## CLI subcommands

- `vizier run [-c <path>] [-d] [--port] [--workspace/--data-dir] [--storage sqlite] [--workers] [--ws-idle-timeout]` — start agents, scheduler, channels, and the command server. Works with **no config file** (config-less mode, see below). `sqlite` is the only supported `--storage` value; a deployment still configured with the legacy `filesystem` backend is migrated into sqlite automatically on first startup after upgrading (see `dependencies.rs`'s migrations below) — `--storage filesystem`/`VIZIER_STORAGE=filesystem` on a *new* invocation is rejected outright.

- `vizier shutdown [-c <path>]`

- `vizier onboard -p <path>` — interactive wizard that writes a seed `.vizier.yaml`.

- `vizier skill install|list|uninstall|update`

- `vizier agent ps` — list running agents. Agents themselves are created/managed at runtime via the WebUI or HTTP API — there's no `agent create` CLI.

### Config-less mode

Resolution order for the config file: explicit `-c` → `$VIZIER_CONFIG` → `./.vizier.yaml` → built-in defaults. In the no-file path, the workspace resolves to `$VIZIER_DATA_DIR` or `$HOME/.vizier`, and storage defaults to `sqlite`.

## Architecture

### Process startup (`src/cli/run.rs::run_server`)

A single tokio multi-thread runtime spawns five long-lived tasks off one shared `VizierDependencies`: `VizierScheduler`, `VizierChannels`, `VizierAgents`, `VizierCommandServer`, and the deps' own command handler (`VizierTransport::run`). Each runs for the life of the process, so if any one stops (error, panic or return), `run_server` logs it and **exits with code 1** for Docker/systemd to restart vizier whole, rather than run half-working (#51). A subsystem's `run` must therefore never return while things are healthy (`VizierChannels::run` awaits the HTTP server itself for this reason). The one tolerated failure is a command socket that can't be bound: it is logged and vizier runs on without `vizier shutdown`/`agent ps`. They all communicate through `VizierTransport` (`src/transport.rs`), an in-process message bus built on `flume` channels — per-agent request channels, a memory-ops channel, agent lifecycle commands, generic command request/response, dream commands, and file commands. There's no shared mutable state beyond what's routed through this transport plus the shared `VizierStorage`. It also carries one `tokio::sync::broadcast` of `SessionEvent { session, frame: SessionFrame::{Response, Job} }` (`publish_session_event` / `subscribe_session_events`): things pushed to whoever watches a session — background job changes, reactions, the frames of a turn a background report woke, and the frames of every WebUI turn. WebUI sockets subscribe (each filtering on its own session) and the WebSocket handler publishes a person's turn there rather than writing it to the socket that sent it, so a page reopened mid-turn, or a second tab, still receives the rest of it. The Discord/Telegram readers subscribe too, for the woken turns of their own channels (`channels/woken.rs`); their own people's turns still stream through a per-message forwarder.

### `VizierDependencies` (`src/dependencies.rs`)

Constructed once at startup: opens storage (sqlite connection or filesystem root), then runs one-time migrations (seed users → superadmin role, YAML providers → provider storage, per-agent MCP/shell config backfill, default CORE.md backfill for agents missing one). Cloned cheaply (`Arc` internals) into every subsystem.

### Agents (`src/agents/`)

- `VizierAgents` (`src/agents/mod.rs`) is the manager: on boot it registers every persisted agent's request channel and spawns a `Supervisor` (`src/agents/supervisor.rs`) per agent, then services `AgentCommand::Create/Update/Delete/HealthCheck`. Registration comes before anything is built, and the channel's receiver passes from supervisor to supervisor across updates, so requests sent while an agent starts or restarts queue instead of failing with "not registered" (this is also why `run.rs` builds `VizierAgents` before the scheduler). The supervisor builds the agent (`BuiltAgent::build`: Ollama pull, indexer, `VizierAgent::new`), runs `agent_process` in its own task so a panic ends the run and not the supervisor, and on a failed build or run records `Failed { last_error, failures, next_retry_at }` (visible through `HealthCheck`, `GET /agents/health` and `vizier agent ps`) and retries with backoff (2s doubling to 5m), answering requests with `ErrorKind::AgentUnavailable` meanwhile. Every stored agent has a supervisor whatever its state, so update and delete always work. Create and update **build before they persist or swap**: a config that can't start is refused and an updated agent keeps running on its previous config. The build runs as its own task off the command loop (`VizierAgents.builds`), so a slow one (an Ollama pull) never delays other agents' commands or health checks; while an agent has a build in flight, other creates/updates/deletes of it are refused as busy. When an agent instance stops (update, delete, restart) its running turns are **aborted**, not detached, and their senders get an `AgentUnavailable` error; requests still queued behind them are re-sent on the agent's channel for the next instance, or rejected if the agent was deleted (#52).

- `VizierAgent` (`src/agents/agent/mod.rs`) is the actual per-agent loop: builds its `VizierModel` (provider abstraction), `VizierTools`, `VizierSkills`, optional STT/TTS/image-gen, loads its CORE.md and owner profile, and runs the LLM request/response/tool-call cycle (`agents/process.rs` drives it against `VizierTransport`'s per-agent channel).

- Each agent owns its **own** MCP client set and shell instance: the shell is built from its `AgentToolsConfig` in `BaseTools::new()`, the MCP clients from the MCP integrations attached to it (see Integrations below), never shared between agents even when the definition is. There are no global MCP/shell singletons — this is deliberate (per-agent isolation). Each MCP server is an `McpServer` (`agents/mcp/`) that connects in the background, caches its tool list (5 min TTL, or until `tools/list_changed`), bounds connects and listings by timeouts, and reconnects with backoff after a crash or hang; a server that's down only drops its own tools from the turn, never fails it.

- `hook/` — lifecycle hooks (debug logging, thinking-block handling, tool-call handling, handover).

- `shell/` — local and Docker (`bollard`) shell execution backends behind one abstraction.

- `skill/` — reusable agent behaviors, separate from the top-level `skill/` module (skill *packages*, install/registry).

### Tools (`src/agents/tools/mod.rs`)

Tools implement the `VizierTool` trait: associated `Input`/`Output` types (both `JsonSchema + Serialize + Deserialize`), `name()`, `category()`, `description()`, `call()`. `category()` is required and returns `Option<ToolCategory>`, one of 15 variants in display order: `Memory, Identity, Scheduling, Agents, Skills, Files, Scratchpad, Web, Audio, Media, Discord, Telegram, Webui, Shell, Integrations`. Only the four direct tools and the two dream-journal tools return `None`; `every_tool_is_in_its_category` in `tools/mod.rs` pins the full table.

**Memory tool names** (`tools/vector_memory/`): `memory_search` searches and returns addressed **passages** (it was `memory_read`), and `memory_read` returns one whole document by `(bundle, path)` (it was `memory_detail`; `memory_detail` is retired, not aliased). The two renames are a breaking change and landed together, because tool names are the dispatch key. The other six `memory_*` tools are untouched. A blanket impl turns any `VizierTool` into the dynamic `VizierToolDyn` used for dispatch. Two built-in toolsets exist per agent — `default_toolset` (always-on: memory, workspace CORE read/write, scheduler — including `list_task_runs`/`get_task_run_detail` for an agent's own task results — skills, subtasks, session files, consult/delegate other agents) and `user_toolset` (conditionally added per agent config: fetch, http client, TTS/STT/image-gen, webui messaging). Both live in `BaseTools`, built once per instance. Integration tools (Discord/Telegram, addons, MCP) come from `IntegrationKind::contribute` into an `IntegrationTools` layer; `tools::compose(&BaseTools, Arc<IntegrationTools>)` builds the `ToolRouter`, `sandbox_toolset` and `categories`, and `VizierAgent` holds the result in an `ArcSwap<VizierTools>`. A turn loads one snapshot at entry and keeps it (system prompt and every tool call); `refresh_integrations()` swaps in a new one for the next turn, reusing unchanged contributions (and their `McpServer` connections). **Addon tools are namespaced** `<addon id>_<tool>` and listed under the addon id as category (Brave: `brave_search_web`, `brave_search_news`, category `brave_search`; the old `web_search`/`news_search` names are gone, not aliased); addon ids are reserved category names, so an MCP server named `brave_search` gets `brave_search_mcp`. MCP tools are named `<server>_<tool>` (sanitised by `python_identifier`), with the server key as their category, or `<key>_mcp` when the key is a built-in category's name. `ToolRouter::catalogue()` is the single source for naming, categories and dispatch: native names are reserved first, MCP names are assigned in `(server, tool)` order, and a taken name gets `_2`, `_3`, …. The old `mcp_<server>__<tool>`, `READ_CORE` and `WRITE_CORE` names are not supported anywhere (now `read_core`/`write_core`).

**Code mode** (`specs/007-code-mode-python-sandbox/`, `specs/014-tool-categories-code-mode/`): every agent is always in code mode; there is no `tools.python` config and no `ToolExposure`. The model is offered only `sandbox_toolset` (`think`, `execute_python`, `list_tool_functions`, `describe_tool_function`), and `VizierTools::call` refuses any other name. Every other tool is reached from scripts through `ToolRouter` (`VizierTools.router`). `VizierTools.categories` (built once from registered tools and configured MCP server keys, never live listings) feeds the `SANDBOX.md` system prompt, which lists the categories and stays byte-identical across turns; `list_tool_functions {category?}` and the in-script `list_tools(category=…)` list one category (an unknown one is `did_you_mean` data, or a catchable `ValueError` in a script). The tools are in `src/agents/tools/python/` (`execute_python`, `list_tool_functions`, `describe_tool_function`, plus `RouterBridge`, which runs script-made tool calls through `ToolRouter` + `ToolContext.hooks` + the per-tool timeout). The engine is `src/sandbox/`, which is agent-agnostic: a monty resume loop on `spawn_blocking`, the `SandboxBridge` trait, JSON⇄`MontyObject` conversion, and `ToolFunctionDoc` built from `ToolDefinition` (`llm::types::ToolDefinition`). Script failures come back as an `ExecutionReport` (data), never `Err`. The sandbox tools are never in `DREAM_TOOL_NAMES`; `dream_tools`/`dream_call` bypass `sandbox_toolset` and call the router directly.

**Scratchpad** (`specs/020-scratchpad-tools/`): `write_tmp`, `append_tmp` and `read_tmp` (`tools/scratchpad.rs`, category `scratchpad`, on `default_toolset`) keep named plain-text files private to one conversation, in the `scratch_file` table behind `ScratchStorage` (`storage/scratch.rs`), keyed `(agent_id, channel slug, topic or '', name)`. Limits (1 MiB per file, 64 files, a safe name rule) are checked in one storage transaction, and every refusal or miss is a `status` result, never `Err`. A conversation's files last while it has a turn running or queued: `agents/process.rs` clears them (awaited inline, so no turn can start in between) when a session goes idle, on `/abort`, and on instance stop for sessions with an empty queue (a handed-on queue keeps them). `dependencies.rs` clears every row at startup, and `delete_session`/`delete_agent` delete theirs. The three names are in `DREAM_TOOL_NAMES`; each dream stage is its own session, so its own scratchpad. `scratchpad` is a reserved category name (an MCP server keyed `scratchpad` becomes `scratchpad_mcp`).

**Background jobs** (`specs/012-background-subagent-results/`): `paralel_subtasks` (misspelt, kept — names are dispatch keys) and `delegate_agent` are **non-blocking and report back**. Both are thin wrappers over `BackgroundJobs::launch` (`src/agents/background/`, held on `VizierDependencies.background_jobs`), which records the job, sends each piece to its own session (`Subagent` for a batch, `InterAgent([origin, target])` for a delegation, always with a fresh topic), and leaves a runner awaiting them under a per-piece time limit (`timeout_secs`, default 600, max 3600; a timed-out piece gets the existing `abort` command). When every piece is final, one `VizierRequestContent::BackgroundReport` is sent into the originating session — it runs through the `Chat` arm (history, `ThinkingStart`) with zero auto-context passages, is queued behind a running turn like any request, and frontmatters as `sender: background`. Its response sender republishes frames onto the session-event broadcast, so the woken reply streams into an open WebUI topic, and the Discord/Telegram readers pick up their own channels' woken turns from the same broadcast (`channels/woken.rs`) and post them through the renderer a person's turn uses. Every other conversation (task, dream, inter-agent, subagent) is woken into history only; `src/agents/background/` never branches on `VizierChannelId`. `list_background_jobs` / `cancel_background_job` (and `POST …/topic/{topic}/jobs/{id}/cancel` for the tray) share `BackgroundJobs::cancel`, which aborts the pieces, cascades to jobs launched from the pieces' sessions, and never sends a report. Exactly-once is a guarded SQL transition out of `running`: the runner moves to `reporting` *before* delivering, so a job is either reported or cancelled, never both. Nesting is `VizierRequest.background_depth` (on the request, not the session — a woken turn shares the launcher's session), copied into `ToolContext.background_depth`; `launch` refuses at `MAX_BACKGROUND_DEPTH` (3). None of these tools are in `DREAM_TOOL_NAMES`; `consult_agent` is unchanged and still blocking.

**Client tools** (`specs/019-client-tools/`, `src/agents/client/`): a WebSocket client may declare tools of its own on any request (`VizierRequest.client_tools`/`client_categories`, `skip_serializing`, so never stored). `handle_socket` checks them (`client::declare::validate`, refusing with `client_error` before any turn starts), then swaps them for `VizierRequest.client` (`ClientBinding`: the declarations plus that socket's `ClientConnection`, `#[serde(skip)]` like `trace`). Only `VizierAgent::chat` turns them into tools: at turn start, if the connection is still open, `ClientTools::resolve` names them against `router.catalogue()` with the MCP rules (`client_category_name` → `_client` suffix on a clash with a built-in category, addon id or MCP server; `assign_names` for `_2`, `_3`), sends `client_tools_resolved`, and sets `ToolContext.client`. Every other `ToolContext` is `client: None`, which is what keeps dreams, handovers, tasks, background reports/pieces and inter-agent turns client-free. Scripts reach them through `tools::turn_catalogue`/`turn_categories` (router's, then client's), read by `RouterBridge`, `list_tool_functions` and `describe_tool_function`; `ToolTarget::Client` is dispatched by `RouterBridge` to `ClientTools::call`, never by the router. A call is saved as `SessionHistoryContent::ClientToolCall` **when made** (its history uid is the call id; skipped by `history_entries_to_messages`), written **directly** to the declaring connection (other sockets get an `answer: false` display copy via `SessionFrame::ClientToolCall`), and awaited under its timeout; a drop guard records `cancelled` if the turn is aborted. A declaration may set `deferrable`: a call made while the declaring socket is gone (or that drops before answering) becomes `deferred` (`defer_client_call`) and the script gets `{"status": "deferred"}`; `handle_socket` delivers deferred calls to the first socket on the session (on connect, and on `SessionFrame::ClientCallDeferred` for an already-open tab), claimed through `DeferredClaims` on `HTTPState`; deferred calls survive restarts (the startup sweep touches only `pending`) and their later result is never reported to the agent. Outcome and saved state live in the `client_tool_call` table (`ClientCallStorage`), filled into the entry on every read by `fill_client_calls` (like `fill_reactions`), deleted with the topic/agent, and swept to `interrupted` at startup. `VizierRequestContent::ClientEvent` (`client_event`: text, data, source call) starts a turn like `Chat`, frontmattered `sender: <person>`, `via: client_event`; it and `client_call_state` saves are rate-limited per conversation/call by the `RateLimiter`s on `HTTPState`. The WebUI registers its tools with `registerClientTool` (`webui/app/lib/clientTools.ts`), declares them on every request in `connectionStore`, and draws `turn.clientCalls` (from `groupHistory`, filtered by each registration's `renderAfterTurn`/`latestPerTurn` through `visibleClientCalls`) inside the reply bubble; its built-in `render_interactive_ui` renders agent HTML in an `<iframe sandbox="allow-scripts" srcdoc>` with a CSP meta and a `window.vizier` bridge (`components/client/InteractivePage.tsx`). The public protocol is `docs/src/api-integration/client-tools.md`; `scripts/client-tools-e2e.mjs` checks it end to end against a dummyplug agent.

**Adding a new tool**: create `src/agents/tools/<name>.rs` implementing `VizierTool` (pick its `ToolCategory`, and add it to the `every_tool_is_in_its_category` table), add `mod <name>;`, then `.tool(YourTool)` it onto `default_toolset` or `user_toolset` inside `BaseTools::new()`. If it should be available to the dream cycle, add its name to `VizierTools::DREAM_TOOL_NAMES`. **Adding an addon** (or any integration kind): implement `IntegrationKind` in `src/integrations/kinds/<id>.rs` (`listing` is its catalogue tile and settings form; `contribute` returns its tools with `addon: Some(id)`, each named `<id>_…` and returning `None` from `category()`), add an icon to `assets/integrations/`, and register it in `IntegrationKinds::builtin()`. No agent-form, API or page change is needed.

### Integrations (`src/integrations/`, `specs/016-integrations-section/`, `specs/017-agent-self-integrations/`)

Discord/Telegram bots, addons (Brave Search) and MCP servers are **integrations**, never fields on `AgentConfig`. **Each belongs to exactly one agent** (017): it is linked once, when created on that agent (`IntegrationStorage`, tables `integration` + `integration_attachment`), never attached elsewhere, and deleted with the agent. `PUT …/attachment {attached:true}` is `410 attach_removed`. Rows from 016 still linked to several agents (or none) are **legacy**: they keep working for each linked agent, only all-agents managers may change them, and deleting one from an agent's page only unlinks that agent (`unlink`); there is no data migration. Every kind implements `kind::IntegrationKind` (validate, redact/merge secrets, `missing`, `contribute`, `reader`, `platform`); nothing branches on a kind id. Rules enforced in the storage transaction: one connection per platform per agent and a token used once per platform (`409 platform_in_use_on_agent` / `token_in_use`), one configuration per addon per agent (`409 addon_installed`), no two MCP servers with the same name on one agent (`name_in_use_on_agent`; names are *not* unique install-wide). **Every write goes through `integrations/service.rs`** (`create`/`update`/`delete`/`set_grants`, taking an `Actor::Person` or `Actor::Agent`), used by the HTTP handlers and the agent tools alike: it validates, saves with its `integration_event` row (who — a `Requester` — what, which setting *names* changed, never values; the agent's session) in the same transaction, and publishes `IntegrationChanged` on the transport *after* commit; subscribers re-read storage (level-triggered). `VizierAgents` sends each affected agent's running instance a `refresh_integrations()` (via the supervisor's `CurrentAgent` handle, never the request queue) — an integration change never restarts an agent or aborts a turn. Channel readers are process-wide in `channels/connections.rs::ChannelConnections` (owned by `VizierChannels`), reconciled against storage at startup and on channel events; they never stop or start with an agent instance, and a `FatalChannelError` marks only that connection failed until its token changes. Reader health lives on `deps.channel_board`, which feeds `AgentHealthStatus.channels`. Access is decided only in `integrations/access.rs`: `all_agents:integrations:manage` manages every agent's integrations, `owned_agents:integrations:manage` those of the agents the person owns; **who created an integration grants nothing** (the 016 `shared`/`owned` scope is gone; the column stays, unread). `creator` (a `Requester`) is kept for display only.

**Agent self-management** (017): three per-agent **grants** (`integration_grant`, one row per family `mcp`/`addon`/`channel` switched on) let an agent manage its own integrations of that family through five code-mode tools in category `integrations` (`list_integration_kinds`, `list_integrations`, `create_integration`, `update_integration`, `delete_integration`, in `agents/tools/self_integrations.rs`). Grants live outside `AgentConfig` on purpose: `PUT /agents` rebuilds the agent and aborts its turns, a grant change only publishes `IntegrationChanged` (empty `integration_id`) so the tools recompose. Only someone who manages the agent's integrations may change a grant (`PUT /agents/{id}/integrations/grants`); `POST`/`PUT /agents` refuse a `self_management` key with `422 use_grants_endpoint`. The tools are contributed by the integration layer (`IntegrationTools.self_management`), so an agent without grants has no `integrations` category at all; each call re-reads the grants, acts only on the agent's own non-legacy integrations, and is absent from `DREAM_TOOL_NAMES`. Disabling or deleting the channel connection the current conversation is on succeeds with a `warning`. HTTP: `/agents/{id}/integrations` (list/create, `DELETE /{iid}`, `/grants`, `/events`) and `/integrations/{id}`, `/integrations/kinds`(+`/{id}/icon`, public). `POST`/`PUT /agents` refuse the 016 moved fields with `422 moved_to_integrations`.

### Storage (`src/storage/`)

`VizierStorageProvider` is a supertrait composing every storage concern (`IntegrationStorage`, `MemoryStorage`, `TaskStorage`, `TaskRunStorage`, `HistoryStorage`, `SessionStorage`, `StateStorage`, `UserStorage`, `AgentStorage`, `ProviderStorage`, `CustomProviderStorage`, `CatalogueCacheStorage`, `GlobalConfigStorage`, `DreamJournalStorage`, `DreamStorage`, `SessionFileStorage`, `BackgroundJobStorage`, `UsageStorage`). Provider tables: `provider_config` holds one `ProviderCredentials` JSON per `ProviderId`, `custom_provider` the operator-defined providers, `catalogue_snapshot`/`catalogue_logo` the refreshed catalogue and logos. `VizierStorage` type-erases the concrete backend (`storage/sqlite` — the sole `VizierStorageProvider` implementation) behind `Arc<Box<dyn VizierStorageProvider>>` and hand-forwards every trait method. **Adding a storage backend** means implementing every one of those traits for the new type, then `impl VizierStorageProvider for it`. The old `storage/fs` (`FileSystemStorage`) backend has been removed as a runtime option; its non-memory trait impls survive only as a read source for the one-time `migrate_filesystem_backend_to_sqlite` startup migration in `dependencies.rs`, for deployments upgrading from a pre-existing `--storage filesystem` install.

`MemoryStorage` is the one exception to "storage backend owns the bytes": memory concept documents are markdown files (YAML frontmatter + body) addressed by `(agent_id, bundle, path)`, read/written through the pluggable `storage::document::DocumentStore` trait (default: `LocalDocumentStore`, rooted at `{workspace}/agents/{agent_id}/memory/...`), with `storage::memory_bundle::BundleMemoryStore` as the single implementation of bundle/concept logic (link parsing, `index.md`/`log.md` maintenance, bundle export/import as `.zip`) and the sole `impl MemoryStorage for SqliteStorage`. SQLite still caches a derived, reconcilable **Memory Graph Index** (`memory_node`/`memory_edge` tables) so listing/graph/related-memory queries never need to read the documents themselves — only `memory_detail`, semantic search, and export ever call into `DocumentStore`. See `specs/004-memory-open-format/` for the full design.

**Passage-level retrieval** (`specs/009-memory-semantic-chunking/`): queries match **passages**, not whole documents. `storage/chunk.rs` splits a document body into structure-aware passages — `scan_blocks` (markdown blocks) then `pack_blocks` (size-packing under `ChunkLimits`, default 1200/400/2400 bytes); both are pure, sync and unit-tested, and `chunk_markdown` consults no embedder (the embedding-seam path was measured out, research Decision 1b). A third derived table, `memory_passage`, holds each passage's coordinates (ordinal, line span, byte span, `continues`, `content_hash`) — **never its text**, which is sliced from the document at query time. The `document_index` vec0 table keeps its DDL and changes only row granularity: one row per passage, keyed `{agent_id}/{bundle}/{path}#{ordinal}`. Absence of `memory_passage` rows for a document is the only marker that it is unconverted, which is what makes the per-agent conversion task resumable; drift is detected by comparing `content_hash`, not by re-chunking. `write_memory`/`delete_memory`/`delete_bundle`/`import_bundle` keep passages in step, and deletion is driven by *stored* ordinals so a shrinking document cannot orphan vectors. Passages are indexed in one batched `DocumentIndexer::add_document_indexes` call per save, with title/tags/heading-breadcrumb prefixed to each passage's index text (never to its returned text). Chunking or indexing failure is logged and never fails the save.

The one-time conversion of an existing corpus runs **per agent in `VizierAgents::spawn_agent`**, not in `dependencies.rs` — the indexer is built per agent from that agent's embedding config and does not exist where the other migrations run.

**Version history** (`specs/006-memory-version-history/`): CORE history lives in the `core_revision` table behind `AgentStorage` (`set_agent_core` upserts `agent_core` and records the revision in one transaction — `agent_core` is the sole home of CORE; the legacy `AgentConfig.core` field is migrated out on startup), memory history in `memory_revision` behind `MemoryStorage` (recorded inside `BundleMemoryStore::{write_memory, delete_memory, delete_bundle, import_bundle}`). Both are recorded *inside* the write path, so every caller only supplies a `RevisionOrigin` (who/why); the two kinds are deliberately separate implementations sharing only `schema/revision.rs` value types and `storage/diff.rs`. Direct on-disk edits to memory files are not versioned.

**Background jobs** (`BackgroundJobStorage`, tables `background_job` + `background_piece`, `ON DELETE CASCADE`): job and piece rows only — **no result text**; a piece's answer is the last message in its own session, read through `storage::background_job::piece_answer`, like a task run's. `dependencies.rs` sweeps `running`/`reporting` jobs and `running` pieces to `interrupted` at startup (a server restart loses in-flight jobs; the WebUI shows them as lost), and `delete_session` deletes the jobs launched from that topic.

### Channels (`src/channels/`)

`VizierChannel` trait: `async fn run(&self)`. Discord/Telegram readers are run by `channels/connections.rs::ChannelConnections`, one per enabled, assigned channel integration; each gets the small supervisor in `channels/readers.rs::supervise`: a `run` that ends is restarted with the agent supervisor's backoff, except on a `FatalChannelError` (a rejected token, Discord close codes that forbid reconnecting), which marks the reader `failed` until its token is replaced. Each reader's state shows in agent health (`AgentHealthStatus.channels`) and `vizier agent ps` (#50). Implementations: `discord/` (twilight), `telegram/` (teloxide), `http/` (axum — REST under `api/v1/`, WebSocket, JWT auth in `auth/`, and it also serves the built WebUI static files from `webui/build/client/`). `VizierChannels::run()` is where new channel spawns get registered.

**Reactions** (`specs/013-reaction-awareness/`): reactions live in `message_reaction` (`ReactionStorage`), keyed by the history uid of an agent `Response` — never in the history row's `data` blob, which `fill_reactions` replaces on every read (a one-time startup migration in `init_reaction_schema` moved the old blob arrays out). A reply's uid travels on `VizierResponse.history_uid`, set by `VizierAgent::chat` after the save; the WebUI keys messages by it, and the Discord/Telegram response loops (and `discord_send_message`/`telegram_send_message`) record a `platform_message_link` row for every message they post from it, chunks and attachments included. Every channel only translates its native event and filters its own bots, then calls `channels::reactions::apply`, which resolves, checks it is an agent reply, writes, and publishes the full new set as `SessionFrame::Reactions` (the WebSocket's ack and cross-tab sync). The agent sees reactions only as the `## Reactions` section of `context_md` (`system_prompt/reactions.rs::reaction_digest`, bounded) and in checkpoint handovers — the route into the dream cycle. **A reaction never starts a turn, by decision.** Reactions on unlinked messages (people's messages, anything posted before this feature) are dropped.

### Providers / models (`src/llm/`, `src/catalogue/`; `specs/015-provider-catalog/`)

vizier has its own provider layer; there is no rig. `src/llm/` is the transport: `types.rs` (`Message`, `Part`, `AssistantPart`, `ToolDefinition`, `CompletionRequest/Response`, `Usage`), `prepare.rs` (replaces attachments a model can't take with a text note, normalises system messages per dialect), five dialects in `dialect/` (`OpenAiChat`, `OpenAiResponses`, `Anthropic`, `Gemini`, `Ollama` — `encode`/`decode` pinned by golden fixtures recorded from rig 0.39), `HttpModel<D>` + `RetryingModel` in `model.rs` (backoff, `attempt_timeout`, the `prompt_timeout` deadline via `complete_until`, and `Retry-After` honoured; exhausted retries surface as `ErrorKind::RateLimited` / `ProviderUnavailable`), `auth.rs` (`CredentialSource`: `StaticKey`, `CopilotExchange`, `ChatGptAccount`), and `media/` (STT/TTS/image/embedding clients implementing the existing `Vizier*Model` traits). `resolve.rs` is the **only** place that turns `(ProviderId, model, overrides)` into a `ResolvedModel` (dialect, base URL, credential, headers, context window per research R8, caps, cost) — quirks live there and in `CredentialSource`, never as a `match` over provider ids in call sites. `VizierModel` (`src/agents/agent/model/`) is a thin `Arc<dyn CompletionModel>` wrapper; `dummyplug` (`model/dummyplug.rs`) is the offline built-in.

**Providers are data**, from the models.dev catalogue: `src/catalogue/` holds the seed embedded at build time (`build.rs` gzips `assets/catalogue/api.json`; logos are embedded too), the `Catalogue` handle on `VizierDependencies` (swappable snapshot), built-ins (`builtin.rs`: ollama, llama-cpp, chatgpt, dummyplug, elevenlabs, voyageai, hyperbolic), the SQLite cache (`cache.rs`) and the lazy refresh (`refresh.rs`). The refresh is **triggered only by a person's read** (`GET /catalogue/*`, `GET /providers`, `GET /providers/custom`) or `POST /catalogue/refresh` — never by agents, the scheduler or dreams; startup makes no network request. A provider id is `ProviderId` (`config/provider.rs`, a validated string; legacy names like `gemini` deserialise to `google`). Operators can add **custom providers** (OpenAI-compatible or Anthropic protocol, any base URL, own model list; `/providers/custom`). Credentials come from the saved `provider_config` row, then the catalogue's `env[]`, then the pre-015 env var names (`provider_keys.rs`). Changing or deleting a provider's credentials restarts the agents that use it (`AgentCommand::Restart`). Replies record their provider/model in `VizierResponseStats`, which the usage API prices into `estimated_cost_usd`.

### Scheduler (`src/scheduler/`)

Cron (`croner`) and one-time task execution, plus `scheduler/dream/` — a separate periodic "dream" cycle per agent (see `dream_interval` in agent config) that runs a restricted tool subset (`VizierTools::DREAM_TOOL_NAMES`) for unattended reflection/journaling.

**Task runs** (`specs/011-task-completion-reports/`): every firing is recorded as one `task_run` row (`TaskRunStorage`), opened before dispatch and closed from the `send_request` response channel the way `DreamScheduler` already does — `answered`, `no_response` (dispatch failed or the turn produced no message) or `interrupted` (swept at startup by `dependencies.rs`, since nothing can legitimately be running before the scheduler starts). A fired one-time task is **deactivated, not deleted**; deleting it before dispatch is what used to leave nothing to hang a result on. The `running` row *is* the overlap lock, replacing an in-memory `HashSet` that was inserted and removed within one loop iteration and so never actually spanned a run. Recording failures are logged and never abort the run.

`VizierChannelId::Task::to_slug()` renders its timestamp **in full**, so each firing gets its own conversation; it used to render `timestamp_subsec_nanos()` of a second-truncated time — always `0` — so every run of a task collided into `task__{slug}__0`. A run's response is never stored: it is the last `Response` carrying message content in that run's session, read through `storage::task_run::task_run_response`. Runs surface over `GET /agents/{id}/tasks/{slug}/runs` (key-set cursor on `(ran_at, id)`) and `.../runs/{run_id}/history`, and to the agent itself through `list_task_runs` / `get_task_run_detail` (both in `DREAM_TOOL_NAMES` — `is_non_user_channel` excludes `task__` from the sessions the dream cycle reflects on, so these are the only route by which a task's outcome reaches memory).

`Task.requester` (a `Requester` enum of `User(String)` / `Agent(AgentId)`) replaced the free-form `Task.user`: `POST`/`PUT /tasks` take it from the authenticated caller and ignore any `user` in the body, and the agent tools have it declare own-initiative versus a named person. A scheduled run's system prompt gains `system_prompt/scheduled_run.rs` and its frontmatter reads `sender: scheduler` with `task` and `requested_by`.

**The one thing to get right here**: the scheduled-run framing is selected on `VizierSession::is_scheduled_task()` — the session's *channel* — never on the request content kind. `VizierRequestContent::Unattended` (renamed from `Task`, since two of its three construction sites are the dream cycle) is sent by the dream cycle too, which carries its own framing; branching on it double-frames every dream.

### Telemetry (`src/telemetry.rs`, `src/llm/telemetry.rs`; docs: `docs/src/configuration/observability.md`)

OpenTelemetry export is a `tracing-opentelemetry` layer next to the fmt layer, installed in `main.rs` only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set (OTLP/HTTP; `OTEL_METRICS_EXPORTER=none` skips metrics). Spans therefore go through the same `EnvFilter` as logs. Span names follow the GenAI semantic conventions and are set through `otel.name`: `invoke_agent {agent}` (`VizierAgent::invoke_agent_span`, opened by `chat`/`dream_chat`), `chat {model}` (`llm/telemetry.rs`, around each `HttpModel` request, plus the `gen_ai.client.*` histograms), `execute_tool {tool}` (`tools/mod.rs`: `traced_tool_call` for the model's own calls, with the call id and the tool timeout; `ToolRouter::call` for calls made from scripts; `dream_call` uses the span-less `dispatch` because its loop already opened one), and `background_job {kind}` (`BackgroundJobs::launch`, kept open by instrumenting the runner, so pieces and the woken report turn are its children). Anything content-bearing (messages, system prompt, tool definitions, tool arguments/results) is recorded only when `llm::telemetry::capture_content()` is true (`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`): keep new content attributes behind it. Non-convention attributes are namespaced `vizier.*`.

**Trace propagation across the transport**: `VizierRequest.trace` (`telemetry::RequestTrace`, `#[serde(skip)]`, so never stored or accepted from clients) is filled by `VizierTransport::send_request` with the sender's current span context and the time it was queued; `invoke_agent_span` adopts it as parent and reports `vizier.queue.wait_ms`. So a request sent from inside a span (a tool call, a background job) continues that trace with no extra code, and one sent from a channel handler (no span) starts a new one. A new `VizierRequest` literal needs `trace: Default::default()` or `..Default::default()`.

### Usage accounting (`src/usage/`, `src/agents/agent/model/usage.rs`; `specs/018-usage-accounting/`)

Every answered completion is recorded by `VizierModel::complete_until` itself: one `usage_event` row (`UsageStorage`) per call, written in a spawned task so it never delays or fails the call, and never on `Err`. `complete`/`complete_until` take a **required `&UsageScope`** (`Turn` with the turn id and round, `Handover`, or `Vision`, plus the session), so a new call site must say what it is for. The activity is fixed at write time: `Handover` → compaction, `Vision` → vision, `Turn` → `VizierChannelId::usage_activity()`. The Channel-breakdown key is `VizierSession::usage_place()`. `prompt_with` and `dream_prompt` hold a `TurnGuard` (`VizierModel::guard()`); dropped without `succeeded()` (an error, the prompt timeout, an aborted task), it inserts the turn into `usage_failed_turn`. "Failed" is that join, never a column, so the mark and the turn's last insert can land in either order. `src/usage/` builds the report: SQL groups (`UsageGrouping`), Rust prices each `(provider, model)` group with the catalogue through the one `metrics::estimate_cost(&TokenCounts, …)`, and labels and links are resolved from storage only (no platform lookups). `migrate_reply_usage` (`dependencies.rs`) copied every saved reply's stats in once as `legacy` rows and set the state key `usage.accounting_since`. Session deletion keeps usage; agent deletion removes it. Not recorded: STT, TTS, image generation, embeddings, and the provider test ping (it bypasses `VizierModel`).

### Config layering

- `.vizier.yaml` (top-level key `vizier:`) is **seed config**, loaded once via `VizierConfig::load`. Supports `${ENV_VAR}` expansion (`shellexpand`). `dev.vizier.yaml` is the local dev config (already has working keys — don't treat it as a template to copy secrets from).

- There is no YAML `providers` section any more: a leftover one is ignored with a warning naming the env var to set. Provider credentials live in storage (Settings › Providers, `/api/v1/providers`) or env vars. `catalogue.refresh` (default `true`), `catalogue.ttl` (default `24h`) and `catalogue.url` control the catalogue refresh (also `--catalogue-refresh`/`--catalogue-ttl` and `VIZIER_CATALOGUE_REFRESH`/`VIZIER_CATALOGUE_TTL`). Agents are **never** defined in YAML — they're created/updated only through the WebUI/HTTP API and persisted to storage.

- `AgentConfig` no longer carries `discord_token`, `telegram_token`, `tools.discord`, `tools.telegram`, `tools.brave_search` or `tools.mcp_servers` (spec 016): those are integrations. Rows saved before still load (unknown keys are ignored) and drop the keys on their next save.

- `AgentConfig` also carries `chunking` (`ChunkLimits`: `target_size`/`min_size`/`max_size`) and `auto_context` (`chat_passages` default 5, `silent_read_passages` default **0**, `threshold` default 0.6, `size_cap`, `per_document`). `SilentRead` fires for every non-mention Discord guild message and every Telegram group message, so its budget is separate and off by default — its cost would otherwise scale with channel traffic. The automatic-context threshold is provisional and meant to be derived by replaying stored session history.

- CLI flags (`--port`, `--workspace`/`--data-dir`, `--storage`, `--workers`, `--ws-idle-timeout`) override whatever the config file loaded, applied via `VizierConfig::apply_overrides`.

- Docker env vars (`VIZIER_CONFIG`, `VIZIER_DATA_DIR`/`VIZIER_WORKSPACE`, `VIZIER_PORT`, `VIZIER_STORAGE`, `VIZIER_WORKERS`, `VIZIER_WS_IDLE_TIMEOUT`, `VIZIER_JWT_SECRET`, `VIZIER_EXTRA_ARGS`) are translated to CLI flags by `docker-entrypoint.sh`, which then `exec`s the binary so signals propagate.

### Agent identity: CORE.md, not `.agent.md` files

Each agent's persistent "self" document is `CORE.md`, stored in storage (not the filesystem) and seeded from `templates/CORE.md` (`constant::CORE_MD`) on agent creation, with a startup backfill migration for any agent missing one. `templates/agent.template.md` / `vizier.agent.md` at the repo root are a legacy/reference identity template format — current agents read/write CORE via the `READ_CORE`/`WRITE_CORE` tools against storage, not a file on disk.

## Key conventions

- **Migration & backward-compat code is tracked for removal before 1.0.0** in [#82](https://github.com/vizier-lab/vizier/issues/82). Keep it working until then: don't delete a migration, legacy shim or `legacy`/`migrated_from` field as "dead code". When you add new code of this kind (a startup migration in `dependencies.rs`, an `add_column_if_missing`/data move in an `init_*_schema`, a `Legacy*` type, a lenient deserializer, an old-name alias, or an API refusal kept for old clients), add a checklist entry to #82 naming the code and anything that depends on it.

- **Errors**: `VizierError(pub String)` (`src/error.rs`); the crate-wide alias is `crate::Result<T> = Result<T, VizierError>`. Convert external errors with `throw_vizier_error(prefix, err)`. Avoid `unwrap()`/`expect()` outside tests and `main`'s bootstrap.

- **Logging**: `tracing` only (never `println!`) — `tracing::info!`, `warn!`, `error!`, etc. `main.rs` sets per-crate directive overrides (twilight, sqlite, reqwest, hyper, bollard, rmcp, etc. quieted to `error`/`off` by default unless `RUST_LOG` is set).

- **Extensibility is trait-based**: new tools/channels/storage backends/providers get added by implementing the relevant trait and registering the impl in its module constructor — not by branching on type inside existing dispatch code (see `.specify/memory/constitution.md` for the fuller rationale if present).

- **Conventional commits**: `feat:`, `fix:`, `doc:`, `perf:`, `refactor:`, `chore:` — changelog is generated by `git-cliff` (`cliff.toml`). Breaking changes flagged `[**breaking**]`.

- **Per-agent isolation**: MCP servers and shell config live on `AgentToolsConfig` per agent, not globally — don't reintroduce a global MCP/shell singleton.

## WebUI (`webui/`)

React Router v7 + React 19 + TypeScript + Tailwind v4. State via Zustand-style stores in `app/hooks/*Store.tsx`. Recharts for charts, highlight.js for syntax highlighting, MDX editor for markdown. Build output (`webui/build/client/`) is what the Rust HTTP channel serves at runtime — see the `build.rs` gotcha above for why you generally need `npm install` done before `cargo build` will succeed.

<!-- SPECKIT START -->

For additional context about technologies to be used, project structure,

shell commands, and other important information, read the current plan:

`specs/021-user-invokable-skills/plan.md`

<!-- SPECKIT END -->