AGENTS.md@backend · git:20261110.fbe7f63 · 2026-11-10 · sha256 0ba035b7cfc865d9

AGENTS.md@backend git:20261110.fbe7f63A

Immutable. This exact content never changes and is served at /api/v1/blob/0ba035b7cfc865d9.

# Duct Backend — agent instructions

Python reporting and synthesis backend for Duct.

## Product role

Per the MVP plan (duct-cloud, private), this backend is the actual product engine:

- read from client-owned destinations with read-only access
- normalize data into typed internal models
- compute signals and comparisons
- synthesize findings into structured output
- deliver via email and alerts

The web app owns HTML rendering. The backend produces JSON payloads only — it does not render HTML.

## MVP architecture

### Current stack

- **AI synthesis:** Every agent runs on V1 (LangChain 1.x / `deepagents`) — the only
  engine. Engine selection is still per request, defaulting from the
  `generate_engine` env var, because stored preferences and requests carry the
  string; `resolve_engine` folds any other value back to `v1`.

  **The consolidation is finished.** Per the engine consolidation review
  (duct-cloud, private), every agent moved to one harness — LangChain 1.x /
  `deepagents` — because customers bring their own model (OpenAI / Gemini /
  Claude / xAI / OpenRouter, an Ollama server, or any OpenAI-compatible endpoint) and the Claude Agent SDK was Anthropic-only by
  design (upstream issue #410, closed `not planned`). Insights
  (`agents/insights/v1/runner.py`) and content (`agents/content/v1/runner.py`)
  are `deepagents` sessions; audit's runner is on the `create_agent` rung
  (`build_session_agent`) driven by the same shared `DeepSession`.

  **V3 (Claude Agent SDK) was removed** last, because it was the hardest: the
  project-scoped audit ran it unconditionally, and four capabilities existed
  nowhere else. Each was ported first — a chat loop and `run_resume` (both
  DeepSession's, which audit now uses), the project artifact library
  (`agents/core/artifact_tools.py`), and the competitor-research pass
  (`agents/audit/enrichment.py`, now `create_agent` + web tools on any
  provider). Verified live on Gemini and OpenAI against the audit rubric before
  the delete.

  One capability genuinely went with it, deliberately: a Claude *subscription*
  (`sk-ant-oat…`) authenticates only through the CLI, and the Messages API
  rejects it (Anthropic disabled third-party OAuth in Feb 2026). Claude needs an
  ANTHROPIC_API_KEY everywhere now. `agents/content/persistence.py` detects the
  token prefix and says so once rather than failing a call per turn.

  **How content and insights got there**, recorded because the moves are the
  pattern any future port follows. Content V3 was removed once its port landed. What moved, and where it went:
  the SDK's `Agent` tool became `deepagents` sub-agents dispatched through `task`
  (`agents/content/subagents/`, now framework-free dicts); the in-process MCP server
  became the LangChain binder `build_content_tools_lc` with the tool bodies
  unchanged; the CLI's `WebSearch` / `WebFetch` became `agents/core/web_tools.py`
  — both as Duct tools, on the rule image generation already set: **a capability
  the running model may not have is a Duct tool, not a provider feature every
  model must support.** The one exception is a built-in that survives a real
  tool-calling loop, and Anthropic's is the only one that does, so it is bound
  there (versioned per model — Opus 5.5 and Sonnet 5.5 take `web_search_20260209`,
  the rest the basic variant). Every other provider gets Duct's own `WebSearch`,
  an ordinary function tool over an isolated grounded Gemini call
  (`service/google/gemini/search.py`), because Gemini refuses `google_search`
  alongside function declarations on 2.5 outright and on 3.x without a
  `tool_config` flag that langchain-google-genai drops whenever `tool_choice` is
  set. `tests/test_web_search.py` holds that matrix, measured, as `live` tests;
  `AskUserQuestion` became a checkpointed `interrupt()`, so a question survives a
  redeploy; and the thread is keyed on the conversation, so a resume continues it
  rather than re-priming from the transcript (the DB re-prime remains for
  conversations recorded before the thread was durable).

  **Images are a second provider inside a content run, and not a fixed one.**
  The conversation runs on whatever key the user brought for chat; the pictures
  run on whatever *image-capable* key they brought — Gemini, OpenAI, xAI or
  OpenRouter, in that order of preference
  (`agents/models.IMAGE_PROVIDER_ORDER`; the gateway is last because a
  first-party key for the same model is one hop fewer). The seam is
  `service/images/`: one request shape, one `ImageAPIError`, one factory
  (`image_client_for`), four backends (`service/google/gemini/client.py`,
  `service/openai/images.py`, `service/xai/images.py`,
  `service/openrouter/images.py`). `routes/content.py`
  resolves the image run once per session (`agents/engines.resolve_image_run`)
  and stashes provider + key on it; the tools spend that or decline, and the
  decline names every provider that would unblock the user. The agent's tool
  schema still defaults to the Gemini model id, so `image_model_for` swaps in
  the resolved provider's default rather than refusing — the agent asked for an
  image, not a Google image. Adding a backend means a client module, an
  `ImageModel` entry whose prefix `provider_of` recognises, a
  `DEFAULT_IMAGE_MODELS` row, and a branch in
  the factory; `/providers/status` and `/models/catalogue` derive the settings
  page's Images row from those same tables, so nothing in the browser lists a
  model.

  **The OpenRouter backend is a short list, not a catalogue.** They front 52
  image models; `ImageModel` names four, because it is a Pydantic enum in the
  content agent's tool schema and opening it to free-form slugs — the way the
  *chat* catalogue deliberately does — would let a run invent a model and fail
  on a slide. Three of the four reach vendors no first-party key here can
  (Seedream, Flux, Recraft, the last emitting SVG); the fourth is the cheap
  Gemini workhorse. Adding one is an enum line plus a `_CAPS` row in
  `service/openrouter/images.py`, and that row is not optional: the four models
  disagree about `resolution`, ratio lists and how many images one call may
  return, so every request is clamped to the table and the clamp is logged
  rather than sent hopefully. The ChatGPT-subscription route (`agents/core/codex.py`) is deliberately
  not an image backend: it can draw through Codex's hosted tool, but OpenAI's
  own docs scope subscription sign-in to Codex products and it is Plus-and-up,
  per-minute-quota'd, and unofficial — the wrong thing to put a customer's slide
  deck on.

  **Video is the same seam, with one rule images do not need** (issue #284).
  `service/videos/` mirrors `service/images/`: one request shape, one
  `VideoAPIError` carrying a *code* (a filter refusal, a refused realistic face,
  a timeout, no credit — each asks the agent for something different), one
  factory, two backends — Veo on a Gemini key (`service/google/gemini/veo.py`)
  and Seedance through OpenRouter (`service/openrouter/videos.py`), resolved by
  `agents/engines.resolve_video_run` on the image rules and a `video_model`
  saved pick. `service/videos/caps.py` fits every request to what its model
  makes (lengths, ratios, 1080p only on an 8 s Veo clip) and says so in
  sentences the agent relays. The rule images do not need: **a clip takes
  minutes and costs dollars**, so `generate_video` holds no DB connection while
  it waits, bills each clip as a `media`-scope `TOKEN_USAGE` (Veo from
  `VIDEO_PRICE_PER_SECOND`, Seedance from the cost OpenRouter reports), and the
  content runner caps it at three a turn (`RunLimits.per_tool_run_limits`).
  A clip is a `content_assets` row with a `video/*` type; `content_posts.video`
  names the take that publishes (`agents/content/video.py`), and publishing a
  video post sends that file alone.

  Two consequences, stated rather than discovered later. **Content on Claude now
  needs an API key** — `routes/content.py` refuses the subscription credential with
  the same 402 the browser already handles. And **the model only sees the images it
  generates on Anthropic**: image blocks inside a tool result are accepted there and
  rejected by the OpenAI chat API, so `VISION_PROVIDERS` decides whether the tools
  return pictures or URLs, and the system prompt says which. Where it does see
  them, `SeenImagePruneMiddleware` (`agents/core/lc.py`) swaps the base64 for a
  note after the model call that looked at it: the thread is durable and the
  Postgres saver writes the whole `messages` channel per superstep, so a
  picture left in state would be copied into every later checkpoint.

  **Insights V3 was removed** earlier for a different reason: nothing dispatched it.
  Both live routes (`routes/generate.py`, `routes/agents.py`) drive
  `AutonomousInsightsRunner`, while the V3 runner still claimed parity with the older
  `GenerateInsightsAgent` fetch/synthesize pair — an interface the routes had already
  left behind.

  **`GenerateInsightsAgent` and its tool registry were removed too**, for the third
  time for the same reason: no route dispatched them. With it went
  `agents/insights/tools.py` (per-connector `StructuredTool` factories) and
  `agents/insights/registry.py` (`goal_relevance` scoring that ranked a set of 12
  entities down to 8 — selection pressure that never existed). The autonomous runner
  reaches every entity through `FetchData(entity_id=…)` against the catalog, so the
  catalog's dispatch key was renamed `tool` → `fetch_fn`: it names an internal
  function, and only looked like a tool reference while those tools existed.

  **A ChatGPT plan is now a live OpenAI credential**, wired where the note above
  said it should be: `agents/core/lc.resolve_chat_model` sends a subscription
  credential to `_ChatOpenAICodex` (`agents/core/codex.py`), so every runner gets
  it. The credential is the *access token* the desktop shell minted — it arrives
  in `X-Provider-OpenAI` beside `X-OpenAI-Account-Id`, `service/auth.py` packs the
  two into one string, and the shape (a JWT, not `sk-…`) is what routes it.
  `ProviderKey.source` is `subscription`; it is the user's own plan, never billed
  to Duct. The refresh token never reaches the backend, `PUT /providers/openai/key`
  refuses to store a token, and background jobs (no request, no header) still
  need an API key. `CHATGPT_AUTH_ENABLED=false` is the kill switch: the app
  reads it from `/api/providers/status` to hide the sign-in, and
  `get_user_provider_keys` drops a plan token it is still sent, so a shell that
  signed in earlier cannot keep using it — the Codex backend is undocumented
  and the path can stop working without notice.

  So a shared change is made once. Claude remains a first-class *model* through
  V1, which is why retiring its SDK cost no model coverage.

  **Which model, on whose key, is one function.** `agents/engines.resolve_run_model`
  is engine → provider → model → key for every V1 runner — including the rule that a
  lone bring-your-own key chooses its own provider. It lived in
  `agents/insights/setup.py` until content became the second runner that needed it;
  a second copy of that rule is the copy that eventually spends the wrong key.

  **V2 (Google ADK) was removed.** Not on framework merit — ADK is actively developed and
  Google-backed — but because nothing dispatched its runner: `routes/generate.py` had been
  hardcoded to V1, so selecting "v2" in the UI silently served V1 while claiming otherwise.
  Its differentiators (Vertex Agent Engine deploy, `adk web`, native A2A, built-in evals)
  do not intersect this stack, and its weakest axis — provider breadth — is exactly what V1
  exists for. Its defaults were identical to V1's, so `resolve_engine` folding a stored
  `"v2"` back to V1 changed no behaviour. `agents/insights/schema_compat.py` outlived it
  for a while — it was never ADK-specific — but it went with insights V3, its last caller.
  V1 asks the provider for a typed object via `with_structured_output`, so nothing needs
  to parse a synthesis out of raw text any more.
- **Ingestion:** Direct Google API clients (`google-ads`, `google-analytics-data`, `google-api-python-client`). Async concurrent fetching in `service/pipeline.py`.
- **Normalization:** Lightweight Python pipeline — raw API response → typed Pydantic/SQLModel brief models. No query layer or transforms yet.
- **Database:** PostgreSQL on Railway — SQLModel ORM, Alembic migrations, `psycopg` driver.
- **Auth:** JWT for users; Google OAuth for connector linking (Ads, GA4, GSC, Sign-In), and a GitHub App for GitHub, with a pasted fine-grained token where no App is configured (self-host). Project access is by membership (`project_members`), not by `projects.user_id` — always go through `service/membership.py`.

  **`validate_api_key` is not an authorization boundary.** `DUCT_API_KEY` ships to
  the browser as `NEXT_PUBLIC_DUCT_API_KEY`, so it proves "this is the Duct app"
  and never "this caller owns that row". A router mounted behind it *looks*
  protected and is not. Any endpoint that reads or writes a project-scoped row
  therefore needs `get_current_user` **plus** a membership check on top:

  - a project named in the request → `get_project_for_user`
  - a row addressed by its own id → `get_project_row_for_user`, which reads the
    project off the **row**. An endpoint that takes a row id and trusts a
    `project_id` from the request is letting the caller vouch for themselves.
  - a listing → scope it to the caller (`accessible_projects`), never to an
    unfiltered query parameter.

  404, not 403, for a non-member, so the reply is not an oracle for which ids
  exist. `routes/artifacts.py` is the reference; `routes/content.py` declares
  `get_current_user` **on the router** so endpoint 45 cannot be written without
  it, and `tests/test_content_access.py` asserts that property directly.
- **Email:** `service/email/` — one seam (`send_email`) over swappable providers in
  `service/email/providers/`, the only place a mail vendor is named. `EMAIL_PROVIDER`
  picks one; unset takes the first with credentials (Cloudflare, then SMTP) and
  falls back to `console`, which logs, so dev/CI/self-host need no vendor account.
  Same shape as the app's analytics seam, for the same reason: a fork swaps one file.
  SMTP is the open-standard provider — any mail server or vendor relay — and the
  way to read local mail as it would arrive: the "Mailpit" editor task (`brew
  install mailpit`) catches it on `:1025` with an inbox at `http://localhost:8025`.
  Resend's own provider was removed in favour of it (#460); Resend takes SMTP too.
- **Observability:** PostHog for product events, error tracking and AI call
  metadata, behind one seam (`service/analytics.py`, the only file that
  imports PostHog): routes call `capture_event`, `get_current_user` attributes
  the request with `identify_user` (the account UUID, the same distinct id the
  app sends), uncaught exceptions are caught by the ASGI middleware and
  `logger.exception` records by a root handler, and the AI callback runs in
  privacy mode, so prompts and replies never leave. Off without
  `POSTHOG_API_KEY`, and off on a loopback `API_PUBLIC_URL` unless
  `POSTHOG_ENABLE_LOCALHOST` is set. And OpenTelemetry tracing of every
  agent turn, model call and tool call (`agents/core/telemetry.py`), shipped
  over OTLP/HTTP to whatever `OTEL_EXPORTER_OTLP_ENDPOINT` names and off when
  it is unset. Locally that is Jaeger (`scripts/jaeger.sh`, Apache-2.0; it
  replaced Phoenix, whose Elastic licence is not open source): the "App + API
  + Jaeger" and "Desktop + API + Jaeger" launch compounds start it
  and point the API at `http://localhost:4318` (the desktop window talks to
  that same API, so it needs no exporter of its own), and every FetchData and verifier
  dispatch is a span with its own latency. Jaeger reads the `gen_ai.*`
  attributes, lists the spans under the `duct-backend` service, and keeps them
  on disk for a week. Its MCP server is registered in `.mcp.json` at
  `/api/ai/mcp/` on the UI port (16686), so an agent can read the traces of a
  slow run instead of querying the transcript table. Every log line carries a request
  id (`[a1b2c3d4]`, from `X-Request-Id` when the caller sends one, minted
  otherwise, echoed in the response); an agent run logs under the id of the
  request that created its session, so one grep follows one press of Send.
  Each turn ends with a `turn 198.0s: 4 tool calls …` line naming the slowest
  three, which answers "where did the time go" without a SQL script.
- **Response headers:** `utils/security_headers.py` gives every response
  `nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer` and
  `Content-Security-Policy: default-src 'none'; frame-ancestors 'none'`, the
  last because nothing here is a page. An HTML response gets
  `frame-ancestors 'none'` alone, for FastAPI's `/docs`, which loads Swagger
  UI from a CDN. A route that needs a different value sets the header itself
  and the middleware leaves it. Pure ASGI like `AccessLogMiddleware`, so SSE streams
  pass through unbuffered; `tests/test_security_headers.py` holds both.
- **Hosting:** Railway — auto-deploys from `main` via GitHub integration; `railway.json` defines Railpack build + uvicorn start.
  `railpack.json` sits beside it and configures the **builder**, where
  `railway.json` configures **Railway**. It exists for one line —
  `deploy.aptPackages: ["...", "libexpat1"]` — and every character of it is
  load bearing, including the `"..."`. Railpack builds with a mise-installed
  CPython, then assembles a slim runtime image carrying only the apt packages
  it inferred from the dependency graph (`libpq5`, from psycopg). That
  interpreter is dynamically linked against `libexpat.so.1`; nothing in the
  graph implies it, so without this the ELF loader fails before Python starts
  and **the container dies on its first line**.
  **`"..."` is not decoration.** Railpack arrays *replace* the inferred value
  rather than extend it; `"..."` is its spread syntax. Writing
  `["libexpat1"]` therefore drops `libpq5` and everything else Railpack
  worked out.
- **Runtime commands must name the venv binary, never `poetry run`.**
  `startCommand` and `preDeployCommand` both run in the deploy image, and
  `poetry` is not a real program there — it is a mise shim onto a private
  virtualenv that mise built with the *builder* image's system interpreter.
  Railpack copies `/mise/installs` into the runtime image but the runtime base
  carries no system Python, so that interpreter is a dangling symlink and the
  container dies with `Could not find platform independent libraries` /
  `Failed to import encodings module` before poetry prints a single line.
  Nothing in the message mentions poetry, which is why this cost three weeks
  and two wrong fixes. The app's own `/app/.venv` is unaffected — it was built
  by the mise CPython that *is* copied — so `/app/.venv/bin/uvicorn` and
  `/app/.venv/bin/python` work where `poetry run` cannot. This surfaced when
  Railpack's Debian base moved to trixie (2026-08-16); every backend deploy
  between then and the fix failed identically, and production served a June
  image for three months because a failed deploy leaves the old one running.
  The other trap: the build **succeeds** and the image pushes, so Railway
  reports a failed deployment that looks like a build failure and is not. Read
  the *deploy* logs (`railway logs --deployment <id>`), not the build logs.
  Also check the deployment is not `SKIPPED` — `railway.json`'s
  `build.watchPatterns` gates whether a push builds at all, so a file it does
  not list (this one, once) can leave a fix sitting on `main` doing nothing.
  JSON has no comments, which is why this note lives here.
- **CI:** GitHub Actions (`backend.yml`) — Ruff lint + pytest on every PR and push to `main`.
- **Tests:** `make test` must stay offline and under two minutes; it is the
  gate on every merge and the thing an agent runs after every change. Two
  rules keep it that way. A test that needs a provider key, a network, or a
  binary on `PATH` is marked `live` — a `skipif` on the key alone is not a
  gate, because `get_configs()` reads `backend/.env.local`, so a developer with
  a key there fires a paid, minutes-long call from a plain `pytest`. And the
  V1 agent loop is driven by the fakes in `tests/fakes.py` (`ToolCallingFake`
  and its failing variants, `fake_llm`, `tool_names`) plus the `emitted`
  fixture in `conftest.py`: the real harness runs, only the model is canned.
  Assert on events, tool names and payloads, not on prompt prose — a wording
  test fails on every copy edit and catches nothing an eval would not.
- **The offline suite cannot open a socket.** `conftest.py`'s autouse
  `no_network` fixture raises on `socket.connect` for anything not marked
  `live`, so a test that forgets to fake its transport fails naming the seam it
  forgot rather than passing on the author's machine and nowhere else. It
  blocks loopback too, deliberately: an agent sandbox or a corporate runner
  exports `HTTPS_PROXY=http://localhost:<port>`, so a guard that waves loopback
  through waves the whole internet through, silently, in exactly the
  environments most likely to be holding a key.
- **Fake at the seam the vendor gives you, and put the fake in
  `tests/fakes.py`.** For the sync reporting connectors that is the vendor's own
  `api()` wrapper; for the retry loop underneath them it is `service/rest.py`,
  covered directly by `tests/test_rest_transport.py` because five connectors
  share it and a regression there lands on all of them at once. For Google Ads
  it is `FakeAdsClient`, which keeps the library's local machinery — `get_type`,
  `enums`, `copy_from`, the GAPIC path helpers — and replaces only
  `get_service`. That distinction is the point: a stub made of attribute bags
  passes while the field is misspelled and the enum is not a member, which is
  the whole class of bug worth catching in code that changes what a customer
  spends. Building a real `GoogleAdsClient` refreshes OAuth against Google at
  construction time, so `FakeAdsClient` is also the only offline way into those
  executors at all. GA4 has the same problem for the same reason —
  `discovery.build()` fetches the discovery document before it returns a client
  — and `FakeDiscoveryService` answers the fluent chain by dotted call path
  (`"properties.keyEvents.list"`). The GTM fake in `test_execution_policy.py`
  stays where it is on purpose: it keeps container state so it can answer a read
  that follows a write, which is a different job from replaying canned answers.
- **A fetcher's test fakes the transport, not the client.** The client-level
  fakes above are right for an executor, where the mutation it sends is what
  matters. They never run the code that *builds* a read request, and that is
  where GA4 landing pages were broken for six weeks (`StringFilter` imported
  from the wrong module) with every test green. So each read fetcher has a
  request-shape test one layer down: `FakeWire` replaces `httpx.request`
  beneath `service/rest.py` so a whole Meta, Apple, Stripe, RevenueCat or
  GitHub pull runs with the vendor's own encoding, headers and pagination real
  (`test_rest_connector_requests.py`); `RecordingHttp` plus
  `discovery_build_offline` let `googleapiclient` build Search Console and the
  GA4 admin API from the discovery document it ships, so method names and
  parameters are validated offline (`test_gsc_fetchers.py`); the GA4 Data API
  test fakes only `BetaAnalyticsDataClient` and lets the real request types
  build the report; and `FakeAdsClient.search_stream` answers the Google Ads
  read fetchers with real `GoogleAdsRow` protos and logs the GAQL in `queries`
  (`test_google_ads_fetchers.py`). When you add a fetcher, add one of these
  with it — assert on the request that would have gone over the wire, then on
  the parsed rows. `test_vendor_contracts.py` covers what none of them can:
  that every lazily imported SDK name and every GAQL field still exists in the
  installed package.

### Desktop (local sidecar) mode

The backend runs in two shapes from one codebase. Railway is unchanged; the
desktop build runs the same FastAPI app as a sidecar on the user's machine —
see the engine consolidation review (duct-cloud, private) §7–8.

- **Entrypoint:** `local_server.py`. Sets `DUCT_LOCAL=1`, resolves the per-user
  data dir, binds **127.0.0.1** on an OS-assigned port, and prints a single JSON
  handshake line on stdout before starting uvicorn:
  `{"duct_sidecar":1,"url":...,"port":...,"api_key":...,"data_dir":...}`.
  The Tauri shell must read the port from that line — never assume one.
  The shell also passes `--exit-on-stdin-eof` with a pipe only it holds, so
  the sidecar stops itself when the shell ends however it ends (SIGTERM to
  uvicorn, then a hard exit two seconds later). Off by default: a sidecar
  run by hand reads a terminal or `/dev/null`, and neither may stop it.
- **Data dir** (`utils/appdirs.py`): macOS `~/Library/Application Support/ai.getduct.desktop`,
  Windows `%APPDATA%\Duct`, Linux `$XDG_DATA_HOME/duct`. Created `0700`.
  Override with `--data-dir` or `DUCT_DATA_DIR`.
- **Local mode defaults** (`Configs._apply_local_mode_defaults`): SQLite at
  `<data_dir>/duct.db`, uploads at `<data_dir>/uploads`, `init_db_on_startup=True`
  (no Alembic on a laptop). Each is only filled when unset, so
  `DATABASE_URL=postgresql://…` still works for a developer running local mode.
- **Local API key:** minted once by the desktop shell and held in its keychain,
  handed down as `DUCT_LOCAL_API_KEY` (the JWT secret travels the same way); a
  headless sidecar with no shell falls back to a `0600` file at
  `<data_dir>/local-api-key`. Exported as `DUCT_API_KEY` so the existing
  `validate_api_key` gate applies unchanged. It only stops other local processes
  driving the sidecar (`docs/engineering/credential-storage.md`).
- **Secrets arrive on stdin, not in the environment.** With
  `--secrets-on-stdin` the shell writes one JSON line
  (`{"duct_secrets":1, "DUCT_LOCAL_API_KEY": …, …}`) on the pipe it then
  holds open for the watchdog, and `bootstrap()` reads it before anything
  else. Precedence is that line, then the environment (the headless and
  developer path), then the file. Once `_report_environment()` has built and
  cached `Configs`, `scrub_secrets_from_environment()` removes every name in
  `SECRET_ENV_NAMES` from `os.environ`, including the `DUCT_API_KEY` and
  `JWT_SECRET` it derived — a process environment is readable by other
  same-user processes and inherited by subprocesses. That scrub is safe only
  because every consumer goes through `get_configs()`; a new reader of one of
  those names from `os.environ` at request time would find it gone, so route
  it through `Configs` instead.
- **JSON columns must use `models/columns.py::json_column()`**, never
  `postgresql.JSONB` directly — raw JSONB fails to compile on SQLite. The variant
  still renders JSONB on Postgres, so it produces no Alembic diff.
- **Never name a module `models/types.py`** — it shadows the stdlib `types`.

### Before adding a table, an event kind or a log

Name which existing store already records it, and propose new storage only
when none does. Four overlap on purpose and each answers one question:

| Question | Store |
|---|---|
| what was said, what did each tool return | `agent_events` |
| who did what to a project, when | `activity_logs` |
| what state is a change in right now | `execution_change_sets` |
| what the agent knows and cannot re-derive | `project_memories` |

The full list, one line per table, is
[`docs/engineering/data-model.md`](../docs/engineering/data-model.md), and
`tests/test_data_model_doc.py` fails when a new table is missing from it.
Why this rule exists: on 2026-09-24 a "decision event" was designed for chat
approve/reject rows while `activity_logs` already recorded every one of them,
with its actor and conversation. The gap was the reader, not the record.

### Database migrations

Schema changes are applied **manually** with Alembic — a normal local dev step,
distinct from an app deploy (the global "deploys go through CI/CD" rule is about
shipping app code, not running migrations). Nothing runs migrations
automatically: `railway.json` only starts uvicorn. CI does *verify* them —
`backend.yml`'s `migrations` job applies the whole chain to an empty
Postgres 16, runs `alembic check` (fails on any model/migration drift, the
diff `--autogenerate` would have written) and undoes the newest one once.
The offline suite runs on SQLite and cannot see any of that;
`make check-migrations` is the same three steps against whatever throwaway
Postgres `DATABASE_URL` names.

- Apply: from `backend/`, run `alembic upgrade head`. The DB URL resolves from
  `backend/.env.local` (the Railway TCP proxy) via `config.get_configs()`.
- Inspect: `alembic current`, `alembic heads`, `alembic history`.
- The proxy host is not resolvable inside the command sandbox, so migration
  commands run with the sandbox disabled — they need outbound network to the
  managed database's proxy domain.
- New models must be imported in `models/__init__.py` so `SQLModel.metadata`
  picks them up for autogenerate.
- Migrations should be additive/reversible — always provide a working `downgrade`.

### Roadmap (not in codebase yet)

- **Ingestion framework:** PyAirbyte for early pilots → client-managed Airbyte later
- **Query layer:** DuckDB + Ibis
- **Transforms:** dbt
- **Orchestration:** Dagster
- **Delivery:** email through `service/email/` + Slack webhooks

## Product-shape constraints

- Do not build a dashboard-first product here.
- The primary value is the brief and alert output.
- The backend should support a thin onboarding app, not depend on a rich frontend.
- Design all outputs for operator clarity: what changed, why it matters, what to do next.

## Current directory structure

- `service/google/brief.py` — Google Ads brief normalization (loads demo from `data/<connector_id>/`, default `google_ads`)
- `service/google/schema.py` — typed Google Ads brief payload (dataclasses / JSON contract)
- `agents/insights/prompts/` — synthesis prompts: `autonomous.py` builds the system and user prompt, `paid_ads.py` and `organic_growth.py` hold the vertical modes `get_system_prompt()` dispatches to
- `routes/auth.py` — OAuth by connector (`/auth/connectors/{connector_id}/oauth/...`).
  Every connect starts with a `?link=` naming the signed-in user and ends with
  that user's page claiming the grant; both clients come home on
  `/connections?connector=&auth_code=` (the desktop one through `/desktop-auth`
  and the shell's deep link).
- `routes/signin.py` — Google sign-in, and the **guest**: `POST /auth/guest`
  mints a real `users` row keyed on an install id so an audit can run before
  anyone signs in; `/auth/guest/link-code` + `?link=` on authorize lets the
  Google callback link the account to that guest or merge the guest into an
  existing one (`service/user_store.py::absorb_guest`, which walks the schema
  for owner columns rather than keeping a list). Never make an owner column
  nullable for this — a guest is why they need not be.
  `?sources=onboarding` on authorize is the **onboarding bundle**: the same
  sign-in also asking for the Search Console and Analytics *read* scopes,
  offline grant, consent forced; the callback stores one
  `connector_credentials` row per scope Google actually granted
  (`service/signin_sources.py`, through the same upsert the Connections page
  uses, `service/connector_store.py`). It exists for exactly one surface —
  the connector prompt on the onboarding audit, for a guest — and every other
  sign-in and every connector flow stays as it was. Never add a write scope to
  the bundle; a connector asks for those itself, with its justification on
  screen (`service/connector_scopes.py`).
- `routes/audit_prefetch.py` — the crawl onboarding starts the moment a URL
  validates (`agents/audit/prefetch.py`): root page now, the rest in the
  background, handed to `run_pipeline` by `crawl_id`. Duct's bandwidth only;
  inference never runs here. `POST /api/audit/documents` beside it reads up
  to five files the user adds on the "Found it" card, keeps their text for
  the prefetch's TTL (`agents/audit/documents.py`) and answers one result per
  file; the run names them in `document_ids`. It parses the form itself,
  after the token, the limiter and the declared length, because a declared
  `File(...)` body is spooled to disk before any dependency runs.
- `service/documents.py` — text out of a PDF (pypdf), Word or PowerPoint
  (zip + XML, any DOCTYPE refused) or plain text, with every reader stopping
  at a page, character and inflated-size cap. Text rather than a PDF block
  for the model, because several providers (DeepSeek among them) read none.
  A file that gives no text raises `DocumentError` with an `Unreadable` code
  the app words; never a message.
- `service/ratelimit.py` — the in-process fixed window every route goes
  through that spends before its caller has proven much: `RateLimit.enforce`
  is the 429 with `Retry-After`. Key an address with `client_address(request)`,
  never `request.client.host`, which on the hosted API is the proxy in front of
  it; keyed on that, a per-address limit is one bucket for everyone. A guest is
  a user anyone can mint, so a route that spends on a per-user limit carries a
  per-address one beside it. Session starts also cap live sessions per user
  (`agents/core/session.live_session_count`), and the teaser — the one run on
  Duct's key — is held to its lead token, its address and a follow-up budget.
  `tests/conftest.py::fresh_rate_limits` empties every window between tests.
- `service/clone_reference.py` — a pasted TikTok link becomes a saved
  reference for a clone (issue #222). The link is reduced to a handle and a
  post id and everything after uses the URL rebuilt from them; the Apify input
  (`single_post_run_input`) takes that parsed post, one result, downloads off.
  A post the project already saved is reused, else it goes through
  `service/discovery.py`'s ingest and capture. The clone itself is
  `ContentRunner.run_clone`: draft_post's system prompt, one structured call on
  the run's own model for why the reference worked (the slides attached only on
  `VISION_PROVIDERS`), and the FIT × PROOF discipline in the USER turn.
  `submit_post_draft` writes `content_posts.clone_source` from the session's
  link plus the model's verdict; the approach is derived, never typed.
- `agents/audit/draft.py` — the project drafted from the crawl, in layers
  (`crawl` deterministic; `documents` one structured call over the user's
  files, only when they added some; `inferred` one structured call), emitted
  as `PROJECT_DRAFT` when a run sets `draft_project`. The documents are read
  in the `read_documents` step, before competitor research, and reach the
  rest of the run only as the business context they seed: their text never
  rides in the audit's own turn. Never infers the North Star; the agent asks
  for it in chat.
- `service/project_draft.py` — where each project field came from
  (`projects.provenance`) and what may overwrite what, the server's copy of
  the rules `app/src/lib/projectDraft.js` runs for an instant answer (a test
  holds the ranks and field names equal). Two writers, two rule sets: a
  draft (`POST /api/user/projects/{id}/draft`, `merge_draft`) is a machine
  and lands only on a blank, a drafted field of equal or lower rank, never
  on the person's; a save (the PUT, `reconcile_save`) is the person and wins
  where they changed something, but a drafted value the client only carried
  along cannot replace an equal one or blank a field, because that is an
  older copy of a draft the server has since refreshed. **Every new writer
  of project fields goes through `merge_draft`** (the MCP server, a GitHub
  re-sync), never a direct column write; a writer the rules do not know is
  how a guess overwrites what someone typed. A draft cannot claim `user`.
  The one exception is the person speaking through their agent: MCP
  `update_project` writes through `edit_fields` as `user`, whatever the field
  held, and records what it replaced in its `activity_logs` row so
  `POST /api/user/projects/{id}/edits/{activity_id}/undo` (`revert_fields`)
  puts it back, skipping any field changed since (issue #512).
  Pillars are one list, `projects.content_pillars.items`: the planner reads
  it, the Brand tab edits it (and labels it `user`), and a draft's names land
  there through `pillar_rows`. Drafts used to write
  `brand_channels.content_pillars`, which nothing read, so MCP-written pillars
  never reached a plan; a save drops that key if an old device sends it.
- `routes/mcp_server.py` + `service/mcp_server/` — Duct as an MCP server at
  `POST /mcp`, for the person's own agent (Claude Code, Codex, Cursor,
  OpenCode). Stateless JSON-RPC with no SDK: five methods, no sessions, no
  event stream. Its one credential is an MCP token (`service/mcp_tokens.py`),
  accepted on `/mcp` only and never by `get_current_user`, and there are two
  ways to get one. **Browser sign-in** (issue #197, `routes/mcp_oauth.py` +
  `service/mcp_server/oauth.py`): Duct is the OAuth authorization server, so
  an agent given only the address finds the sign-in from `/mcp`'s 401
  (RFC 9728), registers itself (RFC 7591), opens the app's `/authorize`
  consent page, and trades the code with PKCE for an hour-long access token
  and a refresh token. Three rules: **registration writes no row** (the
  client id is the registration, signed with a key derived from
  `JWT_SECRET`, never the secret itself, or a client id would be a session);
  **the backend renders no page** (`/oauth/authorize` parks the checked
  request in `service/auth_exchange` and redirects to the app, and an
  unverified client or redirect goes to the app's error state, never to the
  address it named); and **the sidecar has none of it** (`oauth.available()`:
  its address changes every launch and its app is `tauri://`). A
  **personal token**, minted at `/api/user/mcp-tokens` by a signed-in
  session, is the way in for an agent with no browser. Both are rows in
  `mcp_tokens`, listed together with a `kind`, and DELETE ends either. Four tools, and adding a
  fifth is a scope decision, not a refactor: `list_projects`,
  `get_project`, `draft_project`, `update_project`. A draft goes through
  `service/project_writes.apply_draft`, labelled `document` (or `inferred`
  for what the agent guessed), so it never replaces the person or their
  site. An edit (`update_project`, #512) is the person asking in so many
  words: it can replace anything, carries `destructiveHint` so the client
  confirms it, and is undoable from the activity feed. Each write leaves an
  `activity_logs` row (`category="project"`). No tool calls a model on
  Duct's side, and none approves, applies, connects or touches credentials.
- `agents/engines.resolve_job_run` — provider and model over the keys *this
  caller* can spend, so a user holding only an OpenAI key runs on OpenAI
  instead of a 402 for the instance default. All three agents route through
  it now: audit directly, insights via `agents/insights/setup.resolve_run`
  (which serves both the live session and the scheduled brief), content via
  `routes/content._resolve_run_model`. `resolve_run_model` remains for callers
  that have no job to name.
- `models/settings.py` + `service/profile.py` + `routes/profile.py` — the
  operator profile: name, role, writing preset, the language Duct writes to
  them in, and their own instructions. Same argument as the tier map below,
  applied to voice. Two rules worth knowing: **the server row is the truth and
  the request payload is the fallback** (`resolve`), so a signed-out audit
  still gets the voice picked in the browser; and **the preset derives the
  older `communication_style`/`report_depth` pair** rather than replacing it,
  so every prompt that reads those is unchanged. Rendered for a run by
  `agents/core/voice.user_context_block` into the `<user_context>` block — in
  the USER turn, never the system prompt, because
  `build_insights_system_prompt` is cache-stable and one per-customer string
  in it costs the cached prefix on every call.
- `models/settings.py` + `service/model_settings.py` — the tier map and the
  fallback switch, keyed by user. They used to live in `localStorage` and ride
  on each request, which meant the scheduled brief — the run whose owner is
  definitely not watching — could not read the preference its owner had set.
  The browser's copy is now a cache of this, and on disagreement the server
  wins.
- `agents/core/quota.py` — a 429 the retry loop gave up on, remembered for a
  few minutes so the next run resolves to a tier that can serve. Keyed by
  `(HMAC-SHA256(process salt, key)[:16], provider)` and **never by provider
  alone**: Duct is
  multi-tenant with bring-your-own keys, so one customer's rate limit says
  nothing about another's. In-process and deliberately not durable — a
  cooldown is a hint, and the worst case on a fresh worker is one wasted 429.
  Recorded in `ReportedRetryMiddleware._give_up`, consumed by
  `agents/tiers.resolve_tier_model`'s `cooling` set, reported to the browser
  as `tier_skipped` on `PIPELINE_STARTED`. See
  `docs/engineering/2026-09-08-quota-aware-tier-ladder.md`.
- `POST /api/providers/{id}/verify` — one real completion on the provider's
  Light model, classified into `invalid_key` / `no_billing` / `model_access`
  / `rate_limited` / `unreachable`, or for a ChatGPT credential
  `subscription_quota` / `subscription_revoked` / `subscription_blocked`. The
  only endpoint that can tell a pasted key from a working one.
- The audit agent mounts the insights agent's connector tools
  (`agents/core/connector_tools.py`): `ListDataSources` always, `SelectAccount`
  and `RequestConnection` when the run has a project. The system prompt holds
  the rule (Search Console only, after the report, once); the onboarding
  audit's user turn (`draft_project`) is the trigger, so the cached prefix is
  identical across every other audit.
- `service/mcp/` + `routes/mcp_servers.py` + `agents/core/mcp_tools.py` —
  MCP servers a user connects (issue #345). Duct is the *client* here; Duct as
  a server is #197. Eight rules, each held by `tests/test_mcp_*.py`:
  - **Our runtime, not the provider's.** Anthropic's `mcp_servers` and
    OpenAI's Responses `mcp` tool serve only their own models, so the client
    is the reference `mcp` SDK (2.x) in-process, the capability rule above,
    not fastmcp's heavier stack. Its transport takes only an `httpx2` client,
    so everything under `service/mcp/` speaks httpx2, and it turns a 401 into
    a generic JSON-RPC error: `client._session` watches responses for the
    status instead.
  - **A server is a credential row.** `connector_type = "mcp"`, `account_id` the
    canonical URL; no table of its own. A chat reaches only **its own user's**
    servers, in any project, never the project owner's: unlike a vendor key, a
    token reads the whole account behind it. Sessions are their creator's
    alone, so whoever sees a server's data and answers its card owns it.
  - **Only a trusted preset's `readOnlyHint` is believed** (`trust_read_only`,
    off by default). On a server added by address every call shows the
    approval card: a stranger's "read-only" still carries the agent's
    arguments to the stranger. Infrastructure presets (Railway, Cloudflare)
    stay untrusted too, because their reads return secrets. The card gets the arguments
    whole; a call too long to show (`APPROVAL_ARGUMENT_CHARS`) is refused,
    never cut.
  - **Sign-in is GitHub's shape.** Start (signed in) parks the flow under the
    OAuth `state`; the callback parks the tokens as a grant; only the user who
    started may claim it. A client metadata document where the server takes one
    (its URL *is* the client id, served at `/auth/mcp/client-metadata.json`),
    dynamic registration otherwise, a pasted key when neither works. PKCE and
    the RFC 8707 `resource` on both legs. The metadata is held to where it
    was found: a resource that is not this server or a parent of it, or an
    issuer other than the server the document was fetched for (RFC 8414
    §3.3), ends the sign-in with the key prompt.
  - **Every request goes to a public https address**, redirects included:
    `service/mcp/http.guarded_client` runs the crawl fetcher's resolved-address
    check. No stdio servers on the hosted API.
  - **The registry is searched, never trusted.** `service/mcp/registry.py`
    reads the MCP Registry (`MCP_REGISTRY_URL`, empty turns it off) and says
    who runs each server from the name its publisher proved: a reverse-DNS
    namespace that owns the host, or a GitHub login naming its site.
    Official means that, the site is the tool searched for, and the host is
    the site or one service label under it (`SERVICE_LABELS`), so a tenant of
    any platform fails closed; nothing on one that hands out subdomains
    (`SHARED_HOSTS`) qualifies, and a GitHub login vouches for nothing there. A server
    connected from it is a server added by address: every call asks first.
  - **A server says why it needs its owner.** A failed listing or a refused
    refresh records `store.PROBLEM` in the blob (signed out, key refused, not
    answering…); any call that works clears it, a reconnect replaces it. A
    failed tool call records nothing, because a slow job or one tool's 403
    says nothing about the server. `POST /servers/{id}/check` is the
    listing on demand.
  - **Two fixed agent tools, never one per server tool.** `ListConnectedTools`
    (a menu, then one tool's schema on request) and `CallConnectedTool`; a
    call that asks first pauses on `TOOL_APPROVAL_REQUIRED`. Insights
    interactive sessions only, and only when the user has a server, so every
    other prompt keeps its cached prefix.
- `service/discovery.py` — saved TikTok references. `ingest_reference(db,
  project_id, post)` is the one way a `ScrapedPost` becomes a
  `discovered_reference` row (one per post per project); `capture_reference_media`
  copies its cover and slides into project storage and records the outcome in
  `params["media"]`, because TikTok's image URLs carry a signed expiry. The
  Discover save runs capture after the response; `/content/discover/recapture`
  is the backfill, called when Discover opens. A clone-from-URL flow reuses
  both. Every media fetch goes through `service/url_safety.py`: https only, a
  host allowlist (`MEDIA_DOMAINS`), redirects re-checked hop by hop, a size cap
  and an image content type. The URLs arrive in the request body, so without
  that guard a save is a request the caller aims at our network.
- `service/apify/run_cache.py` — an identical Discover search (actor + canonical
  input) inside 30 minutes joins the earlier Apify run instead of paying for a
  new one. In process; the API runs one worker.
- `routes/generate.py` — `POST /api/insights/generate` for interactive brief + LangChain synthesis envelope
- `routes/content_reports.py` — the Analytics tab's **growth report**: the
  insights agent, unattended, over the sources the person ticked, writing
  the `GrowthReport` the app draws (`agents/insights/report.py`) rather than
  a brief. `GET /content/analytics/report` answers templates, the sources
  that can feed the chosen one, the latest stored copy (`stale` after
  `REPORT_TTL`, a day) and whether a run is writing; `POST` starts one, or
  joins the one already writing the same project and template. Nothing
  regenerates by itself: a report is a model run on the person's key, so the
  stale copy is served and Refresh is the only thing that spends again.
  Stored through `ArtifactPersister` as one artifact group per project and
  template (`kind = "growth-report"`, `GROWTH_REPORT_JSON`), so a refresh is
  the next version and the library keeps the history. Three things worth
  knowing before changing it:
  - **The templates are data** (`agents/insights/report_templates.py`): a
    question, the sources that can answer it, and the blocks to fill, each
    naming the sources that can fill it. A block none of the chosen sources
    can fill is left out of the plan the model gets (`report_plan_block`),
    never left for a guess. A source is one of three kinds: a connector with
    an insights catalog, the project's schedulers (one source, however many
    hold accounts), or an MCP preset whose read-only tools Duct trusts
    (OpenSEO, PostHog). A report run mounts the MCP tools with no pause, so
    anything that would ask first answers `needs_approval` instead.
  - **The schedulers are a FetchData entity** (`social_post_metrics`,
    `agents/insights/catalog/publishers.py`), read through
    `service/publishers/metrics.collect_reported_posts`, the same fan-out the
    Analytics tab's per-post table uses, so the agent and the table cannot
    disagree about a post's numbers. They are a vendor key on the owner, not
    a connector on the user, so `fetchers.fetch_entity` branches to
    `fetch_publishers` before resolving connector credentials, and
    `setup.data_sources_block` adds their line by hand.
  - **A refused payload gets a repair turn, never a silent loss.**
    `AutonomousInsightsRunner.run_report` parses the closing tag against
    `GrowthReport`; counts over the limits are clamped, anything else goes
    back to the model as problems, up to `REPORT_REPAIR_TURNS`. The route
    records the failure on its run entry so the tab can say what happened.
- `routes/project_members.py` — project members + email invitations (`docs/engineering/2026-08-16-project-collaboration-plan.md`)
- `service/membership.py` — project access checks (owner vs collaborator) and invite token handling
- `data/google_ads/` — `google-ads-report.json` (demo brief), `raw/demo_raw_payload.json`

## Agent-type architecture

The `agents/` directory is organised by agent type. Each type is independent and has its own goals, tools, prompts, schema, and versioned runners.

```
agents/
├── engines.py          — engine/provider/model registry + resolve_run_model (shared)
├── models.py           — Provider, ModelName enums (shared)
├── core/               — the ports: session registry, events, LangChain adapter (lc.py),
│                         checkpointer, memory/artifact/connector/web tool binders,
│                         the shared DeepSession loop, SDK shims
├── insights/           — Insights agent (paid ads + organic growth intelligence)
│   ├── v1/             — deepagents runner (the only insights engine)
│   └── catalog/, goals/, schema.py, prompts/, subagents/
├── audit/              — SEO audit agent
│   ├── v1/             — create_agent-rung session runner (build_session_agent)
│   ├── crawl.py        — engine-neutral: the session, the crawl, report parsing
│   ├── enrichment.py   — competitor research; create_agent + web tools, any provider
│   └── scoring.py      — the report's scores and counts, computed from its findings on
│                         every submit (both engines); the prompt's tables render from it
└── content/            — Content Studio (plans, posts, images, publishing)
    ├── v1/             — deepagents runner (the only content engine)
    ├── assessment.py   — the pre-publish review's checks and scoring; the agent
    │                     scores six markers, the weights stay here
    ├── channels.py     — one post row serves every platform; what differs is rules,
    │                     not columns. RULES holds each platform's limit, fold,
    │                     replies that publish, media and metrics support; the
    │                     writer, both publish paths, the review and the app (via
    │                     `post.channel`) all read it. A new channel is a row there
    ├── publishing.py   — what a post sends its scheduler (service/publishers: PostBridge,
    │                     Zernio), the media and its bookkeeping, shared
    │                     by the agent's publish_post and the publish route (#272, then
    │                     a button that read only local disk, were the two drifting
    │                     apart): pick_media chooses the files (a video's cut, each
    │                     slide's composed render), load_media reads them through
    │                     service.storage (R2 or disk), send_post uploads and creates.
    │                     While a scheduler holds a scheduled post, push_edit / cancel,
    │                     which every edit, unschedule, delete and agent rewrite goes
    │                     through so the queue never publishes stale words or a deleted
    │                     post; once its time has passed, settle, which the post
    │                     listing runs (throttled per project) so it reads posted or
    │                     failed, the reason on an activity_logs row
    ├── text_prompts.py — X and LinkedIn draft on their own base prompt; nothing in the
    │                     visual playbook applies to a post that is words
    ├── voice_rules.py  — the project's own voice (#271): writing samples, banned words
    │                     and the client-names / protect-the-edge switches, stored in
    │                     content_brand (no migration). The old prompt constants are the
    │                     defaults. Values ride in the USER turn; the text writer refuses
    │                     a banned word as it refuses an over-length post
    ├── formats.py      — the format library: built-in recipes in data/formats/*.md
    │                     (frontmatter on the Agent Skills shape: name is the slug,
    │                     description says what and when, media and channels say
    │                     where it applies) beside a project's own content_formats
    │                     rows, one shape. A row with a built-in's slug replaces it
    │                     for that project; deleting the row brings it back. The
    │                     agent reads it through fetch_format_library(media, channel)
    │                     and follows spec_markdown; the renderer inlines the whole
    │                     caption-style registry, so a format declares no CSS
    ├── plan_period.py  — the period a plan manages: one post per date, at most 30. The
    │                     person picks the month in progress (default; under a week left
    │                     rolls over to the next month) or 7, 14, 30 days (PlanRequest
    │                     start_date + days). One plan per period: a run opened on a plan,
    │                     or while a plan covers its start date, manages that plan; a new
    │                     plan stops short of the next one; submit_plan updates in place
    │                     (days with a drafted post are kept). It used to insert every
    │                     time, so each refinement in chat left a second plan beside the
    │                     first. app/src/lib/contentSchedule.js mirrors these rules
    ├── best_time.py    — when a draft goes out (#266): the account's own best hour on
    │                     that channel once five posts there have numbers, else the
    │                     channel's default, in the reader's zone, never on top of a
    │                     post already scheduled there. performance.py is the plan's half.
    │                     Its route sits beside POST /content/posts/{id}/skip, whose
    │                     reason becomes a feedback memory the next draft reads
    ├── performance.py  — the signals one history gives a plan: type ranking (never
    │                     likes), explore/exploit, graded bets (pillar, hook, funnel stage,
    │                     objective), best and weakest posts, best posting times. An
    │                     unrecorded metric is unknown, never zero; a goal measure that
    │                     sums metrics (saves + shares) sums only the ones recorded.
    │                     Proven is three measured posts that earned something: fewer is
    │                     a hint (unproven), and enough that earned nothing is flat,
    │                     tested at the floor and never scaled
    ├── channel_mix.py  — what a plan gives each channel: the brand's content goal picks
    │                     the measure (reach, engagement, or intent = saves + shares, since
    │                     conversions are not measured), every channel is summarised on
    │                     its own posts (a tweet never ranks a TikTok), and its share of
    │                     the days moves from the last plan's toward the channels that
    │                     return more of the goal per post: a quarter at most per plan,
    │                     never below a post a week, only once two channels can be
    │                     compared. A day counts for its first platform. The plan turn's
    │                     <account_performance> block is this, rendered, and the plan
    │                     keeps its per-channel facts as strategy.mix (mix_snapshot) so
    │                     the board says what each channel leans on without a sentence.
    │                     submit_plan refuses a hook its channel has not proven on more
    │                     than a quarter of that channel's days (overreaching_hooks)
    ├── plan_proposal.py — a plan is proposed, then accepted: what Accept writes, merged
    │                     against the plan as it stands at Accept. The proposal is held
    │                     on the conversation (meta.plan_proposal) so a deploy does not
    │                     lose it; a resumed session shows it again (replay), and the
    │                     accept and reject routes release it and append a
    │                     plan_decision event, which memory consolidation reads so a
    │                     plan only proposed is never remembered as made
    └── tools.py, subagents/, prompts.py, schema.py, artifacts.py, enrichment.py
```

Route convention: each agent type gets its own route prefix, and every agent's
session lifecycle runs through the unified `routes/agents.py`:
- `POST /api/agents/{type}/sessions` → stream → messages — every session
- `POST /api/insights/generate` — the unattended brief
- `POST /api/audit/run` — the audit pipeline
- `/api/content/*` — content CRUD, brand context, the slide-render bridge; the
  legacy `plan/stream` and `post/stream` entry points drive the same runner

Cross-agent invocations are modelled at the frontend level (e.g. audit findings carry an `invoke_insights` action that pre-populates the insights wizard). Backend agents remain decoupled — no direct calls between agent types.

## Agent ports — the harness boundary

We rent an agent harness; we do not marry one. The full declaration (with the
port table and the external standard behind each) lives in
`agents/core/ports/__init__.py` — read it before adding anything that touches a
framework. The rules it implies:

- **Never write an `AgentHarness` interface.** Harnesses differ in capability,
  not just API shape; the intersection loses the reason to use one and the
  union means maintaining a framework. The harness stays harness-shaped inside
  a runner, and only its *boundary* is standardized.
- **Domain code imports no framework.** Tool bodies, prompts, schemas, goals and
  scoring are plain Python. `agents/core/memory_tools.py` is the reference
  shape: `_remember_sync` / `_search_sync` / `_get_sync` hold the logic, and
  `build_memory_tools_lc` / `build_memory_tools_sdk` are thin binders.
- **Framework imports live only in adapters** — runners, binders, and the
  named shims. `tests/test_harness_boundaries.py` enforces this and lists the
  allowlist; adding a file to that list is a deliberate act, not a fix for a
  failing test.
- **Write the adapter on the second implementation, not the first.** One
  implementation is a guess. The human-in-the-loop port is the worked
  example: `PauseFn` in `agents/core/session.py` was declared only once the
  LangGraph `interrupt()` implementation (`agents/core/lc.interrupt_pause`)
  existed beside the Future bridge. A tool body takes a `PauseFn`; the binder
  that mounts it decides which one — the Future for an agent with no
  checkpointer (audit, the slide-render bridge), the interrupt
  for one with durable threads (insights v1, content v1). Same events, same
  route, and the frontend cannot tell them apart.
- **What the user may watch an agent do is one allowlist, in one place.**
  Every tool call passes through `recorder_tool_hooks`; `agents/core/activity.py`
  wraps that pair and emits `TOOL_ACTIVITY` for the tools it maps — a connector
  pull, a web search, a page read, an image, a sub-agent dispatch — twice per
  call (running, then the verdict), with **structured fields and no prose**, so
  the app writes the sentence in the reader's language. Three rules:
  - **Allowlist, never denylist — and the allowlist is nearly everything.**
    A tool is invisible until someone writes its card, so a new tool leaks
    nothing by default; but the standard is transparency, and the only calls
    left off are the ones with a better row of their own: `RememberFact`
    (the "Remembered" note, with undo), the pause tools (the cards),
    `Create/Update/RewriteArtifact` (the artifact card), `write_todos` (the
    strip), the execution proposal (the change-set card) and the audit's
    report builders (its step ladder). Memory searches and reads, the
    connector listing, opening or listing a document, every read the content
    agent makes of what the project holds, and everything it saves, publishes
    or logs are rows. A new tool goes on the list unless it has one of those
    other rows — say which, in the comment above `ACTIVITY_TOOLS`.
  - **The run's own context is a row too.** `announce_context` records the
    CONTEXT row and emits the "Read project context · business, memory,
    connected sources" notice in one call, so a runner cannot do one without
    the other. Every runner calls it where it composes its opening turn;
    a resume records the row and draws no notice (the reopened transcript
    already has one). Memory's two events are stored as rows as well
    (`EventKind.MEMORY_RECALLED` / `MEMORY_WRITTEN`), so what the run
    recalled and remembered survives a reload for every agent.
  - **No bespoke per-tool step events.** The insights runner used to emit a
    STEP per data pull with an English label built in Python, which the ladder
    and a separate pane both rendered; that is one line said twice and
    untranslatable. Add a mapper, not an event.
  - **Every runner wires the hooks.** `tests/test_tool_activity.py` fails a
    runner that opens a `DeepSession` without `activity_hooks`, because the
    audit runner passed no tool hooks at all for months — recording nothing,
    showing nothing — and nothing failed. The card payload carries the notice
    that a tool ran, never its output: the full input and result are already in
    the transcript, and a card holding fetched page text would put
    attacker-authored bytes in every client's memory.
- **A durable thread is the conversation.** The insights runner keys its
  LangGraph thread on the conversation id, so a resumed session continues the
  thread — and a pause the thread is parked on comes back as the same SSE
  event, flagged `replay`, when a session resumes it. Never key a thread on
  a session id; that made "resume" a transcript the agent could not see.
- **A failure is a code before it is a message.** `agents/core/errors.py`
  classifies an exception once (`classify_error`), and that code decides the
  retry (`is_retryable`), rides on the failure event (`error_payload`), and
  picks the copy in the browser. Never emit `str(exc)` to a client, and never
  add a regex on message text in the frontend — add a code, or a class name to
  the classifier's table.
- **Input during a turn is steered or queued, never refused.** A harness that
  can hand the model a message at its next call sets `steer_queue` on its
  session (insights does, via `SteerMiddleware`); the rest fall back to
  `chat_queue`. The route decides; the runner reports `user_input_consumed`
  when it dequeues so the client can drop the "queued" mark. Do not reintroduce
  the 409.
- **Run status is derived from the stream, in one place.** `ConversationRecorder`
  (`agents/content/persistence.py`) already sees every event, so it is what
  writes `agent_conversations.run_status` (`RunStatus` in
  `agents/core/events.py`: idle / running / paused / failed / cancelled) and
  `run_error`, and appends a `failure` event where a turn died. The list and
  state routes carry both; a reload shows the failure where it happened, with
  the same code. Do not set the status from a runner — a second writer is how
  two agents end up disagreeing about one column. A session closed mid-turn is
  recorded as `cancelled` by `recorder.close()` in `_close_and_consolidate`.
  A process that stops writes nothing (server shutdown's `close_all_sessions`
  never reaches a recorder, and a kill runs no shutdown), so every run still
  `running` at the next boot is cancelled by `cancel_orphaned_runs` from the
  lifespan. It assumes one API process; a second replica would need a lease
  or heartbeat instead.
- **A retry says how long, and the provider's `Retry-After` wins.**
  `MODEL_RETRYING` carries `retry_in` (seconds, a duration — the client anchors
  it to its own clock so skew cannot show a countdown already over), computed by
  `retry_delay(attempt, exc)`, which reads `retry_after_seconds(exc)` from
  `agents/core/errors.py` before falling back to the jittered schedule. A
  provider asking for longer than `MODEL_RETRY_HEADER_MAX_DELAY` is not
  retried at all — the failure, with its code, is more useful now than after
  a countdown that fails anyway. The summariser's calls are billed with
  `scope: compaction`, so they count toward the total and never drive the
  gauge.
- **A request too long gets one compaction and one retry.** The automatic
  summariser works from an estimate and the provider counts for real; when
  they disagree the request comes back as `context_window`. The insights
  runner then calls `compact_thread` (`agents/core/lc.py`) — LangChain's own
  `SummarizationMiddleware` forced by a one-message trigger, keeping the last
  `COMPACT_KEEP_TOKENS`, written back "as" the tools node so the graph's next
  step is the request that failed — emits `context_compacting` /
  `context_compacted`, and continues from the checkpoint. A second overflow is
  the ordinary failure. deepagents' own summarisation event is cleared in the
  same write: it indexes into the message list the rewrite just replaced.
  Both paths — the emergency one and the automatic summariser — put the
  **summary text** on `context_compacted` and the recorder writes it as an
  `EventKind.COMPACTED` row, so the transcript can show what the thread now
  opens with, live and after a reload, instead of an unexplained gap. The
  automatic one is deepagents', which summarises *inside* the `model` node,
  so a node name cannot find it: `is_summarization_call` reads the
  `lc_source: summarization` tag both summarisers put on their model call,
  and the divider goes out when the reply that runs on the summary starts
  streaming, not when the node's update lands after it. `compaction_summary`
  reads the same tag off the summary message, minus either summariser's
  framing.
- **A model has a price or it has no cost.** `PRICING` in `agents/models.py`
  mirrors `CONTEXT_WINDOW` (a test holds them equal) and `cost_usd()` prices a
  call from LangChain's usage, taking cached tokens out of the input figure.
  `TOKEN_USAGE` and the state route carry `cost_usd`, `None` when unpriced —
  never a guess, because on BYO keys the figure is what the user pays.
- **A model that leaves the catalogue leaves a row behind.** Replacing a
  `ModelName` value (Opus 5 → Opus 5.5, 2026-09-29) orphans every saved tier
  pick, `GENERATE_MODEL` and stored thread that names the old id, and
  `resolve_engine_model` sends an id it does not know to the *provider
  default*: a Heavy pick of Opus 5 would have run on Sonnet without a word.
  `RETIRED_MODELS` in `agents/models.py` maps the old id to its successor —
  never a step up in price — and keeps the old model's own price and window,
  so a thread it served still reads what it cost. An OpenRouter slug whose
  only successor costs more gets no successor and keeps running as picked
  (OpenRouter passes it through). `tests/test_model_transport.py` holds all
  three. An OpenRouter slug names a *build*, so check the vendor's changelog
  as well as the catalogue: `deepseek/deepseek-v4-flash` still served, and
  looked current, after DeepSeek retired the build.
- **A model's request contract is data, not a check at a call site.**
  `CLAUDE_BOUND_THINKING` (Fable 5.1, Opus 5.5, Sonnet 5.5) answers two
  questions: a forced `tool_choice` is a 400, so a typed answer at the end of a
  tool loop goes through `agents/core/research.research_answer`, never a bare
  `ToolStrategy`; and a thinking block replayed after an earlier turn changed
  is a 400 for accounts opened on or after 2026-08-31, so `resolve_chat_model`
  asks the API to drop it instead. `takes_temperature` drops the sampling
  temperature the Claude 5 family and GPT-6 refuse while they reason. A new
  model is classified in those tables when it joins the catalogue.
- **Before adding an agent or changing one — a prompt, a tool, a model
  default, a middleware, a LangChain bump — read
  [`docs/engineering/agent-engineering.md`](../docs/engineering/agent-engineering.md).**
  It is the rulebook for agent work on this stack: each rule, the LangChain
  primitive behind it and the test that enforces it, the add/change
  checklists, and the ranked open gaps. Fixing a gap means adding its test and
  striking it from that list in the same change.
- **A change to an agent runs the eval gate.** `agent-eval.yml` runs
  `scripts/agent_eval.py` on any PR touching `agents/` (not `agents/content/`
  or `agents/audit/`, which no case covers yet: take the exclusion off when
  one does), `service/memory.py`, `service/profile.py` or the lock: each case
  in `tests/eval/cases/` three times on a PR with no re-run, six times on the
  merge to `main`, on a synthetic account, DeepSeek V4 Flash on OpenRouter with GLM 5.3
  Flash judging, a verdict against `tests/eval/baselines.json` (`make agent-eval`
  locally, a few cents). A baseline is per model: a run on another model is
  INCONCLUSIVE until `--write-baseline` records one for it. The eval runs on
  every push to a ready PR, so open agent PRs as drafts while iterating.
  It is in shadow until an A/A window on unchanged `main` shows no false
  FAIL; from then on FAIL does not merge, and INCONCLUSIVE needs one line in
  the PR saying why it is acceptable. A harness change also adds an offline
  test of what reaches the model for the invariant it touches
  (`tests/test_request_invariants.py` is the model), never a prompt-wording
  test. A bug found in a real session ships with a synthetic case that fails
  without the fix. Never raise k, lower a threshold or delete a case to turn a
  run green: loosening the gate is its own PR, with the numbers.
- **Read the harnesses built in the open before designing a lifecycle
  feature.** [`docs/engineering/agent-harness-references.md`](../docs/engineering/agent-harness-references.md)
  is the watch-list — Codex, OpenCode, pi — with the revision each was last
  read at, findings pinned to `file:line`, and the gaps they expose in ours,
  sized. Pauses that survive a reconnect, typed error codes, steer-versus-queue
  input and visible compaction all have a worked answer there. Refresh the
  table when you read one.
- **Pick the lowest rung that works.** LangChain 1.x is layered, and the layers
  carry different stability guarantees: `init_chat_model` and `create_agent` are
  on the semver-stable 1.x LTS surface (no breaking changes until 2.0), while
  `deepagents` is 0.x with no stability policy and a weekly cadence. Reaching for
  `deepagents` where `create_agent` suffices buys churn for nothing.

  The rung is a property of the agent, not of the agent *type*, and it can move
  when the agent's job does. The autonomous insights session
  (`agents/insights/v1/runner.py`) is on `deepagents` because it needs four
  things `create_agent` lacks — a planning loop, subagents, skills, and the
  `interrupt_on` upgrade path — and the phase plan spends all four. The content
  session (`agents/content/v1/runner.py`) spends three of them from its first
  turn: `write_todos` is the checklist the workspace renders, `research_pillar`
  and `draft_post` are sub-agents, and the virtual scratch space holds drafts.
  Audit's V1 runner is still on the `create_agent` rung, and content's
  enrichment pass (`agents/content/enrichment.py`) is one too — search, fetch,
  structured answer, no planning.

  **The rung is chosen; the middleware is not.** A session graph is built by
  `agents/core/deep_session.py` — `build_deep_session_agent` or, on the
  `create_agent` rung, `build_session_agent` — and both mount the same
  `session_middleware` list (pruning, call limits, fallback, reported retry,
  steer) plus provider prompt caching. The audit was a bare `create_agent`
  from 2026-08-16 to 2026-09-27: its `RunLimits` (added 09-06) were declared
  and never enforced and Anthropic never cached its prefix, all tests green.
  `tests/test_agent_assembly.py` now fails a graph built anywhere else; a
  bounded one-shot pass (the two enrichment modules) is listed there and must
  still pass `prompt_caching_middleware()`.

  Three consumers of the 0.x pin now, so `tests/test_deepagents_harness.py`
  matters more, not less: run it before moving the pin. `tests/test_content_v1_runner.py`
  pins the content contract the same way `tests/test_insights_session.py` pins insights.
- **`deepagents` is pinned exactly**, not with a caret — it changes behaviour in
  minors (task planning became opt-in in 0.7). `tests/test_deepagents_harness.py`
  is the upgrade gate; run it before moving the pin.

## Artifact contract

The app lists top-level `*.json` briefs in `data/google_ads/` (e.g. `google-ads-report.json`) for local dev; user-generated insights are returned from `POST /api/insights/generate` and stored client-side (`localStorage`).

The JSON contract:
- `source_metadata.theme` — theme key (`paid_ads`, `product_intelligence`, `organic_growth`); the app resolves accent colors from this
- `source_metadata.generated_at` — ISO 8601 timestamp
- All other fields follow the typed models in `service/google/schema.py`

Do not write HTML from the backend. Do not reference `themes.json` or HTML templates — those have moved to the app.

## Code design rules

- Normalize first, synthesize second.
- Keep typed schemas central and explicit.
- The backend is a data pipeline, not a renderer. Output JSON; let the app handle presentation.
- Separate ingestion, normalization, synthesis, and delivery concerns.
- Prefer extensible evidence models so future tools can enrich the same findings.

### Shared helpers — check here before writing a local copy

Each of these replaced a family of divergent duplicates (23 private
`_utcnow()` definitions, five hand-rolled retry loops). A new local copy
re-opens that drift, so reach for these first and extend them when they
don't fit.

- `utils/dates.py` — `utcnow()`, `now_iso()`, `parse_iso()`, `last_n_days()`.
  **Never `datetime.now()` or `datetime.utcnow()`**: every persisted timestamp
  is UTC-aware, and naive ones compare and serialise inconsistently across
  SQLite (desktop) and Postgres (Railway). Stdlib-only leaf module, so
  `models/` can use `default_factory=utcnow` without an import cycle.
- `utils/strings.py` — `slugify()`, `titleize()`.
- `utils/formatting.py` — `money()`, `number()`, `percent()`, `multiplier()`,
  `safe_divide()`.
- `service/memory.py` — agent memory (`project_memories`): `remember()` is the
  ONLY write path (it redacts secrets, drops raw ids from titles, dedupes,
  honours the pause switch, and closes the previous value of a state key), `search()` is the only read path
  (Postgres FTS, SQLite LIKE), and `build_memory_context()` assembles the prompt
  blocks. Never write the table directly and never re-render the digest locally
  — the supersession and never-raise contracts live in that module.
  `service/memory_consolidation.py` owns the post-session extraction pass; its
  model output is a proposal that still goes through `remember()`. Tools for
  both harnesses are in `agents/core/memory_tools.py` and the shared prompt
  stanza is `agents/core/prompts.py::MEMORY_DISCIPLINE`. Per-project memory goes
  in the USER message, never the system prompt.
  `search(time_aware=True, rank=True)` is the question-shaped read — it reads a
  date range out of the words, treats a named kind as a filter, matches on ANY
  term and then tightens (all terms → two → one), and ranks by relevance +
  recency + importance + recall. Leave both off for a filter form like the
  timeline, whose inputs are the user's explicit instructions. Retrieval makes
  no model calls, by design. `tests/eval/memory_recall.py` holds the 50-question
  set (`pytest tests/test_memory_retrieval.py -s` prints the per-axis report);
  it exists because it caught the AND-everything query bug that made questions
  retrieve nothing, so extend it before tuning retrieval by feel.

  **Private memory is private whatever the role, and the rule is in the
  service, not in the routes.** `remember()` stores a `PERSONAL_KINDS` entry
  (identity, communication, method) as user scope even when the caller said
  project, because an agent in a project run noticing how someone works is
  describing that person; `feedback` is left out on purpose, since a skipped
  draft's reason is about the project's output. A private row may be bound
  to one project (`private_to_project=True`: "for this client, brief me in
  Spanish"), so it carries that project's id, which is why **every project
  read goes through `shared_rows()`, never `project_id ==` alone**. A bound
  row is read only on runs for its project, never replaces the general one,
  and keys its state to its person so two teammates can bind the same key.
  `search()` never returns the preference keys the profile now owns
  (`_PROFILE_OWNED_KEYS`). Held by `tests/test_memory.py`.

  **The kinds are twelve project and four personal** (`models/memory.py`).
  `rule` is a standing policy, keyed `rule:<topic>` so a new one replaces the
  old, and every rule in force leads the digest's Pinned section; `outcome`
  is what a change did, keyed by the change and the window measured. Merged
  kinds (`LEGACY_KINDS`: milestone, process, tooling) are mapped on write and
  in `search()`, so an old name still works; add to that map rather than
  deleting a kind outright.

  Six rules about when memory is read and written, each held by
  `tests/test_memory_context.py`, `tests/test_memory_phase2.py` or
  `tests/test_memory_retrieval.py`:

  - **`build_memory_context()` returns its blocks in `MEMORY_BLOCKS` order and
    applies the agent's `ContextSpec` itself.** A turn carries the result as one
    string in the `project_memory` slot, so memory's tags stay adjacent in
    `BLOCK_ORDER`. A block the spec switches off is never queried.
  - **The digest budget is whole entries**, `DIGEST_MAX_CHARS` (about 1,500
    tokens) and `DIGEST_MAX_ENTRIES`, spent in section order: pinned, open,
    relevant to this question, recent, artifacts. An entry that does not fit is
    left out and is not reported as recalled.
  - **Recalled means cited or opened, never shown.** `ConversationRecorder`
    credits the `m_…` ids a reply cites (`touch_cited`), and GetMemory credits
    what it opens. Counting the digest fed its own ranking.
  - **Consolidation does not depend on a close.** Close flushes early; the
    sweep in `service/memory_consolidation.py` (every five minutes, for
    conversations quiet for 30) is what guarantees the pass after a deploy,
    crash or quit desktop app. It runs on `Job.MEMORY`, the owner's Light tier.
    "Don't remember this" is stored on the conversation
    (`record_remember_choice`), because the sweep never sees the session.
    The sweep can spend only *saved* keys, so an owner whose keys live in the
    desktop keychain or a browser tab is covered by keys they lend instead:
    a closing session lends its caller's header keys (`LentKeys`), and the app
    calls `POST /api/user/memory/catch-up` when it opens and every 30 minutes
    (`app/src/lib/memoryCatchUp.js`). Lent keys are held in memory, never
    stored, and spent only on projects the lender owns, because the owner pays
    for background work. Plan tokens are never lent: a structured one-shot
    call cannot spend them.
  - **A thread re-reads memory at most hourly, and only when something changed.**
    `memory_refresh` compares the conversation's `memory_primed_at` with the
    newest memory row, excluding the thread's own writes; `routes/agents.py`
    asks on each message, and the session holds when to ask next.
  - **A state outside Duct is shown with its age, never as "present".** A
    campaign's status, a budget, a rolling metric renders `seen <date> · Nd`,
    and past its entity's shelf life (`FRESHNESS_TTL_DAYS`) `verify`, which
    `MEMORY_DISCIPLINE` turns into a re-read before the entry is stated as
    current. Remembering the same fact again stamps `meta.verified_at` and
    starts the age over; a consolidation pass does not, because a transcript
    that repeated memory is not a sighting. Derived at read time
    (`volatility()`, `freshness()`), so no column. The freshness axis of the
    memory eval holds the stale-assertion and false-alarm rates at zero.
- `agents/core/turn.py` — **how every agent's user turn is built.** An agent
  declares a `ContextSpec` in `agents/registry.py` (which shared blocks it
  wants); a run renders a `TurnContext`; `build_turn` orders them. A new agent
  that declares nothing gets everything — business context, the operator's
  profile, stored agent context, prior reports, memory, data sources — which is
  the right default, because an extra block costs a few hundred tokens and a
  missing one costs an agent that does not know who it is answering. That was
  not hypothetical: `display_name` reached insights and content and silently
  missed audit for a release, because audit described the operator its own way.

  Two rules this module exists to hold, both enforced by `tests/test_turn.py`:

  - **Per-user and per-project text goes in the USER turn, never the system
    prompt.** The system prompt is the cached prefix; one customer's name in it
    gives every account a prefix of its own and loses the hit on every call of
    every run. The test builds a profile of distinctive strings and fails if
    any of them appear in a system prompt.
  - **`BLOCK_ORDER` is stable-first, volatile-last, and it is a cache decision
    rather than a formatting one.** Inside one session the order is free (turn
    one is the prefix for turn two either way). It pays *across* runs: two
    insights runs on the same project a week apart share a system prompt, and
    if both turns open with byte-identical `<business_context>` and
    `<user_context>` the cached prefix extends past the system prompt into the
    turn. Lead with the memory digest instead and the match ends at the first
    block, because memory moved in between. Do not "tidy" that tuple into
    declaration order.

  An agent with a block of its own may set an unlisted tag — it renders after
  the ordered blocks and before the request. Anything a *second* agent starts
  using belongs in `BLOCK_ORDER`, where its cache position is a decision
  somebody made on purpose.

- `scripts/session_bundle.py` — pull one stored session (the context the
  model read, every tool call with its payload, artifact versions, memories,
  cost) into a folder for review; `make session-bundle ID=<conversation id>`
  or `ID=list`. Read-only, redacts on the way out, and the folder is
  customer data: it goes in the private audit home, never here. The
  `session-audit` skill (`.agents/skills/session-audit/`) is the review
  procedure built on it. The pull is only as good as what the recorder
  wrote: `EventKind.CONTEXT` (one row per `run_session`, written by the
  runner that composed the turn — the insights runner today) holds the
  composed opening turn, its blocks and the system prompt's fingerprint,
  because the USER row is only the sentence the person typed. A runner
  that assembles its own turn and does not record a CONTEXT row cannot be
  audited against what its model saw; give it one before relying on the
  skill for that agent.
- `scripts/session_replay.py` — re-run a bundled session on exactly its
  own data with the current prompt and any model (`make session-replay
  BUNDLE=<dir>`). FetchData is seeded from the bundle and the connectors
  are closed (`replay=` in `build_data_tools_lc`: a pull the original never
  made returns `not_in_replay`), no recorder, no persister, no memory or
  execution tools. It is how a prompt proposal is checked against the run
  that motivated it before it ships, and it spends a real model call.
- `scripts/dump_prompts.py` — **regenerate after any prompt change**
  (`make dump-prompts`). It renders every agent's system prompt and a sample
  assembled turn to [`docs/engineering/agent-prompts.md`](../docs/engineering/agent-prompts.md),
  checked in, so a prompt change lands in the pull request as prose rather than
  as a Python string edit a reviewer has to assemble in their head.
  `prompts.yml` runs the `--check` form on a PR that touched a prompt, and
  `make check-backend` runs it locally.

  We looked at the prompt file formats (Prompty, dotprompt, POML, BAML) and did
  not adopt one. They solve prompts-as-swappable-config, edited outside the
  repository and hot-reloaded without a deploy. Duct's prompts are composed:
  `agents/content/prompts.py` alone has 47 interpolation sites and 22 branches,
  audit assembles its scoring table from `agents/audit/scoring.py` and varies
  its tool guidance on whether the provider does vision. Moving that into
  Handlebars or Jinja moves real logic into a template language with no types
  and no tests. **The reviewability problem was never the storage format** — it
  was that nobody could read the assembled result. That is what the dump fixes.

  The output must stay deterministic: no timestamps, no run ids. A
  non-deterministic byte fails `--check` on every unrelated pull request, and a
  check that cries wolf gets switched off.

- `agents/core/lc.py` — the LangChain adapter every V1 runner shares:
  `resolve_chat_model` (model transport) and `stream_agent` (LangChain stream →
  the `AgentEvent` vocabulary), plus `build_ask_user_tool`, the LangChain half
  of the human-in-the-loop port. Extracted from `agents/audit/v1/runner.py` when
  insights became the second V1 runner. A V1 runner should not talk to
  `init_chat_model` or drive `astream` itself. A one-shot typed answer goes
  through `structured_output`, not `with_structured_output`: it asks Claude
  for its own JSON-schema output, since the integration's default forces a
  tool call that Fable 5.1 refuses.
- `agents/insights/report.py` + `prompts/autonomous.REPORT_CONTRACT` — the
  structured deliverable beside the brief. One agent, two contracts: the
  system prompt ends with `ARTIFACT_CONTRACT` (prose) or `REPORT_CONTRACT`
  (blocks with inline rows), chosen by `build_insights_system_prompt(
  deliverable=…)`, two cached prefixes rather than one per project; which
  template and sources ride in the user turn as `<report_plan>`.
  `VIZ_PROTOCOL` (which chart answers which question) reaches both, because
  the HTML brief draws charts too and drew a pie of twenty queries once.
- `agents/insights/totals.py` — **Duct adds up a pull; the model quotes the
  sum.** `fetch_entity` puts `totals`, `rates` and per-dimension `subtotals`
  on every row report, from the catalog's `agg`, `ratio` and `weight`, and
  says in `totals_cover` whether they are the window's or a floor, and
  whether the source flagged the report itself: a fetcher puts a source's
  own caveats in `data_quality` as sentences (GA4's thresholding, sampling,
  "(other)" roll-up and date truncation, from the response metadata), and
  the cover points at them. A replay
  of a real brief reasoned right and added wrong (2,504 against 2,681
  sessions); summing is now never the model's job. A ratio metric is total
  over total, never a mean of rates (CPA weighted by conversions would drop
  every row that spent and converted nothing). A new averaged catalog field
  declares `ratio` or `weight`, or `tests/test_insights_totals.py` fails.
- `agents/core/compaction.py` — lossless payload compaction, applied to a
  connector result before it reaches the model. Folds a homogeneous row array
  to typed CSV; **verifies the result structurally and discards a fold that
  lost anything**, because `lossless_only` does not gate every path in the
  library underneath (an array of identical strings is still collapsed). Off
  is not "the model sees everything" — off is the mid-structure cut in
  `agents/insights/data_tools.py`, which on a 900-row pull drops about two
  thirds of the numbers. Absent dependency degrades to plain JSON, so a
  self-host build without the Rust extension still runs. Reached from a
  `UserPreferences` flag (`context_compression`, default on) threaded through
  the runner the same way `artifact_format` is.
- `service/artifact_store.py` — versioned artifact persistence. `ArtifactPersister`
  wraps a runner's emit and stores every `ARTIFACT_VERSION` event; an **adapter**
  (`ArtifactVersion` + a `Callable[[dict], ArtifactVersion]`) reads one version out
  of whatever payload that agent emits. A new agent writes an adapter — audit's
  validates an `AuditReport`, insights' reads a written brief — and never a second
  persistence path. Storing is the store's job; understanding the payload is the
  adapter's.
- `agents/tools/execution_tools.py` — the staged-execution tool surface, as a
  binder pair (`build_execution_tools_lc` / `build_execution_mcp_server`) over
  shared descriptions, arg schemas and domain functions. The surface is
  deliberately asymmetric: propose, inspect and roll back exist; **approve and
  apply do not, in either harness.** Autonomy (`ask | assisted | auto`) changes
  how often an agent interrupts, never what may auto-apply — `service/execution/
  policy.py` is the one place that decides, and it does not consult the model.
- `service/connectors.py` — the registry and the adapter contract. A
  `list_accounts` row has a canonical half the browser reads and a native half
  it never does: `account_id` / `account_name` name the thing being **picked**
  (the GA4 property, the GTM container — not its parent account, which several
  rows share), and the optional `entity_url` / `entity_detail` / `entity_meta`
  (built with `entity_facts`) are what the picker renders as a favicon, a
  disambiguating line and short chips. Vocabulary is server-side too
  (`ConnectorMeta.entity_noun`), so adding a connector stays one registration
  rather than a registration plus an edit to a table in the frontend.
  `ConnectorMeta.server_only_keys` names credential keys **no request may
  write**, and both routes that take credentials from a body refuse them.
  It exists for the GitHub App (`service/github/app.py`): Duct's private key
  mints a token for any installation id it is handed, so an id a browser
  could store is someone else's repository. The only writer is the claim
  route, bound to the user whose signed-in session started the connect.
  The four Google connectors' `refresh_token` is server-only for the same
  reason and by the same route shape (issue #456): `POST
  /api/user/connectors/connect-link` names the user, the OAuth callback parks
  the grant, `POST /api/user/connectors/claim` stores it merged over the row
  (`service/connector_store.store_oauth_grant`, so a reconnect keeps the MCC
  id). **No Google token reaches a client** — not in a redirect, a response or
  browser storage — and every reader (`/api/insights/refresh`, execution,
  FetchData) resolves from stored rows; there is no request override left.
  `ConnectorMeta.settings_keys` is the opposite list: non-secret keys the user
  may change on a stored row (`PATCH /api/user/connectors/{id}`), today only
  the Ads `login_customer_id`, shown back as `settings` on the row.
- `service/content_metrics.py` — a content post's `perf`, read and written one
  way. `METRIC_ALIASES` reconciles the three key conventions in the column
  (PostBridge's `view_count`, migrated `avgWatchTime`, hand-entered `saves`;
  Zernio's adapter writes the canonical names, and leaves out a metric it
  reports as 0 where the platform reports none, because unknown is not zero);
  read through `metric_value`, never `perf.get("…")`. Two writers:
  `merge_manual_metrics` (the `/metrics` route) records typed-in metrics under
  `manual_keys`, and `merge_synced_metrics` (the sync route and the agent's
  `log_metrics`) skips every alias of those, so **a sync never overwrites a
  number somebody typed.** `app/src/lib/contentMetrics.js` mirrors the table
  and `tests/test_content_metrics.py` holds the two equal.
- `service/rest.py` — sync HTTP transport for the reporting connectors:
  retry, backoff, rate-limit pacing, error typing. A new connector declares
  an `Endpoint` and an `ApiError` subclass and writes no transport code;
  auth headers, query encoding and pagination stay vendor-side. Not for
  `service/apify`, `service/post_bridge` or `service/zernio` — those are async, hold a
  long-lived client, and need no retry.
- `service/ollama.py` + `Provider.OLLAMA` — open models on the instance's own
  Ollama server (`OLLAMA_BASE_URL`), the one chat provider with no key: the
  server is the credential (`ProviderKey.source == "instance"`), reachable on
  hosted and local alike once the operator sets it, and only when it holds a
  model that calls tools. Ids are namespaced `ollama:<tag>`, checked before
  `provider_of`'s "/" test because a Hugging Face tag has one. Three rules,
  held by `tests/test_ollama_provider.py`: the pickers list what the server
  holds (`/api/tags`, filtered on `/api/show`'s `tools` capability), never a
  catalogue; every call sends `num_ctx` (`OLLAMA_CONTEXT_LENGTH`, held to the
  model's trained length), because Ollama's own 4,096-token default silently
  cuts the system prompt and tools; and `reasoning` goes only to a model that
  thinks, since Ollama refuses it from one that cannot. A local model is
  priced at zero, not unknown.
- `service/compatible.py` + `Provider.OPENAI_COMPATIBLE` — an OpenAI-compatible
  endpoint a user brings (vLLM, llama.cpp, LM Studio, Together, Groq):
  ChatOpenAI at their address. Its credential is the address, an optional key
  and the models they chose (with each one's reported context length) packed
  into one string (`oaic1.…`), so it is stored, sent as `X-Provider-Compatible`
  and lent to background work exactly like a key; `app/src/lib/
  endpointCredential.js` packs the same bytes, and a shared fixture holds the
  two equal. Ids are `compat:<the server's id>`, checked before the "/" test
  (a vLLM model is named by its Hugging Face repo). Four rules, held by
  `tests/test_compatible_provider.py`: **the address is user input**, so off a
  private network it must be public https, judged on the name when entered
  (`canonical_base_url`), and every request after is resolved once, judged
  and connected to the address that passed, real hostname kept for TLS
  (`_PinnedTransport`, so DNS rebinding has no second lookup to win), with
  discovery read under a size and time cap; a private address
  only where the network is the user's own (desktop sidecar, local dev, or
  `PRIVATE_MODEL_ENDPOINTS`); **connecting proves a tool call**
  (`POST /providers/openai_compatible/probe`), because a model that answers
  in words cannot run an agent; a typed answer asks for a JSON schema first
  (constrained decoding, which a small model follows where it ignores a forced
  tool call) and falls back to a tool call; and the price is
  unknown (`None`), since a hosted endpoint bills and Duct cannot say what.
- `service/publishers/` — social schedulers as connectors (PostBridge,
  Zernio). Nothing publishes by default: a project publishes through a
  scheduler its owner connected, each linked account names the scheduler that
  holds it (`content_social_links.publisher`), and a sent post records its own
  (`content_posts.publisher`, `publisher_post_id`), because only the scheduler
  holding a queued post can change or cancel it. `base.py` is Duct's
  vocabulary (`Publisher` protocol, `RemotePost`, `RemoteStatus`,
  `PublisherError`, `PostAlreadyOut`) and every caller speaks only it; each
  adapter lives beside its client (`service/post_bridge/publisher.py`,
  `service/zernio/publisher.py`) and translates the vendor's ids, statuses,
  replies and metric names. What a scheduler can do differs per platform and
  is data on its `PublisherSpec` — `replies` (PostBridge: one on X and
  Threads; Zernio: a thread on X and Threads, a first comment on LinkedIn,
  Instagram, Facebook), `metrics` (Zernio reports LinkedIn and X too),
  `strips_links` (PostBridge on X) — which `channel_payload` folds into what
  the app reads, so `channels.RULES` keeps only the platform's own facts.
  Adding a scheduler is an adapter, a spec in `registry._SPECS`, a `VendorKey`
  and a transport fake in `tests/fakes.py`; the routes, the agent's tools and
  the Accounts tab read the registry. Adapters speak REST, not the vendors' MCP
  servers: publishing runs the typed checks, the approval and the bookkeeping
  around a call, and settling, metrics and the Analytics tab run with no model
  in the loop at all. A scheduler Duct does not support is still reachable
  through the MCP client (`service/mcp/`), behind its approval card.
- `service/vendor_keys.py` — a third-party key each user brings for
  themselves (PostBridge, Zernio, Apify). A request spends the key its project's
  owner saved; the instance's env key only where `allow_server_provider_keys()`
  holds, because on a hosted instance it is Duct's account and every signup
  would be spending it. A new user-facing vendor declares one `VendorKey`
  beside its client (with a `check`: the cheapest read that proves a pasted
  key works) and joins `_vendor_key` in `routes/content.py`; the routes, the
  app's `VendorKeyForm` and the tests are already generic. Never read the
  vendor's `Configs` key directly. A vendor's 401 is about that key, so it
  never reaches the browser as a 401: the app signs the user out on one.

## Sequencing rules from the plans

- Validate the brief/report shape before building heavy automation.
- Start with one customer, one tool, one brief/report end-to-end.
- Add connectors and orchestration only after the output format is useful.
- Add real-time anomaly detection after the scheduled brief flow works.

## What not to build yet

- no custom auth in backend
- no full Airbyte platform management
- no heavyweight job system beyond the planned orchestration layer
- no broad dashboard experience
- no complex cross-tool logic before the single-source MVP is producing useful output

## Configuration

`config.py` is the source of truth for what the backend reads; `.env.example`
is its documentation, and `tests/test_env_example.py` keeps the two honest.

The reason that test exists is worth knowing before adding a setting: **every
field in `Configs` has a default.** A missing variable therefore never fails —
the feature it powers silently does nothing. That makes an undocumented setting
undiscoverable rather than broken, which is how `.env.example` drifted to
covering about a third of what a running instance sets.

So when you add a setting:

- If it is a credential (matches `api_key|_secret|_token|_dsn|password|client_id|encryption_key|jwt_secret`),
  the test **requires** it in `.env.example`. That is not bureaucracy; it is the
  only signal a new contributor gets.
- If it is read from `os.environ` directly rather than through `Configs`, add it
  to `NOT_CONFIG_FIELDS` in that test with the reason. A bare exemption is a
  hole in the check.
- Renaming a field means renaming it in `.env.example` too — the stale check
  catches it, because a wrong example is worse than a missing one.

Names are shared across processes deliberately. `POSTHOG_API_KEY` is read by
this backend and by the desktop sidecar (only when the shell passes it, on the
user's Preferences switch); `GOOGLE_DESKTOP_OAUTH_CLIENT_SECRET` is read here
and compiled into the Tauri shell via `option_env!`.
One value in `.env.local` serves all of them — do not invent a `DUCT_`-prefixed
variant for the desktop half.

### Creating a migration

Always create Alembic revisions with autogenerate — `alembic revision
--autogenerate -m "..."` or `python scripts/migrations.py revision ...`. Do not
hand-write a revision file. Autogenerate diffs the models against the live
schema, which is the step that catches a column you added to a model and forgot
to migrate; a hand-written revision cannot notice what you did not think of.
Review the generated file before applying it — autogenerate is a good first
draft and a poor final one, especially for renames and server defaults.