# Duct Backend — Claude Code instructions

Python reporting and synthesis backend for Duct.

## Product role

Per the MVP plan (duct-cloud, private), this backend is the actual product engine:

- read from client-owned destinations with read-only access
- normalize data into typed internal models
- compute signals and comparisons
- synthesize findings into structured output
- deliver via email and alerts

The web app owns HTML rendering. The backend produces JSON payloads only — it does not render HTML.

## MVP architecture

### Current stack

- **AI synthesis:** Insights runs on V1 (LangChain / deepagents) only. Audit carries both V1 and V3 runners and defaults to V1; content is still V3. Engine selection is per request, defaulting from the `generate_engine` env var.

  **Consolidating on V1.** Per the engine consolidation review (duct-cloud, private), all
  agents are moving to one harness — LangChain 1.x / `deepagents` — because customers bring
  their own model (OpenAI / Gemini / Claude / OpenRouter) and the Claude Agent SDK is
  Anthropic-only by design (upstream issue #410, closed `not planned`).
  Each engine has a different status, and they imply different rules:

  - **V1 — the target, under construction.** Rebuilt on `create_agent` + structured output;
    `v1/graph.py` is gone. New agent work goes here.
  - **V3 — maintained, and still the production path for content.** It is *not* being
    retired yet. Keep it working: shared-code changes (`agents/core/`, `agents/audit/`,
    `agents/content/`, `schema.py`, `agents/models.py`) must keep V3 at parity, and V1
    ports land **alongside** V3 rather than replacing it.

    **Audit now defaults to V1** (`routes/audit.py::_resolve_agent_config`), with V3
    reachable via `engine: "v3"`. Audit was the cheap place to make the consolidation
    real — both runners already existed with the same `run_pipeline` signature and event
    vocabulary — and running V1 by default *is* how it earns the confidence retirement
    waits on. Content is the remaining port, and the expensive one: its runner is ~1350
    lines with heavy session/resume machinery, so it stays on V3 until audit has soaked.

    One consequence to keep in mind: **V3 is the only engine that can authenticate from a
    Claude subscription.** The Messages API rejects `sk-ant-oat…` tokens (Anthropic
    disabled third-party OAuth in Feb 2026), so retiring V3 means Claude requires an
    ANTHROPIC_API_KEY. Measured cost of that: ~$0.04 per insights synthesis.

    **Insights V3 was removed.** Same reason V2 was: nothing dispatched it. Both live
    routes (`routes/generate.py`, `routes/agents.py`) drive `AutonomousInsightsRunner`,
    while the V3 runner still claimed parity with the older `GenerateInsightsAgent`
    fetch/synthesize pair — an interface the routes had already left behind.

  So a shared change may need doing twice (V1 + V3), never more.
  Claude remains a first-class *model* through V1, so retiring V3 later costs no capability.

  **V2 (Google ADK) was removed.** Not on framework merit — ADK is actively developed and
  Google-backed — but because nothing dispatched its runner: `routes/generate.py` had been
  hardcoded to V1, so selecting "v2" in the UI silently served V1 while claiming otherwise.
  Its differentiators (Vertex Agent Engine deploy, `adk web`, native A2A, built-in evals)
  do not intersect this stack, and its weakest axis — provider breadth — is exactly what V1
  exists for. Its defaults were identical to V1's, so `resolve_engine` folding a stored
  `"v2"` back to V1 changed no behaviour. `agents/insights/schema_compat.py` outlived it
  for a while — it was never ADK-specific — but it went with insights V3, its last caller.
  V1 asks the provider for a typed object via `with_structured_output`, so nothing needs
  to parse a synthesis out of raw text any more.
- **Ingestion:** Direct Google API clients (`google-ads`, `google-analytics-data`, `google-api-python-client`). Async concurrent fetching in `service/pipeline.py`.
- **Normalization:** Lightweight Python pipeline — raw API response → typed Pydantic/SQLModel brief models. No query layer or transforms yet.
- **Database:** PostgreSQL on Railway — SQLModel ORM, Alembic migrations, `psycopg` driver.
- **Auth:** JWT for users; Google OAuth for connector linking (Ads, GA4, GSC, Sign-In). Project access is by membership (`project_members`), not by `projects.user_id` — always go through `service/membership.py`.

  **`validate_api_key` is not an authorization boundary.** `DUCT_API_KEY` ships to
  the browser as `NEXT_PUBLIC_DUCT_API_KEY`, so it proves "this is the Duct app"
  and never "this caller owns that row". A router mounted behind it *looks*
  protected and is not. Any endpoint that reads or writes a project-scoped row
  therefore needs `get_current_user` **plus** a membership check on top:

  - a project named in the request → `get_project_for_user`
  - a row addressed by its own id → `get_project_row_for_user`, which reads the
    project off the **row**. An endpoint that takes a row id and trusts a
    `project_id` from the request is letting the caller vouch for themselves.
  - a listing → scope it to the caller (`accessible_projects`), never to an
    unfiltered query parameter.

  404, not 403, for a non-member, so the reply is not an oracle for which ids
  exist. `routes/artifacts.py` is the reference; `routes/content.py` declares
  `get_current_user` **on the router** so endpoint 45 cannot be written without
  it, and `tests/test_content_access.py` asserts that property directly.
- **Email:** `service/email/` — Resend when `RESEND_API_KEY` is set, otherwise a logging console backend so dev/CI need no vendor account.
- **Observability:** Sentry error tracking; optional OpenTelemetry tracing (wired via Claude Agent SDK).
- **Hosting:** Railway — auto-deploys from `main` via GitHub integration; `railway.json` defines Railpack build + uvicorn start.
- **CI:** GitHub Actions (`backend.yml`) — Ruff lint + pytest on every PR and push to `main`.

### Desktop (local sidecar) mode

The backend runs in two shapes from one codebase. Railway is unchanged; the
desktop build runs the same FastAPI app as a sidecar on the user's machine —
see the engine consolidation review (duct-cloud, private) §7–8.

- **Entrypoint:** `local_server.py`. Sets `DUCT_LOCAL=1`, resolves the per-user
  data dir, binds **127.0.0.1** on an OS-assigned port, and prints a single JSON
  handshake line on stdout before starting uvicorn:
  `{"duct_sidecar":1,"url":...,"port":...,"api_key":...,"data_dir":...}`.
  The Tauri shell must read the port from that line — never assume one.
- **Data dir** (`utils/appdirs.py`): macOS `~/Library/Application Support/ai.getduct.desktop`,
  Windows `%APPDATA%\Duct`, Linux `$XDG_DATA_HOME/duct`. Created `0700`.
  Override with `--data-dir` or `DUCT_DATA_DIR`.
- **Local mode defaults** (`Configs._apply_local_mode_defaults`): SQLite at
  `<data_dir>/duct.db`, uploads at `<data_dir>/uploads`, `init_db_on_startup=True`
  (no Alembic on a laptop). Each is only filled when unset, so
  `DATABASE_URL=postgresql://…` still works for a developer running local mode.
- **Local API key:** generated once, persisted `0600` at `<data_dir>/local-api-key`,
  and exported as `DUCT_API_KEY` so the existing `validate_api_key` gate applies
  unchanged. It only stops other local processes driving the sidecar.
- **JSON columns must use `models/columns.py::json_column()`**, never
  `postgresql.JSONB` directly — raw JSONB fails to compile on SQLite. The variant
  still renders JSONB on Postgres, so it produces no Alembic diff.
- **Never name a module `models/types.py`** — it shadows the stdlib `types`.

### Database migrations

Schema changes are applied **manually** with Alembic — a normal local dev step,
distinct from an app deploy (the global "deploys go through CI/CD" rule is about
shipping app code, not running migrations). Nothing runs migrations
automatically: `railway.json` only starts uvicorn and there is no CI migration job.

- Apply: from `backend/`, run `alembic upgrade head`. The DB URL resolves from
  `backend/.env.local` (the Railway TCP proxy) via `config.get_configs()`.
- Inspect: `alembic current`, `alembic heads`, `alembic history`.
- The proxy host is not resolvable inside the command sandbox, so migration
  commands run with the sandbox disabled — they need outbound network to the
  managed database's proxy domain.
- New models must be imported in `models/__init__.py` so `SQLModel.metadata`
  picks them up for autogenerate.
- Migrations should be additive/reversible — always provide a working `downgrade`.

### Roadmap (not in codebase yet)

- **Ingestion framework:** PyAirbyte for early pilots → client-managed Airbyte later
- **Query layer:** DuckDB + Ibis
- **Transforms:** dbt
- **Orchestration:** Dagster
- **Delivery:** Resend (email) + Slack webhooks

## Product-shape constraints

- Do not build a dashboard-first product here.
- The primary value is the brief and alert output.
- The backend should support a thin onboarding app, not depend on a rich frontend.
- Design all outputs for operator clarity: what changed, why it matters, what to do next.

## Current directory structure

- `service/google/brief.py` — Google Ads brief normalization (loads demo from `data/<connector_id>/`, default `google_ads`)
- `service/google/schema.py` — typed Google Ads brief payload (dataclasses / JSON contract)
- `agents/insights/prompts.py` — synthesis system + user prompts (e.g. Google Ads weekly insight brief)
- `routes/auth.py` — OAuth by connector (`/auth/connectors/{connector_id}/oauth/...`)
- `routes/generate.py` — `POST /api/insights/generate` for interactive brief + LangChain synthesis envelope
- `routes/project_members.py` — project members + email invitations (`docs/engineering/project-collaboration-plan.md`)
- `service/membership.py` — project access checks (owner vs collaborator) and invite token handling
- `data/google_ads/` — `google-ads-report.json` (demo brief), `raw/demo_raw_payload.json`

## Agent-type architecture

The `agents/` directory is organised by agent type. Each type is independent and has its own goals, tools, prompts, schema, and versioned runners.

```
agents/
├── engines.py          — engine/provider/model registry (shared across all agent types)
├── models.py           — Provider, ModelName enums (shared)
├── insights/           — Insights agent (paid ads + organic growth intelligence)
│   ├── v1/             — LangChain runner (the only insights engine)
│   └── goals/, tools/, schema.py, registry.py, prompts/
├── audit/              — future: SEO audit agent
└── content/            — future: Content marketing agent (plans, posts, publishing)
```

Route convention: each agent type gets its own route prefix:
- `POST /api/insights/generate` — exists
- `POST /api/audit/run` — future
- `POST /api/content/plan/stream` / `POST /api/content/post/stream` — future

Cross-agent invocations are modelled at the frontend level (e.g. audit findings carry an `invoke_insights` action that pre-populates the insights wizard). Backend agents remain decoupled — no direct calls between agent types.

## Agent ports — the harness boundary

We rent an agent harness; we do not marry one. The full declaration (with the
port table and the external standard behind each) lives in
`agents/core/ports/__init__.py` — read it before adding anything that touches a
framework. The rules it implies:

- **Never write an `AgentHarness` interface.** Harnesses differ in capability,
  not just API shape; the intersection loses the reason to use one and the
  union means maintaining a framework. The harness stays harness-shaped inside
  a runner, and only its *boundary* is standardized.
- **Domain code imports no framework.** Tool bodies, prompts, schemas, goals and
  scoring are plain Python. `agents/core/memory_tools.py` is the reference
  shape: `_remember_sync` / `_search_sync` / `_get_sync` hold the logic, and
  `build_memory_tools_lc` / `build_memory_tools_sdk` are thin binders.
- **Framework imports live only in adapters** — runners, binders, and the
  named shims. `tests/test_harness_boundaries.py` enforces this and lists the
  allowlist; adding a file to that list is a deliberate act, not a fix for a
  failing test.
- **Write the adapter on the second implementation, not the first.** One
  implementation is a guess. Do not abstract the session registry until the
  LangGraph checkpointer actually lands beside it.
- **Pick the lowest rung that works.** LangChain 1.x is layered, and the layers
  carry different stability guarantees: `init_chat_model` and `create_agent` are
  on the semver-stable 1.x LTS surface (no breaking changes until 2.0), while
  `deepagents` is 0.x with no stability policy and a weekly cadence. Reaching for
  `deepagents` where `create_agent` suffices buys churn for nothing.

  The rung is a property of the agent, not of the agent *type*, and it can move
  when the agent's job does. The legacy insights pipeline
  (`agents/insights/v1/agent.py`: one tool loop, one structured-output call) is
  a `create_agent` job and stays one. The autonomous insights session
  (`agents/insights/v1/runner.py`) is on `deepagents` because it needs four
  things `create_agent` lacks — a planning loop, subagents, skills, and the
  `interrupt_on` upgrade path — and the phase plan spends all four. Audit's V1
  runner is still `create_agent`; content is on the SDK.

  Two consumers of the 0.x pin now, so `tests/test_deepagents_harness.py`
  matters more, not less: run it before moving the pin.
- **`deepagents` is pinned exactly**, not with a caret — it changes behaviour in
  minors (task planning became opt-in in 0.7). `tests/test_deepagents_harness.py`
  is the upgrade gate; run it before moving the pin.

## Artifact contract

The app lists top-level `*.json` briefs in `data/google_ads/` (e.g. `google-ads-report.json`) for local dev; user-generated insights are returned from `POST /api/insights/generate` and stored client-side (`localStorage`).

The JSON contract:
- `source_metadata.theme` — theme key (`paid_ads`, `product_intelligence`, `organic_growth`); the app resolves accent colors from this
- `source_metadata.generated_at` — ISO 8601 timestamp
- All other fields follow the typed models in `service/google/schema.py`

Do not write HTML from the backend. Do not reference `themes.json` or HTML templates — those have moved to the app.

## Code design rules

- Normalize first, synthesize second.
- Keep typed schemas central and explicit.
- The backend is a data pipeline, not a renderer. Output JSON; let the app handle presentation.
- Separate ingestion, normalization, synthesis, and delivery concerns.
- Prefer extensible evidence models so future tools can enrich the same findings.

### Shared helpers — check here before writing a local copy

Each of these replaced a family of divergent duplicates (23 private
`_utcnow()` definitions, five hand-rolled retry loops). A new local copy
re-opens that drift, so reach for these first and extend them when they
don't fit.

- `utils/dates.py` — `utcnow()`, `now_iso()`, `parse_iso()`, `last_n_days()`.
  **Never `datetime.now()` or `datetime.utcnow()`**: every persisted timestamp
  is UTC-aware, and naive ones compare and serialise inconsistently across
  SQLite (desktop) and Postgres (Railway). Stdlib-only leaf module, so
  `models/` can use `default_factory=utcnow` without an import cycle.
- `utils/strings.py` — `slugify()`, `titleize()`.
- `utils/formatting.py` — `money()`, `number()`, `percent()`, `multiplier()`,
  `safe_divide()`.
- `service/memory.py` — agent memory (`project_memories`): `remember()` is the
  ONLY write path (it redacts secrets, dedupes, honours the pause switch, and
  closes the previous value of a state key), `search()` is the only read path
  (Postgres FTS, SQLite LIKE), and `build_memory_context()` assembles the prompt
  blocks. Never write the table directly and never re-render the digest locally
  — the supersession and never-raise contracts live in that module.
  `service/memory_consolidation.py` owns the post-session extraction pass; its
  model output is a proposal that still goes through `remember()`. Tools for
  both harnesses are in `agents/core/memory_tools.py` and the shared prompt
  stanza is `agents/core/prompts.py::MEMORY_DISCIPLINE`. Per-project memory goes
  in the USER message, never the system prompt.
  `search(time_aware=True, rank=True)` is the question-shaped read — it reads a
  date range out of the words, treats a named kind as a filter, matches on ANY
  term and then tightens (all terms → two → one), and ranks by relevance +
  recency + importance + recall. Leave both off for a filter form like the
  timeline, whose inputs are the user's explicit instructions. Retrieval makes
  no model calls, by design. `tests/eval/memory_recall.py` holds the 50-question
  set (`pytest tests/test_memory_retrieval.py -s` prints the per-axis report);
  it exists because it caught the AND-everything query bug that made questions
  retrieve nothing, so extend it before tuning retrieval by feel.
- `agents/core/lc.py` — the LangChain adapter every V1 runner shares:
  `resolve_chat_model` (model transport) and `stream_agent` (LangChain stream →
  the `AgentEvent` vocabulary), plus `build_ask_user_tool`, the LangChain half
  of the human-in-the-loop port. Extracted from `agents/audit/v1/runner.py` when
  insights became the second V1 runner. A V1 runner should not talk to
  `init_chat_model` or drive `astream` itself.
- `service/artifact_store.py` — versioned artifact persistence. `ArtifactPersister`
  wraps a runner's emit and stores every `ARTIFACT_VERSION` event; an **adapter**
  (`ArtifactVersion` + a `Callable[[dict], ArtifactVersion]`) reads one version out
  of whatever payload that agent emits. A new agent writes an adapter — audit's
  validates an `AuditReport`, insights' reads a written brief — and never a second
  persistence path. Storing is the store's job; understanding the payload is the
  adapter's.
- `agents/tools/execution_tools.py` — the staged-execution tool surface, as a
  binder pair (`build_execution_tools_lc` / `build_execution_mcp_server`) over
  shared descriptions, arg schemas and domain functions. The surface is
  deliberately asymmetric: propose, inspect and roll back exist; **approve and
  apply do not, in either harness.** Autonomy (`ask | assisted | auto`) changes
  how often an agent interrupts, never what may auto-apply — `service/execution/
  policy.py` is the one place that decides, and it does not consult the model.
- `service/rest.py` — sync HTTP transport for the reporting connectors:
  retry, backoff, rate-limit pacing, error typing. A new connector declares
  an `Endpoint` and an `ApiError` subclass and writes no transport code;
  auth headers, query encoding and pagination stay vendor-side. Not for
  `service/apify` or `service/post_bridge` — those are async, hold a
  long-lived client, and need no retry.

## Sequencing rules from the plans

- Validate the brief/report shape before building heavy automation.
- Start with one customer, one tool, one brief/report end-to-end.
- Add connectors and orchestration only after the output format is useful.
- Add real-time anomaly detection after the scheduled brief flow works.

## What not to build yet

- no custom auth in backend
- no full Airbyte platform management
- no heavyweight job system beyond the planned orchestration layer
- no broad dashboard experience
- no complex cross-tool logic before the single-source MVP is producing useful output

## Configuration

`config.py` is the source of truth for what the backend reads; `.env.example`
is its documentation, and `tests/test_env_example.py` keeps the two honest.

The reason that test exists is worth knowing before adding a setting: **every
field in `Configs` has a default.** A missing variable therefore never fails —
the feature it powers silently does nothing. That makes an undocumented setting
undiscoverable rather than broken, which is how `.env.example` drifted to
covering about a third of what a running instance sets.

So when you add a setting:

- If it is a credential (matches `api_key|_secret|_token|_dsn|password|client_id|encryption_key|jwt_secret`),
  the test **requires** it in `.env.example`. That is not bureaucracy; it is the
  only signal a new contributor gets.
- If it is read from `os.environ` directly rather than through `Configs`, add it
  to `NOT_CONFIG_FIELDS` in that test with the reason. A bare exemption is a
  hole in the check.
- Renaming a field means renaming it in `.env.example` too — the stale check
  catches it, because a wrong example is worse than a missing one.

Names are shared across processes deliberately. `SENTRY_DSN` is read by this
backend, by the desktop sidecar (only on user consent), and compiled into the
Tauri shell via `option_env!`; `GOOGLE_DESKTOP_OAUTH_CLIENT_SECRET` likewise.
One value in `.env.local` serves all of them — do not invent a `DUCT_`-prefixed
variant for the desktop half.
