git:20260901.39e1f4c to git:20260902.f794219

1 added, 350 removed. Audit A to B.

What this update changes in the checks

No check changed: every fixed check gives the same answer before and after this update.

A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.

- # Duct Backend — Claude Code instructions
-
- Python reporting and synthesis backend for Duct.
-
- ## Product role
-
- Per the MVP plan (duct-cloud, private), this backend is the actual product engine:
-
- - read from client-owned destinations with read-only access
- - normalize data into typed internal models
- - compute signals and comparisons
- - synthesize findings into structured output
- - deliver via email and alerts
-
- The web app owns HTML rendering. The backend produces JSON payloads only — it does not render HTML.
-
- ## MVP architecture
-
- ### Current stack
-
- - **AI synthesis:** Insights runs on V1 (LangChain / deepagents) only. Audit carries both V1 and V3 runners and defaults to V1; content is still V3. Engine selection is per request, defaulting from the `generate_engine` env var.
-
- **Consolidating on V1.** Per the engine consolidation review (duct-cloud, private), all
- agents are moving to one harness — LangChain 1.x / `deepagents` — because customers bring
- their own model (OpenAI / Gemini / Claude / OpenRouter) and the Claude Agent SDK is
- Anthropic-only by design (upstream issue #410, closed `not planned`).
- Each engine has a different status, and they imply different rules:
-
- - **V1 — the target, under construction.** Rebuilt on `create_agent` + structured output;
- `v1/graph.py` is gone. New agent work goes here.
- - **V3 — maintained, and still the production path for content.** It is *not* being
- retired yet. Keep it working: shared-code changes (`agents/core/`, `agents/audit/`,
- `agents/content/`, `schema.py`, `agents/models.py`) must keep V3 at parity, and V1
- ports land **alongside** V3 rather than replacing it.
-
- **Audit now defaults to V1** (`routes/audit.py::_resolve_agent_config`), with V3
- reachable via `engine: "v3"`. Audit was the cheap place to make the consolidation
- real — both runners already existed with the same `run_pipeline` signature and event
- vocabulary — and running V1 by default *is* how it earns the confidence retirement
- waits on. Content is the remaining port, and the expensive one: its runner is ~1350
- lines with heavy session/resume machinery, so it stays on V3 until audit has soaked.
-
- One consequence to keep in mind: **V3 is the only engine that can authenticate from a
- Claude subscription.** The Messages API rejects `sk-ant-oat…` tokens (Anthropic
- disabled third-party OAuth in Feb 2026), so retiring V3 means Claude requires an
- ANTHROPIC_API_KEY. Measured cost of that: ~$0.04 per insights synthesis.
-
- **Insights V3 was removed.** Same reason V2 was: nothing dispatched it. Both live
- routes (`routes/generate.py`, `routes/agents.py`) drive `AutonomousInsightsRunner`,
- while the V3 runner still claimed parity with the older `GenerateInsightsAgent`
- fetch/synthesize pair — an interface the routes had already left behind.
-
- So a shared change may need doing twice (V1 + V3), never more.
- Claude remains a first-class *model* through V1, so retiring V3 later costs no capability.
-
- **V2 (Google ADK) was removed.** Not on framework merit — ADK is actively developed and
- Google-backed — but because nothing dispatched its runner: `routes/generate.py` had been
- hardcoded to V1, so selecting "v2" in the UI silently served V1 while claiming otherwise.
- Its differentiators (Vertex Agent Engine deploy, `adk web`, native A2A, built-in evals)
- do not intersect this stack, and its weakest axis — provider breadth — is exactly what V1
- exists for. Its defaults were identical to V1's, so `resolve_engine` folding a stored
- `"v2"` back to V1 changed no behaviour. `agents/insights/schema_compat.py` outlived it
- for a while — it was never ADK-specific — but it went with insights V3, its last caller.
- V1 asks the provider for a typed object via `with_structured_output`, so nothing needs
- to parse a synthesis out of raw text any more.
- - **Ingestion:** Direct Google API clients (`google-ads`, `google-analytics-data`, `google-api-python-client`). Async concurrent fetching in `service/pipeline.py`.
- - **Normalization:** Lightweight Python pipeline — raw API response → typed Pydantic/SQLModel brief models. No query layer or transforms yet.
- - **Database:** PostgreSQL on Railway — SQLModel ORM, Alembic migrations, `psycopg` driver.
- - **Auth:** JWT for users; Google OAuth for connector linking (Ads, GA4, GSC, Sign-In). Project access is by membership (`project_members`), not by `projects.user_id` — always go through `service/membership.py`.
-
- **`validate_api_key` is not an authorization boundary.** `DUCT_API_KEY` ships to
- the browser as `NEXT_PUBLIC_DUCT_API_KEY`, so it proves "this is the Duct app"
- and never "this caller owns that row". A router mounted behind it *looks*
- protected and is not. Any endpoint that reads or writes a project-scoped row
- therefore needs `get_current_user` **plus** a membership check on top:
-
- - a project named in the request → `get_project_for_user`
- - a row addressed by its own id → `get_project_row_for_user`, which reads the
- project off the **row**. An endpoint that takes a row id and trusts a
- `project_id` from the request is letting the caller vouch for themselves.
- - a listing → scope it to the caller (`accessible_projects`), never to an
- unfiltered query parameter.
-
- 404, not 403, for a non-member, so the reply is not an oracle for which ids
- exist. `routes/artifacts.py` is the reference; `routes/content.py` declares
- `get_current_user` **on the router** so endpoint 45 cannot be written without
- it, and `tests/test_content_access.py` asserts that property directly.
- - **Email:** `service/email/` — Resend when `RESEND_API_KEY` is set, otherwise a logging console backend so dev/CI need no vendor account.
- - **Observability:** Sentry error tracking; optional OpenTelemetry tracing (wired via Claude Agent SDK).
- - **Hosting:** Railway — auto-deploys from `main` via GitHub integration; `railway.json` defines Railpack build + uvicorn start.
- - **CI:** GitHub Actions (`backend.yml`) — Ruff lint + pytest on every PR and push to `main`.
-
- ### Desktop (local sidecar) mode
-
- The backend runs in two shapes from one codebase. Railway is unchanged; the
- desktop build runs the same FastAPI app as a sidecar on the user's machine —
- see the engine consolidation review (duct-cloud, private) §7–8.
-
- - **Entrypoint:** `local_server.py`. Sets `DUCT_LOCAL=1`, resolves the per-user
- data dir, binds **127.0.0.1** on an OS-assigned port, and prints a single JSON
- handshake line on stdout before starting uvicorn:
- `{"duct_sidecar":1,"url":...,"port":...,"api_key":...,"data_dir":...}`.
- The Tauri shell must read the port from that line — never assume one.
- - **Data dir** (`utils/appdirs.py`): macOS `~/Library/Application Support/ai.getduct.desktop`,
- Windows `%APPDATA%\Duct`, Linux `$XDG_DATA_HOME/duct`. Created `0700`.
- Override with `--data-dir` or `DUCT_DATA_DIR`.
- - **Local mode defaults** (`Configs._apply_local_mode_defaults`): SQLite at
- `<data_dir>/duct.db`, uploads at `<data_dir>/uploads`, `init_db_on_startup=True`
- (no Alembic on a laptop). Each is only filled when unset, so
- `DATABASE_URL=postgresql://…` still works for a developer running local mode.
- - **Local API key:** generated once, persisted `0600` at `<data_dir>/local-api-key`,
- and exported as `DUCT_API_KEY` so the existing `validate_api_key` gate applies
- unchanged. It only stops other local processes driving the sidecar.
- - **JSON columns must use `models/columns.py::json_column()`**, never
- `postgresql.JSONB` directly — raw JSONB fails to compile on SQLite. The variant
- still renders JSONB on Postgres, so it produces no Alembic diff.
- - **Never name a module `models/types.py`** — it shadows the stdlib `types`.
-
- ### Database migrations
-
- Schema changes are applied **manually** with Alembic — a normal local dev step,
- distinct from an app deploy (the global "deploys go through CI/CD" rule is about
- shipping app code, not running migrations). Nothing runs migrations
- automatically: `railway.json` only starts uvicorn and there is no CI migration job.
-
- - Apply: from `backend/`, run `alembic upgrade head`. The DB URL resolves from
- `backend/.env.local` (the Railway TCP proxy) via `config.get_configs()`.
- - Inspect: `alembic current`, `alembic heads`, `alembic history`.
- - The proxy host is not resolvable inside the command sandbox, so migration
- commands run with the sandbox disabled — they need outbound network to the
- managed database's proxy domain.
- - New models must be imported in `models/__init__.py` so `SQLModel.metadata`
- picks them up for autogenerate.
- - Migrations should be additive/reversible — always provide a working `downgrade`.
-
- ### Roadmap (not in codebase yet)
-
- - **Ingestion framework:** PyAirbyte for early pilots → client-managed Airbyte later
- - **Query layer:** DuckDB + Ibis
- - **Transforms:** dbt
- - **Orchestration:** Dagster
- - **Delivery:** Resend (email) + Slack webhooks
-
- ## Product-shape constraints
-
- - Do not build a dashboard-first product here.
- - The primary value is the brief and alert output.
- - The backend should support a thin onboarding app, not depend on a rich frontend.
- - Design all outputs for operator clarity: what changed, why it matters, what to do next.
-
- ## Current directory structure
-
- - `service/google/brief.py` — Google Ads brief normalization (loads demo from `data/<connector_id>/`, default `google_ads`)
- - `service/google/schema.py` — typed Google Ads brief payload (dataclasses / JSON contract)
- - `agents/insights/prompts.py` — synthesis system + user prompts (e.g. Google Ads weekly insight brief)
- - `routes/auth.py` — OAuth by connector (`/auth/connectors/{connector_id}/oauth/...`)
- - `routes/generate.py` — `POST /api/insights/generate` for interactive brief + LangChain synthesis envelope
- - `routes/project_members.py` — project members + email invitations (`docs/engineering/project-collaboration-plan.md`)
- - `service/membership.py` — project access checks (owner vs collaborator) and invite token handling
- - `data/google_ads/` — `google-ads-report.json` (demo brief), `raw/demo_raw_payload.json`
-
- ## Agent-type architecture
-
- The `agents/` directory is organised by agent type. Each type is independent and has its own goals, tools, prompts, schema, and versioned runners.
-
- ```
- agents/
- ├── engines.py — engine/provider/model registry (shared across all agent types)
- ├── models.py — Provider, ModelName enums (shared)
- ├── insights/ — Insights agent (paid ads + organic growth intelligence)
- │ ├── v1/ — LangChain runner (the only insights engine)
- │ └── goals/, tools/, schema.py, registry.py, prompts/
- ├── audit/ — future: SEO audit agent
- └── content/ — future: Content marketing agent (plans, posts, publishing)
- ```
-
- Route convention: each agent type gets its own route prefix:
- - `POST /api/insights/generate` — exists
- - `POST /api/audit/run` — future
- - `POST /api/content/plan/stream` / `POST /api/content/post/stream` — future
-
- Cross-agent invocations are modelled at the frontend level (e.g. audit findings carry an `invoke_insights` action that pre-populates the insights wizard). Backend agents remain decoupled — no direct calls between agent types.
-
- ## Agent ports — the harness boundary
-
- We rent an agent harness; we do not marry one. The full declaration (with the
- port table and the external standard behind each) lives in
- `agents/core/ports/__init__.py` — read it before adding anything that touches a
- framework. The rules it implies:
-
- - **Never write an `AgentHarness` interface.** Harnesses differ in capability,
- not just API shape; the intersection loses the reason to use one and the
- union means maintaining a framework. The harness stays harness-shaped inside
- a runner, and only its *boundary* is standardized.
- - **Domain code imports no framework.** Tool bodies, prompts, schemas, goals and
- scoring are plain Python. `agents/core/memory_tools.py` is the reference
- shape: `_remember_sync` / `_search_sync` / `_get_sync` hold the logic, and
- `build_memory_tools_lc` / `build_memory_tools_sdk` are thin binders.
- - **Framework imports live only in adapters** — runners, binders, and the
- named shims. `tests/test_harness_boundaries.py` enforces this and lists the
- allowlist; adding a file to that list is a deliberate act, not a fix for a
- failing test.
- - **Write the adapter on the second implementation, not the first.** One
- implementation is a guess. Do not abstract the session registry until the
- LangGraph checkpointer actually lands beside it.
- - **Pick the lowest rung that works.** LangChain 1.x is layered, and the layers
- carry different stability guarantees: `init_chat_model` and `create_agent` are
- on the semver-stable 1.x LTS surface (no breaking changes until 2.0), while
- `deepagents` is 0.x with no stability policy and a weekly cadence. Reaching for
- `deepagents` where `create_agent` suffices buys churn for nothing.
-
- The rung is a property of the agent, not of the agent *type*, and it can move
- when the agent's job does. The legacy insights pipeline
- (`agents/insights/v1/agent.py`: one tool loop, one structured-output call) is
- a `create_agent` job and stays one. The autonomous insights session
- (`agents/insights/v1/runner.py`) is on `deepagents` because it needs four
- things `create_agent` lacks — a planning loop, subagents, skills, and the
- `interrupt_on` upgrade path — and the phase plan spends all four. Audit's V1
- runner is still `create_agent`; content is on the SDK.
-
- Two consumers of the 0.x pin now, so `tests/test_deepagents_harness.py`
- matters more, not less: run it before moving the pin.
- - **`deepagents` is pinned exactly**, not with a caret — it changes behaviour in
- minors (task planning became opt-in in 0.7). `tests/test_deepagents_harness.py`
- is the upgrade gate; run it before moving the pin.
-
- ## Artifact contract
-
- The app lists top-level `*.json` briefs in `data/google_ads/` (e.g. `google-ads-report.json`) for local dev; user-generated insights are returned from `POST /api/insights/generate` and stored client-side (`localStorage`).
-
- The JSON contract:
- - `source_metadata.theme` — theme key (`paid_ads`, `product_intelligence`, `organic_growth`); the app resolves accent colors from this
- - `source_metadata.generated_at` — ISO 8601 timestamp
- - All other fields follow the typed models in `service/google/schema.py`
-
- Do not write HTML from the backend. Do not reference `themes.json` or HTML templates — those have moved to the app.
-
- ## Code design rules
-
- - Normalize first, synthesize second.
- - Keep typed schemas central and explicit.
- - The backend is a data pipeline, not a renderer. Output JSON; let the app handle presentation.
- - Separate ingestion, normalization, synthesis, and delivery concerns.
- - Prefer extensible evidence models so future tools can enrich the same findings.
-
- ### Shared helpers — check here before writing a local copy
-
- Each of these replaced a family of divergent duplicates (23 private
- `_utcnow()` definitions, five hand-rolled retry loops). A new local copy
- re-opens that drift, so reach for these first and extend them when they
- don't fit.
-
- - `utils/dates.py` — `utcnow()`, `now_iso()`, `parse_iso()`, `last_n_days()`.
- **Never `datetime.now()` or `datetime.utcnow()`**: every persisted timestamp
- is UTC-aware, and naive ones compare and serialise inconsistently across
- SQLite (desktop) and Postgres (Railway). Stdlib-only leaf module, so
- `models/` can use `default_factory=utcnow` without an import cycle.
- - `utils/strings.py` — `slugify()`, `titleize()`.
- - `utils/formatting.py` — `money()`, `number()`, `percent()`, `multiplier()`,
- `safe_divide()`.
- - `service/memory.py` — agent memory (`project_memories`): `remember()` is the
- ONLY write path (it redacts secrets, dedupes, honours the pause switch, and
- closes the previous value of a state key), `search()` is the only read path
- (Postgres FTS, SQLite LIKE), and `build_memory_context()` assembles the prompt
- blocks. Never write the table directly and never re-render the digest locally
- — the supersession and never-raise contracts live in that module.
- `service/memory_consolidation.py` owns the post-session extraction pass; its
- model output is a proposal that still goes through `remember()`. Tools for
- both harnesses are in `agents/core/memory_tools.py` and the shared prompt
- stanza is `agents/core/prompts.py::MEMORY_DISCIPLINE`. Per-project memory goes
- in the USER message, never the system prompt.
- `search(time_aware=True, rank=True)` is the question-shaped read — it reads a
- date range out of the words, treats a named kind as a filter, matches on ANY
- term and then tightens (all terms → two → one), and ranks by relevance +
- recency + importance + recall. Leave both off for a filter form like the
- timeline, whose inputs are the user's explicit instructions. Retrieval makes
- no model calls, by design. `tests/eval/memory_recall.py` holds the 50-question
- set (`pytest tests/test_memory_retrieval.py -s` prints the per-axis report);
- it exists because it caught the AND-everything query bug that made questions
- retrieve nothing, so extend it before tuning retrieval by feel.
- - `agents/core/lc.py` — the LangChain adapter every V1 runner shares:
- `resolve_chat_model` (model transport) and `stream_agent` (LangChain stream →
- the `AgentEvent` vocabulary), plus `build_ask_user_tool`, the LangChain half
- of the human-in-the-loop port. Extracted from `agents/audit/v1/runner.py` when
- insights became the second V1 runner. A V1 runner should not talk to
- `init_chat_model` or drive `astream` itself.
- - `service/artifact_store.py` — versioned artifact persistence. `ArtifactPersister`
- wraps a runner's emit and stores every `ARTIFACT_VERSION` event; an **adapter**
- (`ArtifactVersion` + a `Callable[[dict], ArtifactVersion]`) reads one version out
- of whatever payload that agent emits. A new agent writes an adapter — audit's
- validates an `AuditReport`, insights' reads a written brief — and never a second
- persistence path. Storing is the store's job; understanding the payload is the
- adapter's.
- - `agents/tools/execution_tools.py` — the staged-execution tool surface, as a
- binder pair (`build_execution_tools_lc` / `build_execution_mcp_server`) over
- shared descriptions, arg schemas and domain functions. The surface is
- deliberately asymmetric: propose, inspect and roll back exist; **approve and
- apply do not, in either harness.** Autonomy (`ask | assisted | auto`) changes
- how often an agent interrupts, never what may auto-apply — `service/execution/
- policy.py` is the one place that decides, and it does not consult the model.
- - `service/rest.py` — sync HTTP transport for the reporting connectors:
- retry, backoff, rate-limit pacing, error typing. A new connector declares
- an `Endpoint` and an `ApiError` subclass and writes no transport code;
- auth headers, query encoding and pagination stay vendor-side. Not for
- `service/apify` or `service/post_bridge` — those are async, hold a
- long-lived client, and need no retry.
-
- ## Sequencing rules from the plans
-
- - Validate the brief/report shape before building heavy automation.
- - Start with one customer, one tool, one brief/report end-to-end.
- - Add connectors and orchestration only after the output format is useful.
- - Add real-time anomaly detection after the scheduled brief flow works.
-
- ## What not to build yet
-
- - no custom auth in backend
- - no full Airbyte platform management
- - no heavyweight job system beyond the planned orchestration layer
- - no broad dashboard experience
- - no complex cross-tool logic before the single-source MVP is producing useful output
-
- ## Configuration
-
- `config.py` is the source of truth for what the backend reads; `.env.example`
- is its documentation, and `tests/test_env_example.py` keeps the two honest.
-
- The reason that test exists is worth knowing before adding a setting: **every
- field in `Configs` has a default.** A missing variable therefore never fails —
- the feature it powers silently does nothing. That makes an undocumented setting
- undiscoverable rather than broken, which is how `.env.example` drifted to
- covering about a third of what a running instance sets.
-
- So when you add a setting:
-
- - If it is a credential (matches `api_key|_secret|_token|_dsn|password|client_id|encryption_key|jwt_secret`),
- the test **requires** it in `.env.example`. That is not bureaucracy; it is the
- only signal a new contributor gets.
- - If it is read from `os.environ` directly rather than through `Configs`, add it
- to `NOT_CONFIG_FIELDS` in that test with the reason. A bare exemption is a
- hole in the check.
- - Renaming a field means renaming it in `.env.example` too — the stale check
- catches it, because a wrong example is worse than a missing one.
-
- Names are shared across processes deliberately. `SENTRY_DSN` is read by this
- backend, by the desktop sidecar (only on user consent), and compiled into the
- Tauri shell via `option_env!`; `GOOGLE_DESKTOP_OAUTH_CLIENT_SECRET` likewise.
- One value in `.env.local` serves all of them — do not invent a `DUCT_`-prefixed
- variant for the desktop half.
-
+ AGENTS.md