git:20260922.f55491f to git:20260923.fbd3ec6

# canonry

## Purpose

The publishable npm package (`@canonry/canonry`, plus compatibility publish as `@ainyc/canonry`). Bundles the CLI, local Fastify server, in-process job runner, provider registry, scheduler, and pre-built SPA. This is what users install with `npm install -g @canonry/canonry`.

## Key Files

| File | Role |

|------|------|

| `src/cli.ts` | CLI entry point — shebang, telemetry, command dispatch |

| `src/cli-commands.ts` | `REGISTERED_CLI_COMMANDS` array — declarative command specs (see "How to add a CLI command") |

| `src/commands/` | Command implementations (one file per domain); their CLI specs live in `src/cli-commands/` |

| `src/client.ts` | `ApiClient` class + `createApiClient()` factory (see "ApiClient usage") |

| `src/server.ts` | Fastify server setup — mounts api-routes, serves SPA, registers providers (see "Server and SPA serving") |

| `src/embed.ts` | `resolveEmbedConfig(env, config)` — resolves embed mode (see "Embed mode (#716)") |

| `src/agent-config.ts` | `resolveAgentEnabled(env, config)` — the Aero kill-switch (rules: `src/agent/AGENTS.md`) |

| `src/job-runner.ts` | In-process job runner for visibility sweeps (see "Run completion pipeline") |

| `src/scheduler.ts` | Cron runner for all schedule kinds |

| `src/provider-registry.ts` | `ProviderRegistry` — manages provider adapters (see "Provider registration") |

| `src/run-coordinator.ts` | Post-run orchestrator — dispatches to intelligence + notifications |

| `src/intelligence-service.ts` | Runs analysis after sweeps, persists insights + health snapshots |

| `src/execute-site-audit.ts` | `executeSiteAudit` — the Technical AEO / Site Health crawl (see "Technical AEO / Site Health") |

| `src/data-refresh.ts` | `refreshAllIntegrations` — fires GSC + Bing + GA + GBP + ads syncs for a project (see "Data refresh") |

| `src/gbp-sync.ts` | `executeGbpSync` — Google Business Profile sync (see "Google Business Profile") |

| `src/ads-sync.ts` | `executeAdsSync` — OpenAI ads sync worker and insight-range builders (see "Ads sync") |

| `src/discovery-run.ts` | `executeDiscoveryRun` — Gemini-backed discovery run (see "Discovery") |

| `src/snapshot-service.ts` | One-shot prospect reports (see "Snapshot reports") |

| `src/telemetry.ts` | `trackEvent`, source attribution, per-process `sessionId`, `cli.upgraded` detection (see "Telemetry helpers") |

| `src/mcp/` | `canonry-mcp` stdio adapter over `createApiClient()`; rules in `src/mcp/AGENTS.md` and "MCP adapter" |

| `src/agent/` | Aero, the built-in agent; rules in `src/agent/AGENTS.md` |

| `build-web.ts` | Builds or reuses `apps/web/dist`, then copies changed SPA files into `assets/` (see "Local builds") |

| `assets/` | Bundled SPA output (`index.html` + `assets/*.js/css`) — **do not hand-edit**; regenerated by `build-web.ts` |

File-specific rules live under "Patterns", grouped by topic and naming their file. `docs/CODEMAP.md` is the repo-wide file index.

## Patterns

### Local builds

Use `pnpm build:cli` from the root for CLI and server changes. Use `pnpm build:web` for dashboard changes.

`pnpm build` produces the complete publishable package. `build:web --force` bypasses the dashboard cache.

The web workspace is a development dependency so recursive builds finish it before Canonry starts.

Canonry then reuses the checked web output. Keep this dependency when changing build scripts.

`build-web.ts` preserves `agent-workspace/` and checks asset references before copying.

`build:cli` skips the SPA. `build` runs both `build:cli` and `build:web`.

### How to add a CLI command

1. Create or extend a file in `src/commands/` for the domain.

2. Add a command spec to the `REGISTERED_CLI_COMMANDS` array in `src/cli-commands.ts`:

```typescript

{ path: ['mycommand', 'subcommand'], usage: 'Description', run: myHandler }

```

3. The CLI dispatches based on `path` matching argv.

### ApiClient usage (Critical)

**Always use `createApiClient()`** — never instantiate `ApiClient` directly:

```typescript

import { createApiClient } from '../client.js'

function getClient() {

return createApiClient() // handles basePath, config loading automatically

}

```

All `ApiClient` methods must return typed DTOs from `@ainyc/canonry-contracts`. Never cast responses with `as Record<string, unknown>`. Define response interfaces in `packages/contracts/` when they don't already exist, and add a typed client method for every new API endpoint.

**Every `ApiClient` method delegates to the generated SDK via `invoke()`.** Adding a new method is:

```typescript

import { getApiV1ProjectsByNameMyNewThing } from '@ainyc/canonry-api-client'

async myNewThing(name: string): Promise<MyNewDto> {

return this.invoke<MyNewDto>(() =>

getApiV1ProjectsByNameMyNewThing({ client: this.heyClient, path: { name } }),

)

}

```

`invoke()` handles base-path probing, CliError mapping, structured-error envelopes, and the `CANONRY_TRACE=1` request log. **Do not call `fetch()` directly** — ESLint blocks it in `packages/canonry/src/**` except in a handful of files that legitimately hit external HTTP (`telemetry.ts` → telemetry collector, `update-check.ts` → npm registry, `sitemap-parser.ts` → user sitemap, `commands/daemon.ts` → localhost health probe). If you need raw `fetch()` for a NEW external service, add the file to the `ignores` list in `eslint.config.js` with a one-line comment naming the service.

The legacy `request<T>()` raw-fetch wrapper was removed in v4.51; if you find any reference to it, replace with an SDK call through `invoke()`.

### MCP adapter

Rules for `canonry-mcp`, hosted MCP catalogs, guidance generation, MCP parity, and the Cloudflare deferral live in `src/mcp/AGENTS.md`.

File-level rules for the MCP pieces in this package:

- `src/mcp/server.ts` — `createCanonryMcpServer` registers all API tools, then disables non-core tiers unless `--eager`.

- `src/mcp/tool-registry.ts` — all 188 API tools, including Site Health semantic graph and page-audit evidence reads, sitemap Target discovery, and revision-pinned measurement reports, each tagged with a `tier` (`core` or one of the toolkit names).

- `src/mcp/cli.ts` — `canonry-mcp` stdio entrypoint; parses `--read-only`, `--eager`, `--scope`, plus `CANONRY_MCP_*` env. `resolveEffectiveScope()` best-effort probes `GET /keys/self` at startup and forces `read-only` when the configured key is read-only (auto-restricts the catalog to read tools; falls back to the flag scope on any probe failure).

- `src/mcp/operations-guide.ts` — compact intent routing filtered against the connection's loaded tools; generated source is `docs/agent-operations/v1.md`. No provider calls or permission grants.

- `src/commands/mcp.ts` — MCP client install helpers: `mcp install`, `mcp config` (writes to client config files only — separate from the `canonry-mcp` stdio bin). `src/mcp-clients.ts` is the registry of supported MCP clients (Claude Desktop, Cursor, Codex) — config-path resolvers and format hints used by `mcp install`/`mcp config`.

### Command output

All commands that produce output must support `--format json` for machine-parseable output. Use the format flag to switch between human-friendly tables and JSON.

There is a third, agent-first format: **`--format jsonl`** (newline-delimited JSON). Where `json` returns the whole envelope as one pretty-printed document, `jsonl` streams the command's **primary collection** one compact, self-contained record per line. It exists so an agent can read

```

canonry doctor --check 'google.auth.*' --format jsonl

```

directly — no `2>/dev/null`, no `| jq '(.checks // .results // [])[] | ...'`, no guessing the envelope key. Each line stands alone for `grep`/`head`/per-line `jq`.

`jsonl` is wired into **every collection command** (a command whose primary output is a list) — `insights`, `runs`, `evidence`, `history`, `query/keyword/competitor list`, `notify list/events`, the `google`/`bing`/`ga` reads, `ads geo search`, `ads conversions pixels/event-settings`, `traffic events/sources/status`, `discover list/show`, `content targets/sources/gaps`, `backlinks list/releases`, `project list/locations`, `agent memory list`, `agent providers`, and `doctor`. The per-command primary collection + the context each line carries is catalogued in `skills/canonry/references/canonry-cli.md` ("Output schema per command" + "Output Formats"). Composite/object commands (`status`, `ads account`, `analytics`, `ga traffic`/`attribution`, `gbp summary`, index `coverage`) have **no** `jsonl` — they stay `json`.

Conventions when adding `jsonl` to a command:

- **Emit via `emitJsonl(records)`** from `src/cli-output.js` (one `process.stdout.write`, compact, newline-terminated). Don't hand-roll `console.log(JSON.stringify(...))` per row.

- **Pick the primary collection** the agent is really after (doctor → `checks`, list commands → their array). Carry any context the line loses by leaving the envelope into each record by spreading context first, record last: `emitJsonl(rows.map(r => ({ project, ...r })))`. Project-scoped lists stamp `project`; add whatever else the envelope held (`window` for `ga *-history`, `release`/`targetDomain` for `backlinks list`, `sessionId` for `discover show`, `isDefault` for `project locations`). Global lists whose rows self-identify (`project list`, `notify events`, `backlinks releases`) emit bare.

- **Never let `--format jsonl` fall through to human text (the invariant).** A command that isn't a streamable collection must still emit its JSON document for `jsonl` — gate the machine-output branch on `isMachineFormat(format)`, **never bare `format === 'json'`**. Since `jsonl` was added as a global format value, a `=== 'json'` gate sends a `--format jsonl` request straight into the human-text `else` — an agent asks for machine output and silently gets a decorated table. Every composite/object/mutation command degrades `jsonl` → its `json` document this way; only the collection commands (which have an explicit `else if (format === 'jsonl')` streaming branch) keep a bare `=== 'json'` gate, because their `json` envelope and `jsonl` records are different outputs.

- **Watch for a local `toFormat`.** A few commands (the `agent-*` handlers) historically narrowed format to `json`-or-`text` inline, which silently swallows `jsonl`. If a command has its own `toFormat`, widen it to pass `jsonl` through (or delete it and use `opts.format` / `isMachineFormat`).

- **Errors and exit codes are unchanged** — `printCliError` renders a single-line envelope for `jsonl`, and a failing command still throws its `CliError` (so e.g. `doctor` still exits 1 after printing every check line). Agents branch on the exit code, not on parsing stderr.

- **USAGE strings** are not mass-updated to advertise `json|jsonl` (cosmetic; the global `cnry --help` and the skill reference cover discoverability) — `doctor`'s does as the exemplar.

The stable, machine-readable formats are `json` and `jsonl`; `text` is decorated and not a parse target. `isMachineFormat(format)` distinguishes them.

The update notice is the one piece of chrome that reaches agents as well as people, because an agent is usually the one running an outdated install. `cli.ts` prints it synchronously from the on-disk cache (`readCachedUpdateAvailable`) BEFORE dispatch, so it is always the first stderr line and never interleaves with output; stdout is untouched, and the registry refresh only rewrites the cache in the background. `formatUpdateNotice` picks the shape: the human banner on a terminal, one plain `[canonry] UPDATE_AVAILABLE: ...` line when stderr is captured, and one compact `{"notice":{"code":"UPDATE_AVAILABLE",...}}` line for `--format json|jsonl` (so `2>&1` still yields a valid JSON stream). The upgrade command follows `detectInstallMethod` (`CANONRY_INSTALL_METHOD`, which the published image sets to `docker`; then Homebrew `Cellar/canonry/`; then a container marker; else npm) and comes from `upgradeCommandFor` in contracts, with `upgradeCaveatFor` adding the Homebrew-lags-npm note, and the registry value must pass `isStrictSemver` (contracts) before it is cached or printed, because agents act on this text. Skipped for help and `telemetry`; silenced by the update-check opt-outs. The server-side counterpart is the `canonry.version.current` doctor check (`getServerUpdateStatus`). MCP-only agents never run the CLI, so `canonry-mcp` fetches the connected server's `/health` notice at startup (`createUpdateNoticeSource` in `src/mcp/update-notice.ts`: honours the env opt-outs in the adapter's own environment and refreshes hourly; `ApiClient.getServerUpdateAvailable` reads `/health` and `parseServerUpdateAvailable` keeps only the strict-semver versions and the install-method enum, rebuilding the command and URL from contracts so no server text reaches the initialize instructions, which clients load into the system prompt) and hosted MCP reads the server cache (`McpHttpOptions.getUpdateAvailable`); both append it to the initialize instructions and add `updateAvailable` to every `canonry_help` result. Other interactive chrome (skills auto-sync notice, setup nudge) stays gated on `process.stderr.isTTY`. Errors still go to stderr.

Chrome helpers:

- `src/setup-nudge.ts` — the stalled-setup stderr line: human-mode + TTY only, lazy state read, exempt roots

- `src/activation-notice.ts` — once-ever first-activation notice; marker written before printing, TTY-gated

### Telemetry helpers

The event catalog and envelope are under "Telemetry events" below.

- `src/telemetry.ts` — `trackEvent`, source attribution (incl. the validated `CANONRY_TELEMETRY_SOURCE` harness override), per-process `sessionId`, `cli.upgraded` detection

- `src/run-telemetry.ts` — `classifyRunError`, `buildRunCompletedProps`: keeps the `run.completed` payload composition in one spot

- `src/setup-state.ts` — `buildSetupState`: `{ provider_count, has_keywords, project_count, is_first_run }` snapshot ridden on every `cli.command`

### Logging and runtime diagnostics

- `src/logger.ts` — compatibility exports of the shared runtime logger in api-routes. Application and Fastify logging use the same pure redaction policy before stdout/stderr and durable capture. Do not add a second sanitizer or raw request logger.

- `src/commands/logs.ts` — `canonry logs`: API-backed runtime diagnostics, with identity/time filters and retention/loss metadata. JSONL preserves the complete paginated envelope. Requires host-approved operator authority plus instance-wide `logs.read`; customer admin roles and project scope are insufficient.

### API key commands

`src/commands/keys.ts` — API key commands:

- `key list` (table NAME / PREFIX / SCOPES / REACH / CREATED / LAST USED / STATUS, where REACH is `full instance` or the scoped project's name; `--format json|jsonl`)

- `key create --name <name> [--read-only | --scope <s> ...]` (prints the plaintext key ONCE with a "will not be shown again" warning; JSON mode includes the key; `--read-only` mints `scopes=['read']` and is mutually exclusive with `--scope`)

- `key revoke <id>` (confirms revocation)

- `key whoami` (introspect the CURRENT key — name / scopes / readOnly / reach / status)

Flag-driven, no prompts; delegates to `ApiClient.listApiKeys` / `createApiKey` / `revokeApiKey` / `getApiKeySelf`. Registered via `src/cli-commands/keys.ts`.

### Skills and native plugins

`src/commands/skills.ts` — `installSkills` / `listSkills` — reconciles bundled `skills/<name>/` trees into a user's `.claude/skills/<name>/` **additively**:

- missing files copied without `--force`; upstream-updated files the operator never touched refreshed; genuine local edits preserved and reported as conflicts unless `--force`.

- writes a `.canonry-skill-manifest.json` recording what canonry last wrote, and creates relative `.codex/skills/<name>` symlinks.

- `getBundledSkillSnapshots()` exposes the bundled version + per-file hashes for the `agent.skills.current` doctor check.

- auto-invoked by `canonry init` when cwd looks like a project.

`src/cli-commands/skills.ts`: CLI specs for `skills list` / `skills install [skill...] [--dir <path>] [--client claude|codex|all] [--force]`.

`scripts/copy-agent-assets.ts` is the build-time mirror of repo-root `skills/` into `assets/agent-workspace/skills/` so trees ship in the published package.

`src/agent-plugin.ts` does best-effort, secret-free, user-global Canonry plugin detection for Claude Code JSON settings and Codex TOML settings. It verifies Agent Plugins root manifests for Codex, retains the legacy client-manifest fallback, records each verified version so doctor/nudges can enforce runtime lockstep, and never controls filesystem writes.

### Run completion pipeline

Different run kinds may overlap on one project. Preserve shared provider gates

and same-kind admission guards. Project-only cancellation requires exactly one

active run; otherwise the CLI lists the candidates and requires a run ID.

Boot recovery fails every queued/running run and its active crawl attempts;

it does not resume interrupted work.

Before provider dispatch, `JobRunner` captures the resolved inputs of each official simple run.

The frozen definition records exact query text, identity, classification, location, and requested models.

Capture failure prevents provider calls. Probe and advanced runs retain their existing paths.

Queue-time configuration does not define simple runs because their inputs resolve at dispatch.

This capture does not change current report calculations or reconstruct historical definitions.

When a sweep finishes, the flow is: `JobRunner` → `RunCoordinator.onRunCompleted()` → `IntelligenceService.analyzeAndPersist()` then `Notifier.onRunCompleted()`. The coordinator runs intelligence first (synchronous) so insights are persisted before webhooks fire. Each subscriber is wrapped in an independent try/catch — one failing must not block the others.

`IntelligenceService` reads query snapshots from the DB, calls the pure analysis functions in `packages/intelligence/`, and persists insights + health snapshots. It also provides `backfill()` for reprocessing historical runs chronologically.

The notification system supports `citation.lost`, `citation.gained`, `run.completed`,

`run.failed`, `insight.critical`, `insight.high`. `insight.critical` and

`insight.high` fire when the intelligence engine generates critical- or

high-severity insights after a run — dispatched by `RunCoordinator` after

`IntelligenceService.analyzeAndPersist()` completes.

#### What may serve as a comparison baseline (Critical)

**Scope the history window in SQL, before the `LIMIT`.** `HISTORY_WINDOW_RUNS`

bounds ROWS; the window has to mean *N sweeps at this run's location*. Anything

scoped afterwards is scoped against a page that already dropped the rows it

needed. Both predicates live in `analyzableRunPredicates()` + the location

clause, and both were learned the hard way:

- **Kind.** Only `answer-visibility` writes `query_snapshots`; the other 14

kinds (`ga-sync`, `traffic-sync`, `site-audit`, …) write none. Let one win

the "previous run" slot and every detector gets an empty baseline — not a

neutral one, a baseline in which nothing was ever cited. `detectFirstCitations`

and `detectGains` report every currently-cited query as brand new,

`detectRegressions` finds no prior citation to lose so real regressions vanish

silently, and `detectPersistentGaps` breaks each query's streak on the run

that lacks it. Worse the better an instance is configured: daily syncs +

weekly sweeps means the row before a sweep is almost always a sync.

- **Location.** A sweep fans out to one run per configured location, so

siblings compete for the same row budget. At 3 locations a 5-row page holds

under 2 sweeps per location — the baseline vanishes and ALL transition

detection goes dead. At 6+, an arm falls outside its own window and used to

be refused outright: no insights *and no health snapshot*, so that location

left the dashboard rather than reading as flat.

Neither is visible in a fixture-only test. A test that seeds two sweeps at one

location never puts a foreign kind or a sibling arm in the window, which is why

both survived a green suite for so long.

Two supporting rules:

- **Eligibility is a property of the run, not of its recency.** Resolve the run

being analyzed by id under `analyzableRunPredicates()`; never find it inside

the window. Finding it there silently makes "is this analyzable?" depend on

"is it one of the N newest rows?" — that coupling is what dropped fan-out

arms. The window is then anchored with `lte(runChronologyKey, …)` so it never

looks forward, which also makes re-analyzing a historical run correct instead

of refused.

- **A run with no snapshots is not a comparison point.** The kind filter drops

sync/audit runs, but an `answer-visibility` run can still measure nothing (a

`completed` sweep with no tracked queries at sweep time) and anchor the same

false transitions, so the window also requires the run to have written

snapshots. In `analyzeAndPersist()` that is a third SQL predicate

(`exists(query_snapshots …)`) alongside kind and location — a post-LIMIT drop

would be the same trap as location, letting enough empty rows push the last

real sweep out of the window. `backfill()` loads the full chronology (no

limit), so it filters with `runIdsWithSnapshots(...)` after the read; the

comparison skips to the last run that measured either way.

All of this applies to `backfill()` as well as `analyzeAndPersist()`. Backfill

is the reanalyze path operators run to clear bad rows; without the same

predicates it rewrites exactly the insights it was invoked to remove. (Backfill

loads the full chronology rather than a window, so it needs the predicates but

not the limit reasoning.) Clearing a historical instance is `canonry backfill

insights <project>` — add `--dry-run` first to see the delta.

### Index coverage auto-refresh

`gscUrlInspections` (the index-coverage dashboard's source of truth) is populated **only** by an `inspect-sitemap` run. `gsc-sync` does not inspect URLs at all: each URL Inspection call costs ~7.1s (Google's own live-index-lookup latency plus the ~1.1s pacing its 1 req/sec soft limit needs) and one unit of a 2000/property/day quota, so inspecting inline made the sync scale with the site — measured at 240.9s of a 241.9s run for 31 URLs while the search-analytics work took 1.07s, past the dashboard's 120s poll window. `server.ts` chains a full GSC `inspect-sitemap` off the **success** of both `executeGscSync` (`gsc-sync`) and `executeBingInspectSitemap` (`bing-inspect-sitemap` — Bing's coverage sync, which has no separate `bing-sync` kind). `packages/canonry/test/gsc-sync-no-inspection.test.ts` guards the split. The chaining lives in the `onGscSyncRequested` / `onBingInspectSitemapRequested` callbacks, so it covers UI and CLI uniformly (both hit the same endpoints) and the dashboard "Refresh search data" button.

`maybeRefreshGscCoverage` (`src/coverage-refresh.ts`) owns the decision: it no-ops when GSC isn't connected for the project (so the Bing → GSC chain is silent on Bing-only projects), always skips while an `inspect-sitemap` run is `queued`/`running` (the in-flight guard, which is what collapses the GSC and Bing arms of "Refresh search data" into one sweep), and otherwise applies a spacing window sized by **who asked**:

| Caller | Window | Constant |

|---|---|---|

| Scheduled / chained | 1 h | `COVERAGE_REFRESH_MIN_INTERVAL_MS` |

| User-initiated (`opts.userInitiated`) | 2 min | `COVERAGE_REFRESH_MANUAL_MIN_INTERVAL_MS` |

The split exists because the original single window silently broke the dashboard's "Refresh search data" button. The 1 h figure was chosen when `gsc-sync` still inspected the top 50 pages inline, so skipping the full sweep only meant "slightly less thorough". After inline inspection was removed, `gsc-sync` inspects **nothing**, and a skipped sweep means no URL is inspected at all — index coverage becomes structurally incapable of changing. A user pressing the button 46 min after the daily scheduled refresh got a successful-looking sync and unchanged numbers, with `skip.recent` the only trace. `server.ts` classifies the caller via `runWasUserInitiated(db, runId)`, reading `runs.trigger` off the triggering run so both chain sites and both surfaces (CLI, dashboard) agree.

The chained run is `trigger: scheduled`; a refresh failure is logged, never bubbled into the triggering sync's result.

### Rate gating across runs (Critical)

`inspectUrlsPaced` takes its spacing from a gate named by `deps.rateGateKey` (`src/inspect-rate-gate.ts`). **The key must name the SHARED upstream limit, not the caller** — `bing:<digest of api key>`, `gsc:<propertyId>`. Omitting it gives the sweep a private gate that no other sweep observes.

That is not a style preference. The gate used to be a `let` inside `inspectUrlsPaced`, so it lived exactly as long as one call: every concurrent sweep got its own "1 req/sec" budget while Bing meters the **API key**, which one instance shares across every connected project. Pacing a second caller can ignore is not a rate limit. For the same reason **every** request start takes a slot, including the first — exempting index 0 let each concurrent sweep fire one ungated request the moment it began.

**Do not stack retry layers.** `bing-inspect-sitemap` passes `maxRetries: 0` because `bingFetch` already retries: the two multiplied rather than added (5 client attempts x 4 driver attempts = 20 HTTP requests for one throttled URL), so the breaker's five-failure budget spent ~100 requests and 335 s to inspect nothing. Retry belongs to the client (see the root AGENTS.md "Third-party HTTP calls" rule); the driver owns pacing and the circuit breaker.

`executeInspectSitemap` drives its `inspectUrl` calls through `inspectUrlsPaced` (`src/gsc-inspect-paced.ts`), which paces ~1 req/sec with jitter, retries transient rate responses with jittered exponential backoff, and trips a consecutive-failure circuit breaker. The endpoint signals per-minute quota pressure with a transient 403 (`PERMISSION_DENIED`-shaped) rather than a 429, so a 403 is treated as a soft, retryable rate signal here while genuine 401/400 stay non-retryable. When the breaker trips, `inspect-sitemap` fails the run, so a quota or property-access outage does not overwrite the coverage snapshot with a misleading all-not-indexed reading.

### GSC sitemap submission

`src/gsc-sitemap-submission.ts`: `dedupeGscSitemapUrls` + `resolveDiscoveredGscSitemapUrls` (indexes vs all-files, 4× parallel) + `requireGscSitemapUrls` + `submitGscSitemapBatches` (50-url batches).

### Data refresh

`src/data-refresh.ts`: `refreshAllIntegrations` — fires GSC + Bing + GA + GBP + ads syncs for a project via the in-process API client, `Promise.allSettled` for per-integration isolation. Wired to the scheduler's `data-refresh` kind in `server.ts`.

### Backfill behavior

`canonry backfill answer-visibility` does more than recompute `answerMentioned`. It also reparses stored provider `raw_response` payloads for supported API providers (OpenAI, Claude, Gemini, Perplexity) and refreshes derived snapshot fields such as `citationState`, `citedDomains`, `groundingSources`, and `searchQueries`.

The command lives in `src/commands/backfill.ts` (historical recomputation for answer visibility fields and insights).

### Server and SPA serving

- `assets/`: `server.ts` default `assetsDir` is this dir; `createServer({assetsDir: '/tmp/my-dist'})` override lets tests/custom builds point elsewhere without forking the package.

- `src/server.ts`: `createServer` opts take an optional `assetsDir` override (default = bundled `assets/`) so integration tests can point at a temp `index.html`.

- The SPA document's embed config and headers are covered in "Embed mode (#716)".

### Provider registration

Providers are registered at server startup in `server.ts`. Each provider adapter (from `packages/provider-*`) is imported and added to the `ProviderRegistry`. Projects reference providers by name.

### Health endpoint

`GET /health` (also served at `<basePath>health`) exposes `basePath` for auto-discovery and identifies the build and the deployment it is running as:

```json

{

"status": "ok",

"service": "canonry",

"version": "4.193.0",

"commit": "eed745d5c1f0a4b6e2d8c9a7b3f1e0d2c4b6a8f0",

"instance": { "name": "gjelina-demo", "role": "client-demo" },

"basePath": "/canonry"

}

```

Every field after `version` is optional and is omitted rather than nulled, so consumers key on presence. Adding fields here is fine; renaming or removing one is a breaking change (see root `AGENTS.md` → "API Surface and Stability").

- `commit`: the git sha the bundle was built from. `packages/canonry/tsup.config.ts` stamps it at build time from `git rev-parse HEAD` (`packages/canonry/scripts/build-commit.ts`). A build without git omits the stamp and the server falls back to the `CANONRY_COMMIT` env var at runtime; unset as well means the field is omitted.

- `instance`: read at boot from `CANONRY_INSTANCE` (`name`) and `CANONRY_INSTANCE_ROLE` (`role`). Omitted entirely when `CANONRY_INSTANCE` is unset; `role` is dropped when its var is unset. Role is free text, but use the convention so fleet tooling can group on it: `internal` (our own engines), `client-demo` (a prospect's demo tenant), `client-trial` (a client on trial), `preview` (a branch or PR preview).

- `basePath`: omitted when not configured.

- `updateAvailable`: `{ current, latest, url, upgradeCommand, installMethod }` (`installMethod` is `npm`, `homebrew`, or `docker`, detected from `CANONRY_INSTALL_METHOD`, a Homebrew `Cellar/canonry/` path, or a container marker; `upgradeCommand` and `url` come from fixed contracts helpers, never free text) when a newer `@canonry/canonry` is on npm (`packages/canonry/src/update-check.ts`); omitted otherwise, or when the check is opted out.

### Embed mode (#716)

`canonry serve --embed` (and `canonry start --embed`, which forwards the flags) is an opt-in, read-only, chromeless render guarded by `Content-Security-Policy: frame-ancestors`. The server-side details from `src/embed.ts` and `src/server.ts` are below. Two constraints are easy to miss:

- Cross-origin embeds cannot use the SameSite=Lax session cookie (it is not sent in a cross-site iframe). v1 supports a same-origin embed (cookie flows) OR a self-hosted build with a read-only VITE_API_KEY (client-visible).

- Embed mode is not an `/api/v1` operation, so it has no MCP tool (same precedent as `--base-path`).

`src/embed.ts`:

- `resolveEmbedConfig(env, config)` resolves embed mode (#716) from `CANONRY_EMBED` / `CANONRY_EMBED_ORIGINS` / `CANONRY_EMBED_VIEWS` / `CANONRY_EMBED_PROJECT_TABS` layered over config.yaml `embed:` (env over config, mirroring basePath).

- Delegates origin normalization + the `frame-ancestors` value + the client block to the pure helpers in `@ainyc/canonry-contracts` (`normalizeFrameOrigin` / `parseOriginList` / `frameAncestorsHeaderValue` / `buildEmbedClientConfig`).

- `enabled` is decoupled from origins so `--embed` without origins fails closed; an empty views/projectTabs list collapses to `undefined` (= all).

- `projectTabs` is the project-page TAB allowlist (overview/technical-aeo/search-console/activity/backlinks/...) the embedded dashboard renders — finer than `views`, which only gates whole top-level routes.

- Serve/start flags: `--embed`, `--embed-allow-origin <origin>…`, `--embed-view <view>…`, `--embed-project-tab <tab>…` (in `cli-commands/system.ts` → env via `applyServerEnv`; `start` forwards them through `buildServeForwardArgs` in `commands/daemon.ts`).

`src/server.ts`:

- Read-only embed mode (#716): resolves `resolveEmbedConfig(process.env, config)` at boot; when enabled, `injectConfig` appends an `embed` block to `window.__CANONRY_CONFIG__` and the single `sendSpaDocument` chokepoint (used by `serveIndex` AND the deep-link `setNotFoundHandler` fallback) emits `Content-Security-Policy: frame-ancestors …` (fail-closed to `'none'`).

- The `embed.projectTabs` value may be overridden PER REQUEST by the `X-Canonry-Embed-Tabs` header (`embedClientConfigForRequest` in contracts): the Embed v2 `/e` proxy sets it per dashboard from `embed_dashboards`, and the end client cannot reach the loopback engine to set it (presentational only; the API key scope is the data boundary). Absent header = the boot-wide config.

- When embed is off, the injected config + headers are byte-for-byte unchanged.

### Technical AEO / Site Health

`src/scheduler.ts`: `onSiteAuditRequested` starts a Technical AEO full crawl unless one is active.

`src/execute-site-audit.ts` — `executeSiteAudit` runs `@canonry/aeo-audit`'s `runSiteCrawl`:

- Events update an attempt graph with idempotent receipts. Terminal runs keep immutable complete or partial snapshots. Default reads select only the latest complete snapshot.

- The page budget defaults to 1,000; an unset edge budget is left unset so the engine derives it from the resolved page count (floored at 100,000), and an explicit one is a ceiling that replaces the derivation. Hard limits are 50,000 pages and 1,000,000 edges.

- The engine (>= 7.1.0) derives `maxFetches`/`maxDurationMs`/`maxBytes`/`maxEdges` from the page budget natively and honours explicit values exactly, so this file passes ONLY `maxPages`/`maxEdges` (the operator-facing limits) and never a fetch-side budget — setting one would pin it and fight the derivation.

- The engine also makes the dead-link split itself (6.0.0+): `deadLinks.findings` always carry a real 4xx/5xx status and `deadLinks.unverified` carries the targets the crawl could not check (timeout, reset socket, throttled 429), and ONLY findings are written to `site_crawl_findings` — every reader of that table renders a row as a broken link, and a crawl timeout is not evidence of one.

- `deadLinkCheckedCount` excludes unfetchable targets for the same reason — a URL that never answered was attempted, not checked.

`src/site-crawl-template-links.ts` — template-link classification:

- `classifySiteCrawlTemplateLinks` marks nav, header, and footer links once per attempt, after the crawl and BEFORE graph layout (which excludes them from the physics).

- DOM placement (`placement_*_occurrences`, from the crawler's landmark ruleset) decides every link the page answered for: any content occurrence makes the link editorial, navigation with no content occurrence makes it chrome.

- Ubiquity is the FALLBACK for links the page said nothing about and for pre-4.7.0 scans that recorded no placement; it is a whole-crawl fact, so it cannot be decided per event.

- Links stream in bounded batches; the derivation is the contract's `classifyTemplateLinkEdge`, shared with the tests.

- The result reports which rules ran (`applied-placement`, `applied-placement-with-ubiquity`, `applied-placement-partial`, or `applied`), which the snapshot records beside `link_placement_ruleset_version`. With no placement and fewer than 15 fetched pages it marks nothing and returns `unavailable-too-few-pages`.

- It writes a STRICT BOOLEAN `is_template` in every case: a link no rule could measure is a real `false` graded `unmeasured`, never a NULL, so the layout input, the graph sample, both link filters, the totals, the map legend, and the inspector tiles all keep ONE definition of a content link. NULL is reserved for rows that predate classification.

- The whole-scan state comes from a shared tally (`observeTemplateLinkDetection`) that both this pass and the one-shot classifier fold into, and `applied-placement-partial` is scoped to ANCHOR edges so a redirect cannot report that a well-marked-up site is missing landmarks.

`src/commands/technical-aeo.ts`: dead-link analysis requires `--check-dead-links`. The dead-link renderers print `unverified` as "could not be checked" and never fold it into `found`: those are different claims and only the first may reach a client.

### Advanced Measurement commands

`src/commands/measurement-plan.ts` — Advanced Measurement plan commands: `measurement-plan show` / `versions` / `publish` / `retire` / `discover` / `report`, plus the two per-Property reads:

- `measurement-plan property <project> --target-key <key>` (one Property out of `GET /measurement-overview?scope=property`, rendering mention/citation coverage and the per-answer-engine split)

- `measurement-plan property-evidence <project> --target-key <key>` (cursor-paged `GET /measurement-property-evidence`):

- `--shape answers` switches a row from one cited URL to one measured ANSWER with its cited URLs nested and both signals on the row — the only shape that shows the answers a Property was not cited in

- `--format jsonl` streams one row per line in either shape, under a header line carrying the shape, the measurement state and the cursor, and an unknown mention prints `not measured` rather than `no`

- The per-Property reads share `--query-class` / `--provider` / `--location` / `--run-id`, and both render an unavailable metric as `not measured (<reason>)` — never a percentage, so an unmeasured class can't read as a measured zero. `--format json` is the endpoint response verbatim.

- `measurement-plan advanced <project> <operation> [<json|->]` is the compact typed-JSON bridge for the remaining v2 reads/writes; its paged/list reads stream a metadata header and records with `--format jsonl`.

Registered via `src/cli-commands/measurement-plan.ts`.

### Visibility stats, reports, and exports

`src/commands/visibility-stats.ts`:

- `visibility-stats <project>` — aggregated per-query mention/citation rates with sample size (`--since`/`--until`/`--month <YYYY-MM>`/`--last-runs`/`--by-provider`/`--share-of-voice`); `--share-of-voice` adds pooled project-vs-competitor share of voice to the envelope; collection command → jsonl streams one record per query.

- Also `visibility-compare <project> --from <YYYY-MM> --to <YYYY-MM>` (`showVisibilityCompare`) — month-over-month AEO comparison rendered as the statistician-panel table (share-of-voice-led with `*` drift-robust marker, each cell `point [ci-lo, ci-hi]`, within-noise/moved verdict, basket + low-sweep caveat + model-change note); object command, `--format json` (jsonl degrades to json).

- Delegates to `ApiClient.getVisibilityStats` / `getVisibilityCompare`. Registered via `src/cli-commands/visibility-stats.ts`.

`src/commands/report.ts`: `runReportCommand` — `canonry report <project>` — fetches `/report` JSON, renders self-contained HTML to disk via `renderReportHtml` from `@ainyc/canonry-api-routes`.

`src/commands/results-export.ts`: `exportResults` — `canonry results export <project>` — downloads the historical query × provider observations attachment via `ApiClient.downloadResultsExport`.

- `--format json|csv` selects the artifact type; `--since`/`--until` inclusive run-creation window; `--include-probes` opt-in; `--output <path>` or `-` for stdout, default = server-suggested filename via `path.basename`, path-traversal safe.

- Distinct from `commands/export-cmd.ts` (project CONFIG export).

- Registered via `src/cli-commands/operator.ts`; API op is MCP `excluded-protocol` (bulk attachment).

### Snapshot reports

`src/snapshot-service.ts` — one-shot prospect reports. `snapshot --provider <name>...` / `--provider-mode all|api|browser` map to API `providers` / `providerMode` and `canonry_snapshot` in the discovery toolkit. Selection constrains answers AND analysis. Browser-only selection requires manual queries. Reject unavailable providers and mode conflicts before external work.

### Discovery

`src/discovery-run.ts`:

- `executeDiscoveryRun` — fires `executeDiscovery` (api-routes orchestrator) with Gemini-backed seed/embed/probe/classifyDomains deps, writes the `discovery.basket-divergence` insight, and hands off to `RunCoordinator.onRunCompleted` so Aero wakes up with a bucket-count payload. Forwards the run's resolved `locations` into `executeDiscovery` → `deps.seed`.

- `classifyDomains` is one plain-text `generateText` call per session; `buildClassificationPrompt` / `parseClassificationResponse` are exported pure helpers that build the `domain => category` prompt and forgivingly parse the model's reply into a `DiscoveryDomainClassification`.

- `buildSeedPrompt` / `buildLocationConstraint` are exported pure helpers that build the Gemini seed prompt — when locations are present they geo-constrain the prompt and (for 2+ locations) add a per-area seed quota of `floor(DEFAULT_SEED_COUNT / locationCount)`.

- The probe dep computes BOTH signals from the normalized result: `citationState` from cited domains AND `answerMentioned` from the answer text via the shared `determineAnswerMentioned` (brand names from `effectiveBrandNames`, domains from `canonicalDomains`) — same helper the answer-visibility writer uses, so discovery and sweeps agree on "mentioned".

- The embed dep wraps `embedQueries` in `embedWithRetry` (exported, testable) — shared `withRetry` + `isRetryableHttpError` with a per-attempt wall-clock timeout, so a transient blip between the two paid grounded phases no longer fails the session while a permanent 4xx still does.

- `buildDefaultDeps` is exported for unit-testing `probe()` offline.

`src/commands/discover.ts` — discovery commands:

- `discover run` (kick off ICP → seed → embed → probe pipeline; `--locations` geo-constrains seed generation to a project-location subset)

- `discover seed` (alias for run today; the phase split is a later PR)

- `discover list` (sessions newest-first)

- `discover show` / `discover probe` (session detail with per-query probes — competitor domains print with their classified type)

- `discover harvest` (read the answer engine's issued search-query fan-out — Gemini `groundingMetadata.webSearchQueries` — back out of the session's stored probe `raw_response` via the injected `harvestSearchQueries` seam, gate it through `gateHarvestedSearchQueries` + `applyHarvestSemanticNovelty` (an embedding cosine pass over the tracked queries via the injected `embedQueries` Gemini seam that drops paraphrase/synonym dups; degrades to exact-match when no key), and surface the survivors as candidate seeds ranked by probe recurrence; `--min-probe-hits` raises the floor, `--no-anchor` disables the subject filter; read-only, nothing is probed/tracked/promoted — issue #713)

- `discover promote preview` (read-only preview of bucketed queries + recurring suggested competitor domains of every classified type)

- `discover promote` (adopt a completed session's cited + aspirational queries plus `direct-competitor` domains into the project by default — add-only, idempotent, with `--bucket` / `--competitor-types` / `--no-competitors` scoping)

`src/cli-commands/discover.ts` holds the CLI specs for `discover run / seed / probe / list / show / promote preview / promote`. `discover run` / `discover seed` accept `--locations` (comma-separated or repeated) to override the project's location set for seeding, and `--probe-concurrency <1-8>` (default 1 = serial) to bound the probe worker pool.

### Discovery eval panel

`canonry discover eval` is the discovery quality-regression panel: runs 5 fictional ICP shapes as REAL sessions against the configured instance (~$0.10-0.30/shape), scores each (canonicals, retention, brand share, grounding share, duration), compares against a committed baseline with tolerance bands, exit 1 on regression. Run before adopting a new engine build or after seed-prompt/provider/threshold changes; --update-baseline captures a new baseline (commit it with the motivating change).

`discover run --probe-concurrency` sets the parallel probe workers (default 1 = serial, cap 8); probe rows persist in canonical order regardless.

`src/discovery-eval.ts` + `src/commands/discover-eval.ts` implement `canonry discover eval` — the live half of the discovery quality-regression harness (the replay suite in api-routes pins the deterministic pipeline):

- Pure panel/scorecard/band logic in `discovery-eval.ts` (DISCOVERY_EVAL_PANEL: 5 fictional ICP shapes; scoreSession reads the honest pre-truncation `canonicalCount`, falling back to truncated seedCount only on legacy engines and flagging it; compareToBaseline with generous single-draw tolerance bands + the absolute canonical floor of 8).

- The command runs shapes sequentially against the configured instance via the injectable DiscoverEvalClient seam, writes/compares `canonry-discovery-eval-baseline.json`, exits 1 on regression.

- NO MCP tool by design: it is a long-running, provider-spending, file-writing release gate composed of existing API ops (project PUT + discover run + session reads), not a new API capability.

### Research

`src/commands/research.ts` — research commands:

- `research run <project> <query...>` (one saved direct free-form batch, optional shared API provider/model/location, `--wait`, never modifies tracked queries)

- `research batch <project> <json-file|->` (reviewed, explicit destinations with one root idempotency key)

- `research list`, and `research show` (per-query answer, sources, and independent mention/citation evidence)

Direct and reviewed requests submit final query text: a scope only records a market/Property destination; it never rewrites a query or fans out a group.

### Google Analytics commands

`src/commands/ga.ts` — GA4 commands:

- `ga sync`, `ga traffic`, `ga status`

- `ga ai-referral-daily` (AI sessions per day and per source, landing pages summed inside ONE attribution dimension so the totals match `ga traffic`; sessions only, since GA counts users distinct per grain and no un-dimensioned AI-referral fetch exists)

- `ga ai-referral-history` (raw per-landing-page detail rows, never a total)

- `ga social-referral-history`, `ga social-referral-summary`, `ga attribution`

Date windows and headers:

- Every date-scoped read (`ga traffic`, `ga session-history`, `ga ai-referral-daily`, `ga ai-referral-history`, `ga social-referral-history`) takes `--window` OR the explicit `--start` / `--end` calendar bounds (`YYYY-MM-DD`, inclusive). `--window` is rolling from now and cannot name a calendar month, so a monthly client total had to be hand-filtered out of a wider pull before. Explicit dates win over `--window`, and both are forwarded verbatim so the server (`resolveDateRange` in contracts) is the single place that decides. An unrecognised `--window` is rejected, not silently widened to the full history.

- `ga attribution` and `ga social-referral-summary` print `windowLine` — the measured window and its day count — ABOVE the numbers rather than as a trailing `Period:` footer, and carry `windowStart` / `windowEnd` / `windowDays` in `--format json`. Both render percentages that are only meaningful against that window, and a reader who meets "Direct 69%" before the period has already formed a view of the business by the time a footer tells them what it covers.

### Google Business Profile

`src/commands/gbp.ts` — Google Business Profile commands:

- `gbp connect` / `disconnect`

- `gbp accounts` (list the accounts the OAuth user can access — pick one per project)

- `gbp locations` (list / discover / select / deselect; `discover --account <accounts/{n}>` targets a specific account, `--switch-account` opts into the destructive re-point)

- Phase 2 performance + local signals:

- `gbp sync` (fires the `gbp-sync` run; `--wait` polls to terminal)

- `gbp metrics` (stored daily metrics, totals-by-metric)

- `gbp keywords` (stored search-keyword impressions over the synced `periodStart`..`periodEnd` window with `<N` threshold rendering + thresholded %)

- `gbp place-actions` (booking / reservation / order CTAs per location with the preferred-CTA flag)

- `gbp lodging` (latest Lodging API snapshot per hotel location — readable-group count + sync time; 0 groups is a Hotel details verify signal, not proof amenities are absent)

- `gbp places` (latest Places-API rendered-listing snapshot per location with the server-derived `amenities` list — the GBP-vs-public-listing cross-reference, #648)

- `gbp summary` (composite scorecard over the project's SELECTED locations: performance totals + recent-vs-prior 7d deltas computed over complete days only with a `data through <date> · Nd pending` freshness line, keyword coverage, place-action CTA presence, Lodging API readable-group counts — human-readable metric labels via `formatGbpMetricLabel`, no raw `BUSINESS_*` keys; `--format json` also carries the daily `timeseries` + `freshness` the dashboard charts consume (#658). All numbers come from `GET /gbp/summary`, the command only renders)

- Reviews are NOT here — the v4 reviews API is separately access-gated by Google; Q&A was retired.

`src/gbp-sync.ts`:

- `executeGbpSync` — per selected location (bounded-concurrency 4) pulls daily metrics + search-keyword impressions + place-action links + the lodging resource.

- Metrics / keywords / place-actions are range-replaced for the location in one transaction; lodging is **snapshot-on-change** (a new `gbp_lodging_snapshots` row is inserted only when the content hash differs from the latest stored snapshot, so unchanged hotels don't accrue duplicate rows).

- It also fetches the last `KEYWORD_TREND_MONTHS` (3) **complete** months of per-month keyword impressions (one call per month, since the API aggregates a range into a single figure) and **accumulates** them into `gbp_keyword_monthly` (upsert the fetched months, preserve older in-retention months, prune beyond 18 months) — this is the month-over-month series the `gbp-keyword-drop` insight reads, separate from the range-replaced trailing-window snapshot. `monthMinus` anchors to day 1 before shifting months so 29th–31st syncs don't produce duplicate/skipped months.

- Non-lodging locations (HTTP 400 → null) are skipped.

- For lodging locations that carry a Maps `placeId`, when a Places API key is configured (top-level `places` config) it also fetches Place Details (New) and snapshot-on-changes them into `gbp_place_details` — gated by a refresh-cadence age check (`refreshIntervalDays`, default 7; the latest snapshot's `syncedAt` is re-stamped on every fetch, even when content is unchanged, so the gate throttles re-fetches of stable listings) to control cost, and best-effort (a Places error is logged, never failing the run; #648).

- Captures per-location errors → run status completed / partial / failed. Run completion flows through `runGbpSync` (server.ts) → `RunCoordinator.onRunCompleted` for both manual and scheduled (`gbp-sync` kind) triggers.

### OpenAI ads commands

`src/commands/ads.ts` — OpenAI ads commands:

- connection/status/sync/snapshot reads plus live account metadata, geo ID search, conversion pixels, conversion event settings, and `delivery-diagnostics` (stored snapshot provenance, configuration facts, and historical activity only — never a live OpenAI serving or eligibility verdict)

- `live-delivery` (a bounded, read-only live provider read that returns the provider's current status and metrics unaggregated next to the stored snapshot values and an explicit per-entity delta; at most one per project per minute)

- `operation`, cursor-paged unresolved-operation listing, provider-state reconciliation, activation-receipt recovery, image upload, and campaign/ad-group/ad create/update/pause commands using `--input <json-file|->`

Write and receipt rules:

- Campaign creation supports impression or click bidding; click campaigns require click-billed ad groups, and may attach exact conversion event-setting IDs to optimize delivery without being required to.

- Creates are paused, updates require pause + synced `upstreamUpdatedAt`, and new writes stop behind `pending`, `unknown`, or `reconciling` receipts.

- Generic reconciliation verifies only checkpointed provider IDs on the receipt-bound account and never retries the original mutation; fresh pending receipts wait for the minimum-idle gate, inconclusive reads back off, and the fifth failed inspection quarantines the receipt. `campaign_tree_activate` receipts instead use the bodyless exact-executor resume surface.

- Approval-grant create/revoke commands remain human CLI/API surfaces; grant-bound activation and recovery are also available to the ads MCP/Aero operator.

Money renders via `formatMicros`. Registered via `src/cli-commands/ads.ts`.

`src/ads-config.ts`: helpers for the `openaiAds:` connection block in `~/.canonry/config.yaml` (mirrors `ga4-config.ts`; keyed on projectName).

### Ads sync

`src/ads-sync.ts` is wired to the scheduled `ads-sync` kind and the manual `POST /ads/sync` route via `runAdsSync` in `server.ts`.

Snapshots and rollups:

- `executeAdsSync` — OpenAI ads worker: snapshots campaigns/ad groups/ads and upserts daily paid-performance rollups at campaign + ad-group level, normalizing the insights API's decimal-dollar spend to integer micros at ingest.

- Snapshot refresh: refreshed campaigns replaced, campaigns absent from `listCampaigns` deleted, and a still-listed campaign whose sub-fetch failed keeps its prior row rather than vanishing from the snapshot; its unchanged `syncedAt` is the staleness signal, and `ads_connections.lastSyncedAt` is held back when a cycle refreshes nothing.

- Per-campaign failures degrade the run to `partial`.

Insight date ranges:

- Also exports the two insight-range builders: `trailingAdsInsightHourRange` (the sync's own 90-day trailing window) and `liveAdsInsightHourRange`, which the live-delivery reader in `server.ts` uses. BOTH builders derive `until` from the shared `startOfAccountDay`.

- The live one reads NO clock: `since` is the start of the route-supplied `startDate` in the account's own wall clock, resolved through `startOfDayHourInTimeZone` (so the window's first day is a whole day upstream, matching the whole-day rollup it is diffed against, and a zone that springs forward AT midnight is asked for hour 01 rather than the hour 00 it never has), and `until` is the START of the account-local day the route's frozen read anchor falls in (so every insight call in one walk covers the identical range, and the reported `fetchedAt` describes all of them).

- That upper edge is load-bearing, and it is capped by a captured provider rule: an `until` in the future is refused outright with `400: time_ranges.end cannot be in the future`, which fails the whole call rather than clamping it, so the next local midnight is not a usable edge while the day is open.

- Nothing is lost by staying in the past, because the provider emits a daily bucket only when the range fully covers that bucket's own boundaries and the open day's bucket ends at the next local midnight: no accepted edge returns the day in progress, and a range scoped to the open day alone returns zero rows.

- So the ranged call is for CLOSED days (and is the only shape that carries conversions), and the day in progress comes from a SECOND, unranged call, which is the only shape that returns it and the one shape that may not request conversions.

- `readInsightDays` (also exported, and used by the live-delivery reader) makes both calls, caps the unranged one at ONE page since it cannot bound its own window, and keeps only the anchor-date row from it.

- `since` is untouched, so the 90-day backfill window is unchanged.

- Because today's row is partial and is re-read on every sync, the rollup upsert on (project, level, entity, date) is what keeps it correct, and the open day's `conversions` is deliberately NOT written (a placeholder 0 would turn a number the read could not obtain into a wrong one); the first sync after the day closes fills it from the ranged call.

### Server-side traffic

`src/commands/traffic.ts` — server-side traffic commands: pull-source connects for Cloud Run / WordPress / Vercel plus local-only `traffic connect cloudflare` for direct push or Queue pull.

- Cloudflare writes secret-free `worker.js` / `wrangler.toml` artifacts and safely updates recognizable generated files.

- Queue tokens are accepted only through `--api-token-file` and remain in local config; direct deployment installs bearer/HMAC through a mode-0600 temporary file, while Queue deployment uses only a producer binding and requires a separate `wrangler queues consumer http add <queue>` step.

- Both require `--zone-id --confirm-route --confirm-fail-open`, deploy no route, and require explicit `traffic activate` cutover when staged.

- Other commands: `traffic sync`, async `backfill`, explicit `reset --advance-to-now`, `sources`, `status`, and windowed `events`.

`src/cloud-run-config.ts` / `src/wordpress-traffic-config.ts` / `src/vercel-traffic-config.ts` / `src/cloudflare-traffic-config.ts` — helpers for provider connection blocks in `~/.canonry/config.yaml`:

- Cloudflare direct bearer/HMAC and Queue API tokens stay only in local config; generated source/TOML, stdout, API descriptions, and transcripts must never contain them.

- Entries are keyed by source ID so a paused staged mode can coexist with the active source.

### Backlinks

`src/commands/backlinks.ts` — Common Crawl backlink commands: `backlinks install`, `doctor`, `status`, `sync`, `list`, `sources`, `extract`, `releases`, `cache prune`. Historical source filters remain readable, but no non-Common-Crawl ingestion runs.

- `src/commoncrawl-sync.ts`: `executeReleaseSync` — workspace-level Common Crawl release download + DuckDB query job

- `src/backlink-extract.ts`: `executeBacklinkExtract` — per-project backlink extraction run

## Common Mistakes

- Resolve managed run kinds at boot: new environment list, legacy environment

boolean, new YAML list, legacy YAML boolean. Validate `managedRunKinds` against

`schedulableRunKindSchema`; an unknown kind names the key and refuses boot.

The legacy boolean maps to answer-visibility only. Inject `managedRunKinds`

only when non-empty so an unset deployment's client config stays identical.

- `CANONRY_RESEARCH_ALLOW_VIEWERS` overrides `research.allowViewers` and

defaults to false. Inject the client research block only when true, including

the resolved positive `viewerDailyRunLimit` (default 20). This grant applies

to viewer sessions and is the role ceiling for OAuth `research.run` consent;

it does not widen existing read-only grants, sweeps, or settings. Explicit

`research.run` API keys work independently of viewer opt-in. All limited

research callers share the per-project UTC daily cap. MCP preserves the

originating account through an internal delegated key and rechecks current

role authority on each request. CLI and MCP return the same research receipts,

safe provider catalog, and `access` metadata as REST; explicit read-only

endpoints/flags never expose the research-start tool.

- `competitor landscape --by-model` reads stored requested-model groups.

Keep served identity separate. A model filter requires a provider.

JSONL preserves the complete response as one compact document.

- Large Aero competitor results can shorten text with explicit omission counts.

Preserve the complete structured response in the tool result's `details`.

- **Instantiating `ApiClient` directly** — use `createApiClient()` which handles basePath and config.

- **Casting API responses** — use typed DTOs from contracts, not `as { ... }`.

- **Forgetting `--format json` support** — every output command needs it.

- **Forgetting to register command in `cli-commands.ts`** — the command won't be accessible.

## Agent layer (Aero)

Aero's rules (native loop, tool surface, remote MCP, prompt seams, result truncation, disabling Aero, webhooks) live in `src/agent/AGENTS.md`.

## Telemetry events

Anonymous fire-and-forget telemetry, opt-out via `canonry telemetry disable`,

`CANONRY_TELEMETRY_DISABLED=1`, `DO_NOT_TRACK=1`, or any truthy `CI` env var.

All events POST to `https://canonry.ai/api/telemetry` with the following

top-level envelope:

```jsonc

{

"eventId": "uuid-v4", // unique per emission; receiver deduplication key

"anonymousId": "uuid-v4", // stable per-install (~/.canonry/config.yaml)

"sessionId": "uuid-v4", // per-process — same for every event in one CLI invocation / serve boot

"source": "cli", // surface that emitted; see TelemetrySource below

"sourceContext": "...", // optional sub-source ("php/8.2 wp-cron")

"event": "cli.command",

"timestamp": "...",

"version": "4.15.0",

"nodeVersion": "...",

"os": "darwin",

"arch": "arm64",

"errorCode": "NO_PROVIDERS", // present only when the event represents a failure

"properties": { ... }

}

```

### Source taxonomy (`TelemetrySource`)

| Source | When |

|--------|------|

| `cli` | One-shot CLI command (`canonry run`, `canonry status`, …) |

| `cli-server` | Long-running `canonry serve` process (set via `setTelemetrySource('cli-server')` after the server boots, so dashboard/API/scheduler-driven events ride this source) |

| `api` | Reserved — direct API caller (cloud `apps/api`) |

| `mcp-server` | Reserved — `canonry-mcp` stdio adapter (currently forbidden from emitting per `Surface Priority → Agent & automation design principles → MCP adapter boundary`; emission would require a separate adapter) |

| `wp-plugin` | Reserved — WordPress plugin |

| `dashboard` | Setup milestones forwarded by the local API; still obeys the instance telemetry opt-out |

| `agent-runtime` | Reserved — Aero / external agent runtimes |

### Event catalog

| Event | Properties | Notes |

|-------|-----------|-------|

| `cli.command` | `{ command, setup_state?, agent, interactive }` | Fires on every CLI invocation except `telemetry` and `--help`. `setup_state: { provider_count, has_keywords, project_count, is_first_run }` lets the receiver cohort by configured / not-configured. `agent` and `interactive` (`cliRuntimeContext`, `src/runtime-context.ts`) also ride `cli.command.finished`, `cli.init`, and `serve.started`: `agent` is the coding agent from `detectAgentRuntime` in contracts (`claude`, `codex`, `cursor`, `gemini`, ..., or `none`; `CANONRY_AGENT` overrides), `interactive` is true when stdin and stdout are both terminals. An absent `agent` means a client older than the field. |

| `cli.command.finished` | `{ command, success, duration_bucket, setup_state? }` | Terminal command outcome. Command is resolved from the registered command catalog; raw argv and exact duration are never sent. Failures use top-level stable `errorCode`. |

| `cli.init` | `{ providerCount, providers, setup_state?, googleConfigured, agentConfigured, setupState, skillsInstalled }` | Fires after successful `canonry init`; `setup_state` is the post-init snapshot. `setupState` is retained temporarily for report compatibility. |

| `cli.upgraded` | `{ fromVersion, toVersion }` | Fires once when the on-disk `lastSeenVersion` differs from the running build. Suppressed on a fresh install (no prior version recorded). |

| `serve.started` | `{ providerCount, providers }` | Fires after `canonry serve` opens its listener; this is also when `source` flips to `cli-server`. |

| `run.completed` | `{ status, providerCount, providers, queryCount, durationMs, trigger?, domainHash?, phases?, location? }` | `trigger` mirrors `runs.trigger` (`manual` / `scheduled` / `config-apply`). `domainHash` = SHA-256 of the project canonical hostname (no raw domains stored). `phases = { setup_ms, provider_call_ms, total_ms }`. Failures additionally set top-level `errorCode` from `RunErrorCode` (`PROJECT_NOT_FOUND` / `RUN_NOT_FOUND` / `RUN_NOT_EXECUTABLE` / `NO_PROVIDERS` / `QUOTA_EXCEEDED` / `RUN_CANCELLED` / `PROVIDER_ERROR` / `INTERNAL`). |

| `onboarding.started` | `{ flowVersion, onboardingSessionId, step, resumed }` | Dashboard setup entry/resume. Accepted through the strict `/api/v1/telemetry/onboarding` contract; the browser-generated top-level `eventId` is preserved across one delivery retry. |

| `onboarding.step_completed` | `{ flowVersion, onboardingSessionId, step, method, countBucket? }` | Durable setup milestone; raw domains, queries, and keys are forbidden by the contract. |

| `onboarding.blocked` | `{ flowVersion, onboardingSessionId, step, action, reasonCode }` | Stable recovery point and low-cardinality blocker. |

| `run.requested` | `{ flowVersion, onboardingSessionId, origin, result, kind?, providerCountBucket, queryCountBucket, reasonCode? }` | Setup launch attempt, including synchronous rejection. `kind` is `answer_visibility` or `site_health`; absent on older clients means `answer_visibility`. |

| `activation.completed` | `{ flowVersion, kind, status, providerCountBucket?, queryCountBucket?, snapshotCountBucket?, pagesAuditedBucket? }` | Once per project per `kind`. `answer_visibility` (with the provider/query/snapshot buckets): the first non-probe answer-visibility run that persists at least one snapshot. `site_health` (with `pagesAuditedBucket`): the first site audit that scores at least one page. Events without `kind` predate the field and are `answer_visibility`. |

| `site_audit.completed` | `{ status, durationMs, trigger?, domainHash?, complete?, termination?, pagesDiscovered?, pagesFetched?, pagesAudited?, pagesErrored?, aggregateScore?, pageBudget?, checkDeadLinks?, deadLinksFound? }` | Every terminal Technical AEO / Site Health crawl (`executeSiteAudit`), manual or scheduled. Crawl counts are present only for `completed` / `partial`; `aggregateScore` only when a page was audited; `deadLinksFound` only when the run checked dead links. Failures set top-level `errorCode` `UNKNOWN`, cancellations `RUN_CANCELLED`. A crawl interrupted by a server restart is reported at the next boot by `JobRunner.recoverStaleRuns` as `failed` with `errorCode` `SERVER_RESTARTED` (no crawl counts; `durationMs` spans the downtime). Composed by `buildSiteAuditCompletedProps` in `src/run-telemetry.ts`. |

| `traffic.synced` | `{ status, sourceType, sourceId, pulledEvents, selfTrafficExcluded, crawlerHits, aiReferralHits, durationMs }` | Every server-side traffic sync. Failures set top-level `errorCode`. |

| `api.request` | `{ surface, agent, method, route, statusClass, durationBucket, mcpClient?, mcpTool?, mcpCallId?, droppedBefore? }` | Server-side, one per API request from an AGENT SURFACE: `mcp-stdio`, `mcp-http`, `aero`, or a raw `api` caller. CLI requests (already `cli.command`) and dashboard requests (polling) are skipped, as are health, OpenAPI, and telemetry routes. `route` is the template, never a URL. `surface` comes from the `x-canonry-surface` label a first-party `ApiClient` sends (`ApiClientOptions.surface`), else the user agent. `agent` is the client's env-detected agent, falling back to the MCP client name. Clients the server builds for itself (hosted MCP, Aero) send no env-detected agent, because whichever agent launched `canonry serve` is not their caller. Scheduler callbacks (data refresh, traffic sync, doctor, backlinks) use a separate unlabelled `schedulerClient`, so automation is never reported as Aero traffic. `mcpTool` / `mcpCallId` come from `runWithUsageTags`, which the MCP server and Aero wrap around every tool call; group by `mcpCallId` to count tool calls. Capped per process by a token bucket (burst 20, one token per 10s, so at most ~360/hour); suppressed requests are reported as `droppedBefore` on the next event. The cap is sized to the canonry.ai collector's per-IP limit (100/minute, 1,000/hour), past which it drops every event from that IP; do not raise one without the other. Built by `createApiUsageTelemetry` (`src/usage-telemetry.ts`) from the `onRequestCompleted` api-routes hook. The labels are caller-controlled, validated to enums/slugs, and never identity. |

| `mcp.session.started` | `{ surface, agent, mcpClient? }` | Once per MCP connection (keyed by the client's `x-canonry-actor-session` correlation value), on its first tool call, which is the first request that carries the MCP client name. |

| `telemetry.disabled` | `{ method }` | The last event an install sends, so the opt-out rate is measurable. Whether to send it is decided while telemetry is still on; it is sent only after the opt-out is written, so an opt-out that could not be persisted (read-only config) is never announced. `method` is `cli` (`canonry telemetry disable`) or `api` (`PUT /telemetry`: dashboard checkbox, settings, MCP, or `--target server`). Not sent when telemetry is already effectively off, including env overrides. All preference writes go through `setTelemetryPreference` in `src/telemetry.ts`; the CLI awaits delivery so process exit cannot drop it. |

A new event name must also be added to `ALLOWED_EVENTS` in the canonry.ai collector (`lib/telemetry/validation.ts`) before it ships; the collector rejects unknown names with a 400, and `trackEvent` never surfaces that failure. Property values must be string, number, boolean, string array, or a flat object: `null` rejects the whole event.

### Adding a new event

1. Pick a stable `event` name and add it to the catalog above.

2. Call `trackEvent(event, properties, options)` from the originating code path. Pass `options.source` if the default global source is wrong; pass `options.errorCode` for failure events.

3. If the event represents a run/job result, compose the payload through a helper in `src/run-telemetry.ts` (or a sibling) so the shape stays in one place — never inline a new property bag in three call sites.

4. Add tests that assert the new field/event in `packages/canonry/test/telemetry.test.ts` (or a dedicated file like `run-telemetry.test.ts`).

5. Bump the version in both `package.json` files.

## See Also

- `packages/api-routes/` — the route handlers this server mounts

- `packages/contracts/` — DTOs returned by the API client

- `docs/architecture.md` — how CLI, server, and job runner interact

### Saved-result cleanup

`canonry results clear <project> --run <id> --research-run <id>` previews exact saved IDs.

`--confirm` deletes those visibility/research results after the operator backs them up.

The MCP equivalent is `canonry_results_clear`; exclude it from the in-product Aero agent.

Site Health, backlinks, configuration, audit records, and usage accounting remain intact.

### Public sample server

`canonry demo` starts a separate in-memory, synthetic, view-only runtime. It must never use the normal server factory, personal config/database paths, provider callbacks, or background workers. The exact GET/HEAD allowlist in `src/demo/access.ts` is intentional; new route registrations are denied until explicitly audited. Startup has no MCP parity because it controls a host listener. See `docs/public-demo.md`.

Native `agent ask` supports `--context <json>`, `--max-tool-calls`, and

`--timeout-ms`, sharing `AgentPromptRequest` with REST/dashboard. Context is per

turn; it does not widen tool scope. Preserve terminal `aero_turn_status` in JSON

output and fail unexpected SSE EOF. See `src/agent/AGENTS.md` and

`docs/aero-evaluation.md` for runtime boundaries and captured-turn evaluation.