Immutable. This exact content is served forever at /api/v1/blob/a13bead55812f11f.
# canonry
## Purpose
The publishable npm package (`@canonry/canonry`, plus compatibility publish as `@ainyc/canonry`). Bundles the CLI, local Fastify server, in-process job runner, provider registry, scheduler, and pre-built SPA. This is what users install with `npm install -g @canonry/canonry`.
## Key Files
| File | Role |
|------|------|
| `src/cli.ts` | CLI entry point — shebang, telemetry, command dispatch |
| `src/telemetry.ts` | `trackEvent`, source attribution (incl. the validated `CANONRY_TELEMETRY_SOURCE` harness override), per-process `sessionId`, `cli.upgraded` detection |
| `src/setup-nudge.ts` | The stalled-setup stderr line: human-mode + TTY only, lazy state read, exempt roots |
| `src/activation-notice.ts` | Once-ever first-activation notice; marker written before printing, TTY-gated |
| `src/run-telemetry.ts` | `classifyRunError`, `buildRunCompletedProps` — keeps the `run.completed` payload composition in one spot |
| `src/setup-state.ts` | `buildSetupState` — `{ provider_count, has_keywords, project_count, is_first_run }` snapshot ridden on every `cli.command` |
| `src/cli-commands.ts` | `REGISTERED_CLI_COMMANDS` array — declarative command specs |
| `src/commands/` | Command implementations (one file per domain) |
| `src/commands/competitor.ts` | Competitor commands: `competitor add`, `remove`/`delete`, `list` |
| `src/commands/query.ts` | Query commands: `query add`, `replace`, `remove`/`delete`, `list`, `import`, `generate` |
| `src/commands/keys.ts` | API key commands: `key list` (table NAME / PREFIX / SCOPES / REACH / CREATED / LAST USED / STATUS, where REACH is `full instance` or the scoped project's name; `--format json\|jsonl`), `key create --name <name> [--read-only \| --scope <s> ...]` (prints the plaintext key ONCE with a "will not be shown again" warning; JSON mode includes the key; `--read-only` mints `scopes=['read']` and is mutually exclusive with `--scope`), `key revoke <id>` (confirms revocation), `key whoami` (introspect the CURRENT key — name / scopes / readOnly / reach / status). Flag-driven, no prompts; delegates to `ApiClient.listApiKeys` / `createApiKey` / `revokeApiKey` / `getApiKeySelf`. Registered via `src/cli-commands/keys.ts`. |
| `src/commands/mcp.ts` | MCP client install helpers: `mcp install`, `mcp config` (writes to client config files only — separate from the `canonry-mcp` stdio bin) |
| `src/mcp-clients.ts` | Registry of supported MCP clients (Claude Desktop, Cursor, Codex) — config-path resolvers and format hints used by `mcp install`/`mcp config` |
| `src/commands/skills.ts` | `installSkills` / `listSkills` — reconciles bundled `skills/<name>/` trees into a user's `.claude/skills/<name>/` **additively** (missing files copied without `--force`; upstream-updated files the operator never touched refreshed; genuine local edits preserved and reported as conflicts unless `--force`), writes a `.canonry-skill-manifest.json` recording what canonry last wrote, and creates relative `.codex/skills/<name>` symlinks. `getBundledSkillSnapshots()` exposes the bundled version + per-file hashes for the `agent.skills.current` doctor check. Auto-invoked by `canonry init` when cwd looks like a project. |
| `src/cli-commands/skills.ts` | CLI specs for `skills list` / `skills install [skill...] [--dir <path>] [--client claude\|codex\|all] [--force]`. |
| `build-web.ts` | Builds `apps/web` (`pnpm --filter @ainyc/canonry-web build`) then `cp apps/web/dist → assets/` (preserves `agent-workspace/`, verifies hashed refs) — entry via `pnpm --filter @canonry/canonry run build:web` or `pnpm --filter @canonry/canonry run build` (which runs `copy-agent-assets.ts` + `tsup` + `build-web.ts`) |
| `assets/` | Bundled SPA output (`index.html` + `assets/*.js/css`) — **do not hand-edit**; regenerated by `build-web.ts`. `server.ts` default `assetsDir` is this dir; `createServer({assetsDir: '/tmp/my-dist'})` override lets tests/custom builds point elsewhere without forking the package |
| `scripts/copy-agent-assets.ts` | Build-time mirror of repo-root `skills/` into `assets/agent-workspace/skills/` so trees ship in the published package. |
| `src/agent-plugin.ts` | Best-effort, secret-free, user-global Canonry plugin detection for Claude Code JSON settings and Codex TOML settings. It verifies Agent Plugins root manifests for Codex, retains the legacy client-manifest fallback, records each verified version so doctor/nudges can enforce runtime lockstep, and never controls filesystem writes. |
| `src/gsc-sitemap-submission.ts` | `dedupeGscSitemapUrls` + `resolveDiscoveredGscSitemapUrls` (indexes vs all-files, 4× parallel) + `requireGscSitemapUrls` + `submitGscSitemapBatches` (50-url batches) |
| `src/client.ts` | `ApiClient` class + `createApiClient()` factory |
| `src/mcp/` | `canonry-mcp` stdio adapter over `createApiClient()` |
| `src/mcp/server.ts` | `createCanonryMcpServer` — registers all API tools, then disables non-core tiers unless `--eager` |
| `src/mcp/tool-registry.ts` | All 188 API tools, including Site Health semantic graph and page-audit evidence reads, sitemap Target discovery, and revision-pinned measurement reports, each tagged with a `tier` (`core` or one of the toolkit names) |
| `src/mcp/toolkits.ts` | Toolkit catalog (`monitoring`, `setup`, `gsc`, `ga`, `gbp`, `ads`, `traffic`, `agent`, `discovery`) — name, title, description, when-to-load |
| `src/mcp/dynamic-catalog.ts` | `DynamicToolCatalog` — drives `canonry_help` and `canonry_load_toolkit` (enables tools, emits `tools/list_changed`) |
| `src/mcp/cli.ts` | `canonry-mcp` stdio entrypoint — parses `--read-only`, `--eager`, `--scope`, plus `CANONRY_MCP_*` env. `resolveEffectiveScope()` best-effort probes `GET /keys/self` at startup and forces `read-only` when the configured key is read-only (auto-restricts the catalog to read tools; falls back to the flag scope on any probe failure). |
| `src/server.ts` | Fastify server setup — mounts api-routes, serves SPA, registers providers. Read-only embed mode (#716): resolves `resolveEmbedConfig(process.env, config)` at boot; when enabled, `injectConfig` appends an `embed` block to `window.__CANONRY_CONFIG__` and the single `sendSpaDocument` chokepoint (used by `serveIndex` AND the deep-link `setNotFoundHandler` fallback) emits `Content-Security-Policy: frame-ancestors …` (fail-closed to `'none'`). The `embed.projectTabs` value may be overridden PER REQUEST by the `X-Canonry-Embed-Tabs` header (`embedClientConfigForRequest` in contracts): the Embed v2 `/e` proxy sets it per dashboard from `embed_dashboards`, and the end client cannot reach the loopback engine to set it (presentational only; the API key scope is the data boundary). Absent header = the boot-wide config. When embed is off, the injected config + headers are byte-for-byte unchanged. `createServer` opts take an optional `assetsDir` override (default = bundled `assets/`) so integration tests can point at a temp `index.html`. |
| `src/embed.ts` | `resolveEmbedConfig(env, config)` — resolves embed mode (#716) from `CANONRY_EMBED` / `CANONRY_EMBED_ORIGINS` / `CANONRY_EMBED_VIEWS` / `CANONRY_EMBED_PROJECT_TABS` layered over config.yaml `embed:` (env over config, mirroring basePath). Delegates origin normalization + the `frame-ancestors` value + the client block to the pure helpers in `@ainyc/canonry-contracts` (`normalizeFrameOrigin` / `parseOriginList` / `frameAncestorsHeaderValue` / `buildEmbedClientConfig`). `enabled` is decoupled from origins so `--embed` without origins fails closed; an empty views/projectTabs list collapses to `undefined` (= all). `projectTabs` is the project-page TAB allowlist (overview/technical-aeo/search-console/activity/backlinks/...) the embedded dashboard renders — finer than `views`, which only gates whole top-level routes. Serve/start flags: `--embed`, `--embed-allow-origin <origin>…`, `--embed-view <view>…`, `--embed-project-tab <tab>…` (in `cli-commands/system.ts` → env via `applyServerEnv`; `start` forwards them through `buildServeForwardArgs` in `commands/daemon.ts`). |
| `src/agent-config.ts` | `resolveAgentEnabled(env, config)` — the Aero kill-switch. Resolves whether the built-in agent runs from `CANONRY_AGENT_DISABLED` env layered over `agent.mode: 'disabled'` in `config.yaml` (env over config; `=1`/`true` off, `=0`/`false` force on). `server.ts` reads it once at boot and guards the three Aero wiring points: the `SessionRegistry`, the proactive run-completion wake, and the interactive agent routes. Does not touch data syncs / intelligence / notifications. |
| `src/job-runner.ts` | In-process job runner for visibility sweeps |
| `src/provider-registry.ts` | `ProviderRegistry` — manages provider adapters |
| `src/scheduler.ts` | Cron runner for all schedule kinds. `onSiteAuditRequested` starts a Technical AEO full crawl unless one is active. |
| `src/execute-site-audit.ts` | `executeSiteAudit` runs `@canonry/aeo-audit`'s `runSiteCrawl`. Events update an attempt graph with idempotent receipts. Terminal runs keep immutable complete or partial snapshots. Default reads select only the latest complete snapshot. Defaults are 1,000 pages and 100,000 edges. Hard limits are 50,000 pages and 1,000,000 edges. The engine (>= 7.1.0) derives `maxFetches`/`maxDurationMs`/`maxBytes`/`maxEdges` from the page budget natively and honours explicit values exactly, so this file passes ONLY `maxPages`/`maxEdges` (the operator-facing limits) and never a fetch-side budget — setting one would pin it and fight the derivation. The engine also makes the dead-link split itself (6.0.0+): `deadLinks.findings` always carry a real 4xx/5xx status and `deadLinks.unverified` carries the targets the crawl could not check (timeout, reset socket, throttled 429), and ONLY findings are written to `site_crawl_findings` — every reader of that table renders a row as a broken link, and a crawl timeout is not evidence of one. `deadLinkCheckedCount` excludes unfetchable targets for the same reason — a URL that never answered was attempted, not checked. |
| `src/site-crawl-template-links.ts` | `classifySiteCrawlTemplateLinks` marks nav, header, and footer links once per attempt, after the crawl and BEFORE graph layout (which excludes them from the physics). DOM placement (`placement_*_occurrences`, from the crawler's landmark ruleset) decides every link the page answered for: any content occurrence makes the link editorial, navigation with no content occurrence makes it chrome. Ubiquity is the FALLBACK for links the page said nothing about and for pre-4.7.0 scans that recorded no placement; it is a whole-crawl fact, so it cannot be decided per event. Links stream in bounded batches; the derivation is the contract's `classifyTemplateLinkEdge`, shared with the tests. The result reports which rules ran (`applied-placement`, `applied-placement-with-ubiquity`, `applied-placement-partial`, or `applied`), which the snapshot records beside `link_placement_ruleset_version`. With no placement and fewer than 15 fetched pages it marks nothing and returns `unavailable-too-few-pages`. It writes a STRICT BOOLEAN `is_template` in every case: a link no rule could measure is a real `false` graded `unmeasured`, never a NULL, so the layout input, the graph sample, both link filters, the totals, the map legend, and the inspector tiles all keep ONE definition of a content link. NULL is reserved for rows that predate classification. The whole-scan state comes from a shared tally (`observeTemplateLinkDetection`) that both this pass and the one-shot classifier fold into, and `applied-placement-partial` is scoped to ANCHOR edges so a redirect cannot report that a well-marked-up site is missing landmarks. |
| `src/commands/measurement-plan.ts` | Advanced Measurement plan commands: `measurement-plan show` / `versions` / `publish` / `retire` / `discover` / `report`, plus the two per-Property reads — `measurement-plan property <project> --target-key <key>` (one Property out of `GET /measurement-overview?scope=property`, rendering mention/citation coverage and the per-answer-engine split) and `measurement-plan property-evidence <project> --target-key <key>` (cursor-paged `GET /measurement-property-evidence`; `--shape answers` switches a row from one cited URL to one measured ANSWER with its cited URLs nested and both signals on the row — the only shape that shows the answers a Property was not cited in; `--format jsonl` streams one row per line in either shape, under a header line carrying the shape, the measurement state and the cursor, and an unknown mention prints `not measured` rather than `no`). `measurement-plan advanced <project> <operation> [<json|->]` is the compact typed-JSON bridge for the remaining v2 reads/writes; its paged/list reads stream a metadata header and records with `--format jsonl`. The per-Property reads share `--query-class` / `--provider` / `--location` / `--run-id`, and both render an unavailable metric as `not measured (<reason>)` — never a percentage, so an unmeasured class can't read as a measured zero. `--format json` is the endpoint response verbatim. Registered via `src/cli-commands/measurement-plan.ts`. |
| `src/commands/technical-aeo.ts` | Technical AEO commands for runs, scorecards, crawl pages, path structure, internal links, page neighbors, dead links, and trends. Dead-link analysis requires `--check-dead-links`. The dead-link renderers print `unverified` as "could not be checked" and never fold it into `found`: those are different claims and only the first may reach a client. |
| `src/data-refresh.ts` | `refreshAllIntegrations` — fires GSC + Bing + GA + GBP + ads syncs for a project via the in-process API client, `Promise.allSettled` for per-integration isolation. Wired to the scheduler's `data-refresh` kind in `server.ts`. |
| `src/snapshot-service.ts` | Snapshot creation and diff logic |
| `src/intelligence-service.ts` | Runs analysis after sweeps, persists insights + health snapshots |
| `src/run-coordinator.ts` | Post-run orchestrator — dispatches to intelligence + notifications |
| `src/commands/insights.ts` | `insights` and `insights dismiss` command implementations |
| `src/commands/health-cmd.ts` | `health` command implementation |
| `src/commands/visibility-stats.ts` | `visibility-stats <project>` — aggregated per-query mention/citation rates with sample size (`--since`/`--until`/`--month <YYYY-MM>`/`--last-runs`/`--by-provider`/`--share-of-voice`); `--share-of-voice` adds pooled project-vs-competitor share of voice to the envelope; collection command → jsonl streams one record per query. Also `visibility-compare <project> --from <YYYY-MM> --to <YYYY-MM>` (`showVisibilityCompare`) — month-over-month AEO comparison rendered as the statistician-panel table (share-of-voice-led with `*` drift-robust marker, each cell `point [ci-lo, ci-hi]`, within-noise/moved verdict, basket + low-sweep caveat + model-change note); object command, `--format json` (jsonl degrades to json). Delegates to `ApiClient.getVisibilityStats` / `getVisibilityCompare`. Registered via `src/cli-commands/visibility-stats.ts`. |
| `src/commands/doctor.ts` | `canonry doctor` — runs the doctor check registry via `ApiClient.runDoctor` |
| `src/cli-commands/doctor.ts` | CLI spec for `canonry doctor [--project <name>] [--check <id>...]` |
| `src/commands/backfill.ts` | Historical recomputation for answer visibility fields and insights |
| `src/commands/report.ts` | `runReportCommand` — `canonry report <project>` — fetches `/report` JSON, renders self-contained HTML to disk via `renderReportHtml` from `@ainyc/canonry-api-routes` |
| `src/commands/results-export.ts` | `exportResults` — `canonry results export <project>` — downloads the historical query × provider observations attachment via `ApiClient.downloadResultsExport` (`--format json|csv` selects the artifact type; `--since`/`--until` inclusive run-creation window; `--include-probes` opt-in; `--output <path>` or `-` for stdout, default = server-suggested filename via `path.basename`, path-traversal safe). Distinct from `commands/export-cmd.ts` (project CONFIG export). Registered via `src/cli-commands/operator.ts`; API op is MCP `excluded-protocol` (bulk attachment). |
| `src/cli-commands/report.ts` | CLI spec for `canonry report <project> [--output <path>] [--format json]` |
| `src/commands/ga.ts` | GA4 commands: `ga sync`, `ga traffic`, `ga status`, `ga ai-referral-daily` (AI sessions per day and per source, landing pages summed inside ONE attribution dimension so the totals match `ga traffic`; sessions only, since GA counts users distinct per grain and no un-dimensioned AI-referral fetch exists), `ga ai-referral-history` (raw per-landing-page detail rows, never a total), `ga social-referral-history`, `ga social-referral-summary`, `ga attribution`. Every date-scoped read (`ga traffic`, `ga session-history`, `ga ai-referral-daily`, `ga ai-referral-history`, `ga social-referral-history`) takes `--window` OR the explicit `--start` / `--end` calendar bounds (`YYYY-MM-DD`, inclusive). `--window` is rolling from now and cannot name a calendar month, so a monthly client total had to be hand-filtered out of a wider pull before. Explicit dates win over `--window`, and both are forwarded verbatim so the server (`resolveDateRange` in contracts) is the single place that decides. An unrecognised `--window` is rejected, not silently widened to the full history. `ga attribution` and `ga social-referral-summary` print `windowLine` — the measured window and its day count — ABOVE the numbers rather than as a trailing `Period:` footer, and carry `windowStart` / `windowEnd` / `windowDays` in `--format json`. Both render percentages that are only meaningful against that window, and a reader who meets "Direct 69%" before the period has already formed a view of the business by the time a footer tells them what it covers. |
| `src/commands/gbp.ts` | Google Business Profile commands: `gbp connect` / `disconnect`, `gbp accounts` (list the accounts the OAuth user can access — pick one per project), `gbp locations` (list / discover / select / deselect; `discover --account <accounts/{n}>` targets a specific account, `--switch-account` opts into the destructive re-point), and Phase 2 performance + local signals: `gbp sync` (fires the `gbp-sync` run; `--wait` polls to terminal), `gbp metrics` (stored daily metrics, totals-by-metric), `gbp keywords` (stored search-keyword impressions over the synced `periodStart`..`periodEnd` window with `<N` threshold rendering + thresholded %), `gbp place-actions` (booking / reservation / order CTAs per location with the preferred-CTA flag), `gbp lodging` (latest Lodging API snapshot per hotel location — readable-group count + sync time; 0 groups is a Hotel details verify signal, not proof amenities are absent), `gbp places` (latest Places-API rendered-listing snapshot per location with the server-derived `amenities` list — the GBP-vs-public-listing cross-reference, #648), `gbp summary` (composite scorecard over the project's SELECTED locations: performance totals + recent-vs-prior 7d deltas computed over complete days only with a `data through <date> · Nd pending` freshness line, keyword coverage, place-action CTA presence, Lodging API readable-group counts — human-readable metric labels via `formatGbpMetricLabel`, no raw `BUSINESS_*` keys; `--format json` also carries the daily `timeseries` + `freshness` the dashboard charts consume (#658). All numbers come from `GET /gbp/summary`, the command only renders). Reviews are NOT here — the v4 reviews API is separately access-gated by Google; Q&A was retired. |
| `src/ads-sync.ts` | `executeAdsSync` — OpenAI ads worker: snapshots campaigns/ad groups/ads (range-replaced per project) and upserts daily paid-performance rollups at campaign + ad-group level, normalizing the insights API's decimal-dollar spend to integer micros at ingest. Per-campaign failures degrade the run to `partial`. Wired to the scheduled `ads-sync` kind and the manual `POST /ads/sync` route via `runAdsSync` in `server.ts`. Also exports the two insight-range builders: `trailingAdsInsightHourRange` (the sync's own 90-day trailing window) and `liveAdsInsightHourRange`, which the live-delivery reader in `server.ts` uses. The live one reads NO clock: `since` is the start of the route-supplied `startDate` in the account's own wall clock, resolved through `startOfDayHourInTimeZone` (so the window's first day is a whole day upstream, matching the whole-day rollup it is diffed against, and a zone that springs forward AT midnight is asked for hour 01 rather than the hour 00 it never has), and `until` is the START of the account-local day the route's frozen read anchor falls in (so every insight call in one walk covers the identical range, and the reported `fetchedAt` describes all of them). BOTH builders derive `until` from the shared `startOfAccountDay`. That upper edge is load-bearing, and it is capped by a captured provider rule: an `until` in the future is refused outright with `400: time_ranges.end cannot be in the future`, which fails the whole call rather than clamping it, so the next local midnight is not a usable edge while the day is open. Nothing is lost by staying in the past, because the provider emits a daily bucket only when the range fully covers that bucket's own boundaries and the open day's bucket ends at the next local midnight: no accepted edge returns the day in progress, and a range scoped to the open day alone returns zero rows. So the ranged call is for CLOSED days (and is the only shape that carries conversions), and the day in progress comes from a SECOND, unranged call, which is the only shape that returns it and the one shape that may not request conversions. `readInsightDays` (also exported, and used by the live-delivery reader) makes both calls, caps the unranged one at ONE page since it cannot bound its own window, and keeps only the anchor-date row from it. `since` is untouched, so the 90-day backfill window is unchanged. Because today's row is partial and is re-read on every sync, the rollup upsert on (project, level, entity, date) is what keeps it correct, and the open day's `conversions` is deliberately NOT written (a placeholder 0 would turn a number the read could not obtain into a wrong one); the first sync after the day closes fills it from the ranged call. |
| `src/commands/ads.ts` | OpenAI ads commands: connection/status/sync/snapshot reads plus live account metadata, geo ID search, conversion pixels, conversion event settings, and `delivery-diagnostics` (stored snapshot provenance, configuration facts, and historical activity only — never a live OpenAI serving or eligibility verdict) plus `live-delivery` (a bounded, read-only live provider read that returns the provider's current status and metrics unaggregated next to the stored snapshot values and an explicit per-entity delta; at most one per project per minute); also `operation`, cursor-paged unresolved-operation listing, provider-state reconciliation, activation-receipt recovery, image upload, and campaign/ad-group/ad create/update/pause commands using `--input <json-file|->`. Campaign creation supports impression or click bidding; click campaigns require click-billed ad groups, and may attach exact conversion event-setting IDs to optimize delivery without being required to. Creates are paused, updates require pause + synced `upstreamUpdatedAt`, and new writes stop behind `pending`, `unknown`, or `reconciling` receipts. Generic reconciliation verifies only checkpointed provider IDs on the receipt-bound account and never retries the original mutation; fresh pending receipts wait for the minimum-idle gate, inconclusive reads back off, and the fifth failed inspection quarantines the receipt. `campaign_tree_activate` receipts instead use the bodyless exact-executor resume surface. Approval-grant create/revoke commands remain human CLI/API surfaces; grant-bound activation and recovery are also available to the ads MCP/Aero operator. Money renders via `formatMicros`. Registered via `src/cli-commands/ads.ts`. |
| `src/ads-config.ts` | Helpers for the `openaiAds:` connection block in `~/.canonry/config.yaml` (mirrors `ga4-config.ts`; keyed on projectName). |
| `src/gbp-sync.ts` | `executeGbpSync` — per selected location (bounded-concurrency 4) pulls daily metrics + search-keyword impressions + place-action links + the lodging resource. Metrics / keywords / place-actions are range-replaced for the location in one transaction; lodging is **snapshot-on-change** (a new `gbp_lodging_snapshots` row is inserted only when the content hash differs from the latest stored snapshot, so unchanged hotels don't accrue duplicate rows). It also fetches the last `KEYWORD_TREND_MONTHS` (3) **complete** months of per-month keyword impressions (one call per month, since the API aggregates a range into a single figure) and **accumulates** them into `gbp_keyword_monthly` (upsert the fetched months, preserve older in-retention months, prune beyond 18 months) — this is the month-over-month series the `gbp-keyword-drop` insight reads, separate from the range-replaced trailing-window snapshot. `monthMinus` anchors to day 1 before shifting months so 29th–31st syncs don't produce duplicate/skipped months. Non-lodging locations (HTTP 400 → null) are skipped. For lodging locations that carry a Maps `placeId`, when a Places API key is configured (top-level `places` config) it also fetches Place Details (New) and snapshot-on-changes them into `gbp_place_details` — gated by a refresh-cadence age check (`refreshIntervalDays`, default 7; the latest snapshot's `syncedAt` is re-stamped on every fetch, even when content is unchanged, so the gate throttles re-fetches of stable listings) to control cost, and best-effort (a Places error is logged, never failing the run; #648). Captures per-location errors → run status completed / partial / failed. Run completion flows through `runGbpSync` (server.ts) → `RunCoordinator.onRunCompleted` for both manual and scheduled (`gbp-sync` kind) triggers. |
| `src/commands/traffic.ts` | Server-side traffic commands: pull-source connects for Cloud Run / WordPress / Vercel plus local-only `traffic connect cloudflare` for direct push or Queue pull. Cloudflare writes secret-free `worker.js` / `wrangler.toml` artifacts and safely updates recognizable generated files. Queue tokens are accepted only through `--api-token-file` and remain in local config; direct deployment installs bearer/HMAC through a mode-0600 temporary file, while Queue deployment uses only a producer binding and requires a separate `wrangler queues consumer http add <queue>` step. Both require `--zone-id --confirm-route --confirm-fail-open`, deploy no route, and require explicit `traffic activate` cutover when staged. Other commands: `traffic sync`, async `backfill`, explicit `reset --advance-to-now`, `sources`, `status`, and windowed `events`. |
| `src/cloud-run-config.ts` / `src/wordpress-traffic-config.ts` / `src/vercel-traffic-config.ts` / `src/cloudflare-traffic-config.ts` | Helpers for provider connection blocks in `~/.canonry/config.yaml`. Cloudflare direct bearer/HMAC and Queue API tokens stay only in local config; generated source/TOML, stdout, API descriptions, and transcripts must never contain them. Entries are keyed by source ID so a paused staged mode can coexist with the active source. |
| `src/commands/backlinks.ts` | Common Crawl backlink commands: `backlinks install`, `doctor`, `status`, `sync`, `list`, `sources`, `extract`, `releases`, `cache prune`. Historical source filters remain readable, but no non-Common-Crawl ingestion runs. |
| `src/commoncrawl-sync.ts` | `executeReleaseSync` — workspace-level Common Crawl release download + DuckDB query job |
| `src/backlink-extract.ts` | `executeBacklinkExtract` — per-project backlink extraction run |
| `src/discovery-run.ts` | `executeDiscoveryRun` — fires `executeDiscovery` (api-routes orchestrator) with Gemini-backed seed/embed/probe/classifyDomains deps, writes the `discovery.basket-divergence` insight, and hands off to `RunCoordinator.onRunCompleted` so Aero wakes up with a bucket-count payload. Forwards the run's resolved `locations` into `executeDiscovery` → `deps.seed`. `classifyDomains` is one plain-text `generateText` call per session; `buildClassificationPrompt` / `parseClassificationResponse` are exported pure helpers that build the `domain => category` prompt and forgivingly parse the model's reply into a `DiscoveryDomainClassification`. `buildSeedPrompt` / `buildLocationConstraint` are exported pure helpers that build the Gemini seed prompt — when locations are present they geo-constrain the prompt and (for 2+ locations) add a per-area seed quota of `floor(DEFAULT_SEED_COUNT / locationCount)`. The probe dep computes BOTH signals from the normalized result: `citationState` from cited domains AND `answerMentioned` from the answer text via the shared `determineAnswerMentioned` (brand names from `effectiveBrandNames`, domains from `canonicalDomains`) — same helper the answer-visibility writer uses, so discovery and sweeps agree on "mentioned". The embed dep wraps `embedQueries` in `embedWithRetry` (exported, testable) — shared `withRetry` + `isRetryableHttpError` with a per-attempt wall-clock timeout, so a transient blip between the two paid grounded phases no longer fails the session while a permanent 4xx still does. `buildDefaultDeps` is exported for unit-testing `probe()` offline. |
| `src/commands/discover.ts` | Discovery commands: `discover run` (kick off ICP → seed → embed → probe pipeline; `--locations` geo-constrains seed generation to a project-location subset), `discover seed` (alias for run today; the phase split is a later PR), `discover list` (sessions newest-first), `discover show` / `discover probe` (session detail with per-query probes — competitor domains print with their classified type), `discover harvest` (read the answer engine's issued search-query fan-out — Gemini `groundingMetadata.webSearchQueries` — back out of the session's stored probe `raw_response` via the injected `harvestSearchQueries` seam, gate it through `gateHarvestedSearchQueries` + `applyHarvestSemanticNovelty` (an embedding cosine pass over the tracked queries via the injected `embedQueries` Gemini seam that drops paraphrase/synonym dups; degrades to exact-match when no key), and surface the survivors as candidate seeds ranked by probe recurrence; `--min-probe-hits` raises the floor, `--no-anchor` disables the subject filter; read-only, nothing is probed/tracked/promoted — issue #713), `discover promote preview` (read-only preview of bucketed queries + recurring suggested competitor domains of every classified type), `discover promote` (adopt a completed session's cited + aspirational queries plus `direct-competitor` domains into the project by default — add-only, idempotent, with `--bucket` / `--competitor-types` / `--no-competitors` scoping). |
| `src/commands/research.ts` | Research commands: `research run <project> <query...>` (one saved free-form batch, optional shared API provider/model/location, `--wait`, never modifies tracked queries), `research list`, and `research show` (per-query answer, sources, and independent mention/citation evidence). |
| `src/discovery-eval.ts` + `src/commands/discover-eval.ts` | `canonry discover eval` — the live half of the discovery quality-regression harness (the replay suite in api-routes pins the deterministic pipeline). Pure panel/scorecard/band logic in `discovery-eval.ts` (DISCOVERY_EVAL_PANEL: 5 fictional ICP shapes; scoreSession reads the honest pre-truncation `canonicalCount`, falling back to truncated seedCount only on legacy engines and flagging it; compareToBaseline with generous single-draw tolerance bands + the absolute canonical floor of 8). The command runs shapes sequentially against the configured instance via the injectable DiscoverEvalClient seam, writes/compares `canonry-discovery-eval-baseline.json`, exits 1 on regression. NO MCP tool by design: it is a long-running, provider-spending, file-writing release gate composed of existing API ops (project PUT + discover run + session reads), not a new API capability. |
| `src/cli-commands/discover.ts` | CLI specs for `discover run / seed / probe / list / show / promote preview / promote`. `discover run` / `discover seed` accept `--locations` (comma-separated or repeated) to override the project's location set for seeding, and `--probe-concurrency <1-8>` (default 1 = serial) to bound the probe worker pool. |
| `src/agent-webhook.ts` | `AGENT_WEBHOOK_EVENTS` — event list subscribed to by `canonry agent attach` |
| `src/commands/agent.ts` | `agentAttach` / `agentDetach` — wire an external agent's webhook to a project |
| `src/commands/agent-ask.ts` | `agentAsk` — one-shot turn against the built-in Aero agent, streams events to stdout |
| `src/cli-commands/agent.ts` | CLI specs for `agent ask / attach / detach` |
| `src/agent/session.ts` | `createAeroSession` — constructs a pi-agent-core Agent scoped to a canonry project (composes `soul.md` + `SKILL.md` into the system prompt, wires model, tools, API-key resolver) |
| `src/agent/session-registry.ts` | Hybrid session registry — in-memory `Map<project, Agent>` + durable `agent_sessions` row per project. Handles hydration, persistence, follow-up queueing, post-`agent_end` auto-drain, and the `<memory>` hydrate block appended to every new session's system prompt. `acquireForTurn` is async and awaits transcript compaction before returning. |
| `src/agent/memory-store.ts` | CRUD helpers for `agent_memory`: `listMemoryEntries`, `upsertMemoryEntry`, `deleteMemoryEntry`, `loadRecentForHydrate`, `writeCompactionNote`. Enforces the 2 KB value cap and the `compaction:` reserved-prefix rule. |
| `src/agent/compaction.ts` | Transcript compaction — `shouldCompact`, `findSafeSplit` (snaps to user-message boundaries), `runSummaryLlm` (one-shot pi-ai `complete()` call), and `compactMessages` which persists the summary as a `compaction:` memory row and returns the kept suffix. |
| `src/agent/compaction-config.ts` | Tuning constants for compaction — token threshold, target ratio, preserved-tail size, max-messages hard cap. |
| `src/agent/token-counter.ts` | `estimateMessageTokens` / `estimateTranscriptTokens` — chars/4 heuristic handling user/assistant/toolResult content shapes. Used only to decide when to compact, not to enforce provider limits. |
| `src/agent/tools.ts` | Thin wrapper around `mcp-to-agent-tool.ts` — `buildReadTools(ctx)` and `buildAllTools(ctx)` delegate to `buildMcpAgentTools(canonryMcpTools, ctx)`. Adding a new tool to `mcp/tool-registry.ts` automatically exposes it to Aero — no separate registration in this file. |
| `src/agent/mcp-to-agent-tool.ts` | Adapter that converts every `CanonryMcpTool` into a pi-agent-core `AgentTool`. Strips `project` from the LLM-visible schema and injects `ctx.projectName` at call time. `AERO_EXCLUDED_MCP_TOOLS` lists tools that ride the registry but should not reach Aero (e.g. `canonry_agent_clear` — Aero must not erase the operator's transcript). |
| `src/agent/remote-mcp.ts` | `loadExternalMcpTools(servers, opts)`, the injected remote-MCP load path. For each configured `{ url, token, label? }` it connects to a REMOTE MCP server over the FROZEN transport (bearer-gated MCP Streamable HTTP, `connectStreamableHttp`), `listTools()`, and adapts each tool into an `AgentTool` (mirroring `mcp-to-agent-tool.ts`). Read-only filter: a remote tool is adopted ONLY when `annotations.readOnlyHint === true` AND its name is not in the local `AERO_EXCLUDED_MCP_TOOLS` set. Fail-soft: a server that fails to connect/list is logged and skipped, never throwing the whole load; no servers configured returns `[]`. The transport is the contract a remote MCP server must speak (see "Injected remote-MCP load path" below). |
| `src/agent/skill-tools.ts` | 2 skill-doc tools (`list_skill_docs`, `read_skill_doc`) — progressive disclosure of bundled reference playbooks. Ride in every scope. |
| `src/agent/skill-paths.ts` | `resolveAeroSkillDir` — finds the on-disk `skills/aero/` (prod/dev/repo candidate paths) for the prompt loader and skill-doc tools |
| `src/agent/agent-routes.ts` | Fastify routes — `GET/DELETE transcript` + `POST prompt` (SSE) for the dashboard Aero bar |
| `src/agent/pi-runtime.ts` | Thin factory re-exporting pi-agent-core types with canonry-scoped construction |
## Patterns
### How to add a CLI command
1. Create or extend a file in `src/commands/` for the domain.
2. Add a command spec to the `REGISTERED_CLI_COMMANDS` array in `src/cli-commands.ts`:
```typescript
{ path: ['mycommand', 'subcommand'], usage: 'Description', run: myHandler }
```
3. The CLI dispatches based on `path` matching argv.
### ApiClient usage (Critical)
**Always use `createApiClient()`** — never instantiate `ApiClient` directly:
```typescript
import { createApiClient } from '../client.js'
function getClient() {
return createApiClient() // handles basePath, config loading automatically
}
```
All `ApiClient` methods must return typed DTOs from `@ainyc/canonry-contracts`. Never cast responses with `as Record<string, unknown>`.
**Every `ApiClient` method delegates to the generated SDK via `invoke()`.** Adding a new method is:
```typescript
import { getApiV1ProjectsByNameMyNewThing } from '@ainyc/canonry-api-client'
async myNewThing(name: string): Promise<MyNewDto> {
return this.invoke<MyNewDto>(() =>
getApiV1ProjectsByNameMyNewThing({ client: this.heyClient, path: { name } }),
)
}
```
`invoke()` handles base-path probing, CliError mapping, structured-error envelopes, and the `CANONRY_TRACE=1` request log. **Do not call `fetch()` directly** — ESLint blocks it in `packages/canonry/src/**` except in a handful of files that legitimately hit external HTTP (`telemetry.ts` → telemetry collector, `update-check.ts` → npm registry, `sitemap-parser.ts` → user sitemap, `commands/daemon.ts` → localhost health probe). If you need raw `fetch()` for a NEW external service, add the file to the `ignores` list in `eslint.config.js` with a one-line comment naming the service.
The legacy `request<T>()` raw-fetch wrapper was removed in v4.51; if you find any reference to it, replace with an SDK call through `invoke()`.
### MCP adapter
`canonry-mcp` is the only MCP executable. It is allowed only as a stdio adapter over `createApiClient()` and must not import DB modules, API routes, job runners, CLI command dispatch, telemetry, or loggers. It must never write to stdout except MCP protocol frames. Add tools only when the same capability already exists through the public API/CLI, and keep input schemas tied to `packages/contracts` Zod schemas.
MCP parity is the default for every new public API/CLI capability. When adding a command or `ApiClient` method, either add the matching tool in `src/mcp/tool-registry.ts` and update `docs/mcp.md` + MCP tests, or classify the OpenAPI operation as `deferred` / `excluded-protocol` in `src/mcp/openapi-classification.ts` with a short rationale. Security-sensitive credential/token operations may be deferred, but the PR must explain the exception.
Cloudflare connect is intentionally deferred. It is a local deployment workflow that reads Canonry's local credential store. Direct push installs Worker secret bindings; Queue pull keeps its API token server-side. Do not add `canonry_traffic_connect_cloudflare` to MCP or Aero; agents may instruct the operator to run the CLI, inspect the exact zone route, attach it manually with Fail open, and pass both acknowledgement flags. Credentials and deployment material must not enter a transcript.
### Command output
All commands that produce output must support `--format json` for machine-parseable output. Use the format flag to switch between human-friendly tables and JSON.
There is a third, agent-first format: **`--format jsonl`** (newline-delimited JSON). Where `json` returns the whole envelope as one pretty-printed document, `jsonl` streams the command's **primary collection** one compact, self-contained record per line. It exists so an agent can read
```
canonry doctor --check 'google.auth.*' --format jsonl
```
directly — no `2>/dev/null`, no `| jq '(.checks // .results // [])[] | ...'`, no guessing the envelope key. Each line stands alone for `grep`/`head`/per-line `jq`.
`jsonl` is wired into **every collection command** (a command whose primary output is a list) — `insights`, `runs`, `evidence`, `history`, `query/keyword/competitor list`, `notify list/events`, the `google`/`bing`/`ga` reads, `ads geo search`, `ads conversions pixels/event-settings`, `traffic events/sources/status`, `discover list/show`, `content targets/sources/gaps`, `backlinks list/releases`, `project list/locations`, `agent memory list`, `agent providers`, and `doctor`. The per-command primary collection + the context each line carries is catalogued in `skills/canonry/references/canonry-cli.md` ("Output schema per command" + "Output Formats"). Composite/object commands (`status`, `ads account`, `analytics`, `ga traffic`/`attribution`, `gbp summary`, index `coverage`) have **no** `jsonl` — they stay `json`.
Conventions when adding `jsonl` to a command:
- **Emit via `emitJsonl(records)`** from `src/cli-output.js` (one `process.stdout.write`, compact, newline-terminated). Don't hand-roll `console.log(JSON.stringify(...))` per row.
- **Pick the primary collection** the agent is really after (doctor → `checks`, list commands → their array). Carry any context the line loses by leaving the envelope into each record by spreading context first, record last: `emitJsonl(rows.map(r => ({ project, ...r })))`. Project-scoped lists stamp `project`; add whatever else the envelope held (`window` for `ga *-history`, `release`/`targetDomain` for `backlinks list`, `sessionId` for `discover show`, `isDefault` for `project locations`). Global lists whose rows self-identify (`project list`, `notify events`, `backlinks releases`) emit bare.
- **Never let `--format jsonl` fall through to human text (the invariant).** A command that isn't a streamable collection must still emit its JSON document for `jsonl` — gate the machine-output branch on `isMachineFormat(format)`, **never bare `format === 'json'`**. Since `jsonl` was added as a global format value, a `=== 'json'` gate sends a `--format jsonl` request straight into the human-text `else` — an agent asks for machine output and silently gets a decorated table. Every composite/object/mutation command degrades `jsonl` → its `json` document this way; only the collection commands (which have an explicit `else if (format === 'jsonl')` streaming branch) keep a bare `=== 'json'` gate, because their `json` envelope and `jsonl` records are different outputs.
- **Watch for a local `toFormat`.** A few commands (the `agent-*` handlers) historically narrowed format to `json`-or-`text` inline, which silently swallows `jsonl`. If a command has its own `toFormat`, widen it to pass `jsonl` through (or delete it and use `opts.format` / `isMachineFormat`).
- **Errors and exit codes are unchanged** — `printCliError` renders a single-line envelope for `jsonl`, and a failing command still throws its `CliError` (so e.g. `doctor` still exits 1 after printing every check line). Agents branch on the exit code, not on parsing stderr.
- **USAGE strings** are not mass-updated to advertise `json|jsonl` (cosmetic; the global `cnry --help` and the skill reference cover discoverability) — `doctor`'s does as the exemplar.
The stable, machine-readable formats are `json` and `jsonl`; `text` is decorated and not a parse target. `isMachineFormat(format)` distinguishes them.
The CLI also keeps interactive chrome (the "new version available" banner) off a non-interactive stderr — it's gated on `process.stderr.isTTY`, so piped/captured output stays clean without `2>/dev/null`. Errors still go to stderr.
### Run completion pipeline
When a sweep finishes, the flow is: `JobRunner` → `RunCoordinator.onRunCompleted()` → `IntelligenceService.analyzeAndPersist()` then `Notifier.onRunCompleted()`. The coordinator runs intelligence first (synchronous) so insights are persisted before webhooks fire. Each subscriber is wrapped in an independent try/catch — one failing must not block the others.
`IntelligenceService` reads query snapshots from the DB, calls the pure analysis functions in `packages/intelligence/`, and persists insights + health snapshots. It also provides `backfill()` for reprocessing historical runs chronologically.
#### What may serve as a comparison baseline (Critical)
**Scope the history window in SQL, before the `LIMIT`.** `HISTORY_WINDOW_RUNS`
bounds ROWS; the window has to mean *N sweeps at this run's location*. Anything
scoped afterwards is scoped against a page that already dropped the rows it
needed. Both predicates live in `analyzableRunPredicates()` + the location
clause, and both were learned the hard way:
- **Kind.** Only `answer-visibility` writes `query_snapshots`; the other 14
kinds (`ga-sync`, `traffic-sync`, `site-audit`, …) write none. Let one win
the "previous run" slot and every detector gets an empty baseline — not a
neutral one, a baseline in which nothing was ever cited. `detectFirstCitations`
and `detectGains` report every currently-cited query as brand new,
`detectRegressions` finds no prior citation to lose so real regressions vanish
silently, and `detectPersistentGaps` breaks each query's streak on the run
that lacks it. Worse the better an instance is configured: daily syncs +
weekly sweeps means the row before a sweep is almost always a sync.
- **Location.** A sweep fans out to one run per configured location, so
siblings compete for the same row budget. At 3 locations a 5-row page holds
under 2 sweeps per location — the baseline vanishes and ALL transition
detection goes dead. At 6+, an arm falls outside its own window and used to
be refused outright: no insights *and no health snapshot*, so that location
left the dashboard rather than reading as flat.
Neither is visible in a fixture-only test. A test that seeds two sweeps at one
location never puts a foreign kind or a sibling arm in the window, which is why
both survived a green suite for so long.
Two supporting rules:
- **Eligibility is a property of the run, not of its recency.** Resolve the run
being analyzed by id under `analyzableRunPredicates()`; never find it inside
the window. Finding it there silently makes "is this analyzable?" depend on
"is it one of the N newest rows?" — that coupling is what dropped fan-out
arms. The window is then anchored with `lte(runChronologyKey, …)` so it never
looks forward, which also makes re-analyzing a historical run correct instead
of refused.
- **A run with no snapshots is not a comparison point.** The kind filter drops
sync/audit runs, but an `answer-visibility` run can still measure nothing (a
`completed` sweep with no tracked queries at sweep time) and anchor the same
false transitions, so the window also requires the run to have written
snapshots. In `analyzeAndPersist()` that is a third SQL predicate
(`exists(query_snapshots …)`) alongside kind and location — a post-LIMIT drop
would be the same trap as location, letting enough empty rows push the last
real sweep out of the window. `backfill()` loads the full chronology (no
limit), so it filters with `runIdsWithSnapshots(...)` after the read; the
comparison skips to the last run that measured either way.
All of this applies to `backfill()` as well as `analyzeAndPersist()`. Backfill
is the reanalyze path operators run to clear bad rows; without the same
predicates it rewrites exactly the insights it was invoked to remove. (Backfill
loads the full chronology rather than a window, so it needs the predicates but
not the limit reasoning.) Clearing a historical instance is `canonry backfill
insights <project>` — add `--dry-run` first to see the delta.
### Index coverage auto-refresh
`gscUrlInspections` (the index-coverage dashboard's source of truth) is populated **only** by an `inspect-sitemap` run. `gsc-sync` does not inspect URLs at all: each URL Inspection call costs ~7.1s (Google's own live-index-lookup latency plus the ~1.1s pacing its 1 req/sec soft limit needs) and one unit of a 2000/property/day quota, so inspecting inline made the sync scale with the site — measured at 240.9s of a 241.9s run for 31 URLs while the search-analytics work took 1.07s, past the dashboard's 120s poll window. `server.ts` chains a full GSC `inspect-sitemap` off the **success** of both `executeGscSync` (`gsc-sync`) and `executeBingInspectSitemap` (`bing-inspect-sitemap` — Bing's coverage sync, which has no separate `bing-sync` kind). `packages/canonry/test/gsc-sync-no-inspection.test.ts` guards the split. The chaining lives in the `onGscSyncRequested` / `onBingInspectSitemapRequested` callbacks, so it covers UI and CLI uniformly (both hit the same endpoints) and the dashboard "Refresh search data" button.
`maybeRefreshGscCoverage` (`src/coverage-refresh.ts`) owns the decision: it no-ops when GSC isn't connected for the project (so the Bing → GSC chain is silent on Bing-only projects), always skips while an `inspect-sitemap` run is `queued`/`running` (the in-flight guard, which is what collapses the GSC and Bing arms of "Refresh search data" into one sweep), and otherwise applies a spacing window sized by **who asked**:
| Caller | Window | Constant |
|---|---|---|
| Scheduled / chained | 1 h | `COVERAGE_REFRESH_MIN_INTERVAL_MS` |
| User-initiated (`opts.userInitiated`) | 2 min | `COVERAGE_REFRESH_MANUAL_MIN_INTERVAL_MS` |
The split exists because the original single window silently broke the dashboard's "Refresh search data" button. The 1 h figure was chosen when `gsc-sync` still inspected the top 50 pages inline, so skipping the full sweep only meant "slightly less thorough". After inline inspection was removed, `gsc-sync` inspects **nothing**, and a skipped sweep means no URL is inspected at all — index coverage becomes structurally incapable of changing. A user pressing the button 46 min after the daily scheduled refresh got a successful-looking sync and unchanged numbers, with `skip.recent` the only trace. `server.ts` classifies the caller via `runWasUserInitiated(db, runId)`, reading `runs.trigger` off the triggering run so both chain sites and both surfaces (CLI, dashboard) agree.
The chained run is `trigger: scheduled`; a refresh failure is logged, never bubbled into the triggering sync's result.
### Rate gating across runs (Critical)
`inspectUrlsPaced` takes its spacing from a gate named by `deps.rateGateKey` (`src/inspect-rate-gate.ts`). **The key must name the SHARED upstream limit, not the caller** — `bing:<digest of api key>`, `gsc:<propertyId>`. Omitting it gives the sweep a private gate that no other sweep observes.
That is not a style preference. The gate used to be a `let` inside `inspectUrlsPaced`, so it lived exactly as long as one call: every concurrent sweep got its own "1 req/sec" budget while Bing meters the **API key**, which one instance shares across every connected project. Pacing a second caller can ignore is not a rate limit. For the same reason **every** request start takes a slot, including the first — exempting index 0 let each concurrent sweep fire one ungated request the moment it began.
**Do not stack retry layers.** `bing-inspect-sitemap` passes `maxRetries: 0` because `bingFetch` already retries: the two multiplied rather than added (5 client attempts x 4 driver attempts = 20 HTTP requests for one throttled URL), so the breaker's five-failure budget spent ~100 requests and 335 s to inspect nothing. Retry belongs to the client (see the root AGENTS.md "Third-party HTTP calls" rule); the driver owns pacing and the circuit breaker.
`executeInspectSitemap` drives its `inspectUrl` calls through `inspectUrlsPaced` (`src/gsc-inspect-paced.ts`), which paces ~1 req/sec with jitter, retries transient rate responses with jittered exponential backoff, and trips a consecutive-failure circuit breaker. The endpoint signals per-minute quota pressure with a transient 403 (`PERMISSION_DENIED`-shaped) rather than a 429, so a 403 is treated as a soft, retryable rate signal here while genuine 401/400 stay non-retryable. When the breaker trips, `inspect-sitemap` fails the run, so a quota or property-access outage does not overwrite the coverage snapshot with a misleading all-not-indexed reading.
### Backfill behavior
`canonry backfill answer-visibility` does more than recompute `answerMentioned`. It also reparses stored provider `raw_response` payloads for supported API providers (OpenAI, Claude, Gemini, Perplexity) and refreshes derived snapshot fields such as `citationState`, `citedDomains`, `groundingSources`, and `searchQueries`.
### Provider registration
Providers are registered at server startup in `server.ts`. Each provider adapter (from `packages/provider-*`) is imported and added to the `ProviderRegistry`. Projects reference providers by name.
## Common Mistakes
- **Instantiating `ApiClient` directly** — use `createApiClient()` which handles basePath and config.
- **Casting API responses** — use typed DTOs from contracts, not `as { ... }`.
- **Forgetting `--format json` support** — every output command needs it.
- **Forgetting to register command in `cli-commands.ts`** — the command won't be accessible.
## Agent layer (Aero)
Canonry ships a built-in AI agent called **Aero**, built on
[`@mariozechner/pi-agent-core`](https://github.com/badlogic/pi-mono). Users
who already have their own agent (Claude Code, Codex, custom) can still
consume Canonry through the external-agent webhook.
### Built-in agent (native loop)
- **CLI**: `canonry agent ask <project> "<prompt>"` — one-shot turn. Streams
`AgentEvent` lines to stdout (or JSON with `--format json`). Supports
`--provider claude|openai|gemini|zai|deepinfra` and `--model <id>`. `zai` and
`deepinfra` are agent-only; `deepinfra` is an OpenAI-compatible host outside
pi-ai's catalog (`agent/providers.ts` builds a custom `openai-completions`
model against `https://api.deepinfra.com/v1/openai`; key from `DEEPINFRA_TOKEN`,
base URL overridable via `DEEPINFRA_BASE_URL` for proxy/LiteLLM-gateway routing).
- **Dashboard**: bottom command bar (`AeroBar`) on every project-scoped
route. SSE-streamed via `POST /api/v1/projects/:name/agent/prompt`.
- **Proactive**: `RunCoordinator` enqueues a synthesized `[system]` follow-up
into the project's session after every `run.completed`; `SessionRegistry.drainNow`
wakes the agent unprompted so insights/failures get analyzed without a
user click.
- **Persistence**: one `agent_sessions` row per project. Transcript + queued
follow-ups survive `canonry serve` restarts. See `docs/data-model.md`.
- **Memory**: durable project-scoped notes in `agent_memory` (key/value +
source). Written via `remember` tool (or CLI / API), read via `recall`, and
the N most-recent rows are injected into every new session's system prompt
under a `<memory>` block so notes take effect immediately on next session.
Hydrate is capped at 20 rows / 32 KB, oldest-first truncation. Keys with
the `compaction:` prefix are reserved for summarized transcript slices.
- **Compaction**: once a transcript crosses `COMPACTION_TOKEN_THRESHOLD` or
`COMPACTION_MAX_MESSAGES`, `acquireForTurn` awaits a one-shot summarizer
(`pi-ai` `complete()` on the session's current model) that rolls the
oldest half of the transcript into a `compaction:<sessionId>:<iso>`
memory row, removes those messages from `agent.state.messages`, and
rehydrates the system prompt so the next LLM call sees the summary in
its `<memory>` block. Splits are snapped to user-message boundaries to
avoid orphaning tool calls from their results. Concurrent compaction
runs for the same project dedupe via an in-flight promise map.
Tool surface has two layers:
- **Canonry state** (`src/agent/tools.ts` → `mcp-to-agent-tool.ts`) — every
tool from `src/mcp/tool-registry.ts` minus the `AERO_EXCLUDED_MCP_TOOLS`
set, adapted into pi-agent-core `AgentTool`s. The adapter strips the
top-level `project` property from each tool's JSON schema and injects
`ctx.projectName` at call time, so the LLM never sees raw project ids and
cannot target the wrong project. Result: **adding a new tool to the MCP
registry automatically makes it available to Aero — no second
registration**. Tool intent surfaces via `tool_execution_start` events.
- **Skill docs** (`src/agent/skill-tools.ts`) — 2 tools (`list_skill_docs`,
`read_skill_doc`) for progressive disclosure of bundled reference playbooks.
These stay Aero-only because they read on-disk skill files, not API state.
Ride in every scope. `SKILL.md` stays lightweight; detailed playbooks
(workflows, regression diagnosis, reporting templates, integrations) load
on-demand via slug.
- **Aero tool profiles** (`src/agent/tools.ts`) — the default profile exposes
the full local MCP-derived tool surface for the requested scope. The
`ads-operator` profile narrows local state tools to an explicit typed
allow-list for ads reads, durable lifecycle writes, and prep, and prepends `canonry_ads_operator_context`, an
Aero-only context-packing helper that composes existing project overview,
ads, doctor, and memory reads through `ApiClient`. It does not expose a new
capability outside Aero; promote it to API/CLI/MCP only if operators need
that exact bundle as a public contract rather than as a long-session prompt
optimization.
- **Injected remote MCP tools** (`src/agent/remote-mcp.ts`), read-only tools
loaded from external MCP servers configured via `config.externalMcpServers`
(or the `CANONRY_EXTERNAL_MCP` env var, a JSON array of `{ url, token, label? }`).
Loaded once per `SessionRegistry` lifetime (cached promise) and merged into
each session's tool list in `acquireForTurn`, after local-scope/profile
alignment. This is intentional: profile narrowing applies to the local
Canonry tools, while injected remote tools are separately accepted only when
their MCP annotations mark them read-only.
See "Injected remote-MCP load path" below for the frozen transport + filter.
### Injected remote-MCP load path (OSS-A)
Aero can load tools from a REMOTE, externally-hosted MCP server injected via
config/env. This is a generic capability, no host names, no domain logic.
- **Frozen transport (the contract a remote MCP server must speak):**
**token-gated MCP Streamable HTTP**, `StreamableHTTPClientTransport` from
`@modelcontextprotocol/sdk/client/streamableHttp.js`, with the bearer token
carried in the transport's request headers (`Authorization: Bearer <token>`).
SSE is legacy and is NOT used. The remote server is platform/external code,
never co-located in the OSS container; the injected env carries only
`{ url, token, label? }` and Aero connects out. Per-tenant isolation is the
token's responsibility (scoped server-side), NOT the container boundary.
- **Read-only filter (always applied):** a remote tool is adopted ONLY when it
is read-only, keyed off the MCP `annotations.readOnlyHint === true` flag -
AND its name is not in `remote-mcp.ts`'s `AERO_EXCLUDED_MCP_TOOLS`. Write
tools and excluded tools never reach Aero.
- **Resilience:** a server that fails to connect or list its tools is logged
and skipped; one bad server never aborts the whole load. No servers
configured returns `[]` (default path is byte-identical to today).
- **Config:** `CanonryConfig.externalMcpServers?: ExternalMcpServerConfig[]`
(`config.ts`), parsed from `CANONRY_EXTERNAL_MCP` (env wins over config.yaml)
by `parseExternalMcpEnv`. Malformed/non-array/entry-missing-url-or-token
values are ignored fail-soft.
- **Seam:** `loadExternalMcpTools(servers, opts)` is the async load function.
`createAeroSession` stays synchronous; the registry awaits the load in its
already-async `acquireForTurn` and merges the returned tools into
`agent.state.tools`. `opts.connect` is the test injection point (the
InMemory transport replaces the production StreamableHTTP transport).
### Generic system-prompt append seam (OSS-D)
`appendSystemPromptExtras(base, env?)` (`session.ts`) appends
`AERO_SYSTEM_PROMPT_APPEND` (inline) and/or the contents of
`AERO_SYSTEM_PROMPT_FILE` (a mounted file path) AFTER the base soul+SKILL
prompt, separated by a divider. Empty by default => byte-identical. Generic, no
product vocabulary. It lives inside `loadAeroSystemPrompt`, so it covers BOTH
the one-shot `createAeroSession` default path and the registry (which builds on
`loadAeroSystemPrompt`, then layers the `<memory>` block AFTER, so the appended
rules frame the task and precede per-session memory). A `systemPromptOverride`
(tests / explicit full control) deliberately bypasses it. A missing/unreadable
file is skipped, never breaking the agent. The FILE variant exists so a multi-KB
prompt is mounted, not crammed into a single `-e` arg.
### Evidence-safe tool-result truncation (OSS-C)
`truncateToolResult` (`mcp-to-agent-tool.ts`) renders a tool result under the
20 KB cap WITHOUT cutting a row mid-structure. The previous guard blind-sliced
the serialized JSON, which could split an array element halfway (invalid JSON)
and silently drop a cited evidence row mid-object. Now: an object whose largest
field is an array drops WHOLE trailing rows and stamps `__truncated` +
`__omittedRows`; a top-level array is wrapped as `{ items, __truncated,
__omittedRows }`; only a giant scalar with nothing structured to drop falls back
to a marked string slice. Every retained row stays byte-intact; the programmatic
`details` envelope is never trimmed, only the model-facing text.
System prompt is composed from `skills/aero/soul.md` (identity/voice/values)
+ `skills/aero/SKILL.md` (task rules). Soul is prepended so identity frames
the task instructions. Both files ship in `assets/agent-workspace/skills/aero/`.
The `<memory>` hydrate block is appended at session-build time by
`SessionRegistry.buildHydratedSystemPrompt` — the DB row keeps the raw
(unhydrated) prompt so every new session sees the latest notes.
### Disabling Aero
Aero is enabled by default. Set `agent.mode: 'disabled'` in
`~/.canonry/config.yaml` (or `CANONRY_AGENT_DISABLED=1` in the environment;
env wins, `=0` forces it back on) to turn the agent OFF entirely — the
proactive auto-wake on `run.completed`, the `SessionRegistry`, and the
interactive agent routes + `canonry agent ask` are all skipped. `server.ts`
resolves this once at boot via `resolveAgentEnabled(process.env, config)`
(`src/agent-config.ts`) and guards the three Aero wiring points. Data syncs,
intelligence, and notifications are unaffected. Note it stops only the
*automatic* per-run agent stream — the on-demand `analyze`-tier recommendation
`explain` / `brief` routes (Sonnet) still bill on explicit user action.
### External agents (webhook lifecycle)
`canonry agent attach <project> --url <webhook-url>` registers an agent
webhook subscribing to `run.completed`, `insight.critical`, `insight.high`,
`citation.gained`. Idempotent — skipped if one already exists on the project.
`canonry agent detach <project>` removes it.
## Telemetry events
Anonymous fire-and-forget telemetry, opt-out via `canonry telemetry disable`,
`CANONRY_TELEMETRY_DISABLED=1`, `DO_NOT_TRACK=1`, or any truthy `CI` env var.
All events POST to `https://canonry.ai/api/telemetry` with the following
top-level envelope:
```jsonc
{
"eventId": "uuid-v4", // unique per emission; receiver deduplication key
"anonymousId": "uuid-v4", // stable per-install (~/.canonry/config.yaml)
"sessionId": "uuid-v4", // per-process — same for every event in one CLI invocation / serve boot
"source": "cli", // surface that emitted; see TelemetrySource below
"sourceContext": "...", // optional sub-source ("php/8.2 wp-cron")
"event": "cli.command",
"timestamp": "...",
"version": "4.15.0",
"nodeVersion": "...",
"os": "darwin",
"arch": "arm64",
"errorCode": "NO_PROVIDERS", // present only when the event represents a failure
"properties": { ... }
}
```
### Source taxonomy (`TelemetrySource`)
| Source | When |
|--------|------|
| `cli` | One-shot CLI command (`canonry run`, `canonry status`, …) |
| `cli-server` | Long-running `canonry serve` process (set via `setTelemetrySource('cli-server')` after the server boots, so dashboard/API/scheduler-driven events ride this source) |
| `api` | Reserved — direct API caller (cloud `apps/api`) |
| `mcp-server` | Reserved — `canonry-mcp` stdio adapter (currently forbidden from emitting per `Surface Priority → Agent & automation design principles → MCP adapter boundary`; emission would require a separate adapter) |
| `wp-plugin` | Reserved — WordPress plugin |
| `dashboard` | Setup milestones forwarded by the local API; still obeys the instance telemetry opt-out |
| `agent-runtime` | Reserved — Aero / external agent runtimes |
### Event catalog
| Event | Properties | Notes |
|-------|-----------|-------|
| `cli.command` | `{ command, setup_state? }` | Fires on every CLI invocation except `telemetry` and `--help`. `setup_state: { provider_count, has_keywords, project_count, is_first_run }` lets the receiver cohort by configured / not-configured. |
| `cli.command.finished` | `{ command, success, duration_bucket, setup_state? }` | Terminal command outcome. Command is resolved from the registered command catalog; raw argv and exact duration are never sent. Failures use top-level stable `errorCode`. |
| `cli.init` | `{ providerCount, providers, setup_state?, googleConfigured, agentConfigured, setupState, skillsInstalled }` | Fires after successful `canonry init`; `setup_state` is the post-init snapshot. `setupState` is retained temporarily for report compatibility. |
| `cli.upgraded` | `{ fromVersion, toVersion }` | Fires once when the on-disk `lastSeenVersion` differs from the running build. Suppressed on a fresh install (no prior version recorded). |
| `serve.started` | `{ providerCount, providers }` | Fires after `canonry serve` opens its listener; this is also when `source` flips to `cli-server`. |
| `run.completed` | `{ status, providerCount, providers, queryCount, durationMs, trigger?, domainHash?, phases?, location? }` | `trigger` mirrors `runs.trigger` (`manual` / `scheduled` / `config-apply`). `domainHash` = SHA-256 of the project canonical hostname (no raw domains stored). `phases = { setup_ms, provider_call_ms, total_ms }`. Failures additionally set top-level `errorCode` from `RunErrorCode` (`PROJECT_NOT_FOUND` / `RUN_NOT_FOUND` / `RUN_NOT_EXECUTABLE` / `NO_PROVIDERS` / `QUOTA_EXCEEDED` / `RUN_CANCELLED` / `PROVIDER_ERROR` / `INTERNAL`). |
| `onboarding.started` | `{ flowVersion, onboardingSessionId, step, resumed }` | Dashboard setup entry/resume. Accepted through the strict `/api/v1/telemetry/onboarding` contract; the browser-generated top-level `eventId` is preserved across one delivery retry. |
| `onboarding.step_completed` | `{ flowVersion, onboardingSessionId, step, method, countBucket? }` | Durable setup milestone; raw domains, queries, and keys are forbidden by the contract. |
| `onboarding.blocked` | `{ flowVersion, onboardingSessionId, step, action, reasonCode }` | Stable recovery point and low-cardinality blocker. |
| `run.requested` | `{ flowVersion, onboardingSessionId, origin, result, providerCountBucket, queryCountBucket, reasonCode? }` | Setup launch attempt, including synchronous rejection. |
| `activation.completed` | `{ flowVersion, status, providerCountBucket, queryCountBucket, snapshotCountBucket }` | Emitted once for the first non-probe answer-visibility run that persists at least one snapshot. |
### Adding a new event
1. Pick a stable `event` name and add it to the catalog above.
2. Call `trackEvent(event, properties, options)` from the originating code path. Pass `options.source` if the default global source is wrong; pass `options.errorCode` for failure events.
3. If the event represents a run/job result, compose the payload through a helper in `src/run-telemetry.ts` (or a sibling) so the shape stays in one place — never inline a new property bag in three call sites.
4. Add tests that assert the new field/event in `packages/canonry/test/telemetry.test.ts` (or a dedicated file like `run-telemetry.test.ts`).
5. Bump the version in both `package.json` files.
## See Also
- `packages/api-routes/` — the route handlers this server mounts
- `packages/contracts/` — DTOs returned by the API client
- `docs/architecture.md` — how CLI, server, and job runner interact