source-review · diff
git:20261104.8d35018 to git:20261114.9f9908d
5 added, 2 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: source-review
description: Review a new or changed Omnesis data source against the complete provider contract before opening its pull request.
---
Review a new or changed Omnesis **data source** — a provider package under
`packages/providers/<name>/` and everything it must touch outside that
package — against the full source contract. The goal is that nothing a
source needs is forgotten and nothing is built in a way the gateway,
collector, portal, mobile apps, Watch, or the people graph cannot use.
The source contract is spread over many files; this skill is the one place
that lists all of it. Most of the ways a source can be wrong are **silent**:
the package builds, its tests pass, and the source syncs — but it shows a
placeholder icon, cannot be named by a watch, never contributes people, drops
inbound links, or duplicates itself on a second device. Treat "it works on
my machine" as no evidence for any check below.
Do § 0 yourself. When delegation is available and permitted, use the host
agent's sub-agent tools to review these groups in parallel, each with the scope
from § 0 (file list, descriptor inventory, baseline output) and its sections.
Otherwise, review every relevant group yourself and record any checks you
cannot perform:
- **A — viability, terms, encapsulation, descriptor, auth** (§ 1–4). The
terms-of-service check needs network access; if the sub-agent has none,
it records the URLs it would read and the questions to answer.
- **B — sync runtime, deletions, documents and rows** (§ 5–7)
- **C — people and URLs** (§ 8–9)
- **D — presentation and Watch** (§ 10–11)
- **E — multi-instance, multi-device, phone push** (§ 12–14)
- **F — touchpoints outside the package, tests** (§ 15–16)
Only apply checks that are relevant to the source under review (a push-only
phone source has no sync loop; a source with no human counterpart emits no
people). Every finding names the file and line, the contract it violates,
and the user-visible consequence.
---
## 0. Scope and baseline
Identify the source under review. The user's accompanying instructions may name the provider package
(`github`, `packages/providers/github`) or a source id; when no source is specified, derive the
scope from `git diff main...HEAD --stat` plus the staged and unstaged diffs and new files — everything
under `packages/providers/<name>/`, its synth twin under
`packages/providers-synth/<name>/`, and every file outside them the branch
touched.
Read the source first, then the contract only where the source touches it:
1. `packages/providers/<name>/src/index.ts` (the descriptor) and its
normalizer, end to end.
2. For each descriptor field the source sets — and each one it does not —
the doc comment in `packages/source-sdk/src/define-source.ts`
(`SourceDefinition`, `ProviderDefinition`, `SourceInstance`,
`CreateOptions`) and `packages/source-sdk/src/source-descriptor.ts`.
Those comments are the contract; quote them in findings.
3. `packages/source-sdk/src/source.ts` (`SyncResult`, `SourceIcon`,
`SourceFreshness`) and, for analytics or Watch declarations,
`packages/source-sdk/src/structured-source.ts`.
4. `packages/types/src/document.ts` (`DocumentInput`, `DocumentMetadata`,
`PersonMention`, `PERSON_ROLES`).
5. `website/docs/building-sources.html` — the condensed public contract.
It does not list every field; where the source **contradicts** it, one of
the two is wrong and the review says which.
6. `source-contract.ts`, `source-state.ts`, `state-runtime.ts`,
`config-schema.ts`, `auth-session.ts`, `source-host.ts`, and `table-write.ts`
under `packages/source-sdk/src/` for the declared v2 features. Verify host
wiring too: a type existing in the SDK does not prove the host implements it.
Build a **descriptor inventory** in one pass and hand it to every
sub-agent — which of these the source declares, with the value:
```bash
rg -n "contract|apiVersion|outputRevision|requires|config:|authenticate|credentialState|probeLocalStores|acceptsAuthCode|experimental|unitName|primaryCount|icon|attribution|urlPatterns|ownedWebDomains|urlCanonicalizer|conversational|selfIdentity|singleInstance|multiDevice|execution|pushBased|params|historyImport|analyticsSchemas|documentTemporalProjections|documentEventProfile|defaultSyncInterval|supportedPlatforms|credentials|freshness|watchPaths|onResync|suspend|resume|dispose|isAuthenticated|cleanupCredentials|discover" \
packages/providers/<name>/src/index.ts
rg -n "console\.|opts\?\.signal|signal[,)]|DEFAULT_CONFIG_DIR|process\.env" packages/providers/<name>/src -g '*.ts' -g '!*.test.ts'
```
Run the mechanical signals and carry their output into the report:
```bash
npm run typecheck:fast -- packages/providers/<name>
npm run test:unit:vitest -- packages/providers/<name>
npm run test:unit:vitest -- packages/source-sdk/src/snapshot-contract.test.ts \
packages/collector/src/document-event-profiles.test.ts \
packages/collector/src/record-citation-contract.test.ts
npm run validate-universes
npx eslint packages/providers/<name>
node scripts/pii-scan.mjs --all
npx knip
npm run checks:plan -- --base origin/main
npm run checks:affected -- --base origin/main
```
What these do and do not prove: the snapshot-contract test reddens on an
unregistered snapshot emitter; the document-event-profiles test validates
every **published** profile and their mutual distinguishability but does not
require that this source publishes one; the record-citation test covers
sources with analytics schemas. ESLint's source-encapsulation guard only
matches a `sourceType === "<literal>"` comparison in the shared packages, at
warn level — the greps in § 2 are the real encapsulation check. The full
`npm run lint` is slow and its result is dominated by unrelated files; do not
run it here.
---
## 1. Viability and platform terms
- **Auth once, then unattended.** After a single authentication the collector
must be able to sync forever with no user action in the ingestion path.
Reject any design where the only way data arrives is a user-triggered
export, upload, or paste (CSV files, "download your data" archives, a
browser tab that must stay open, an app that must be in the foreground, a
session that expires without a refresh path). The one sanctioned
user-touched path is `historyImport` — a one-time, idempotent bulk import
that sits **beside** a working background sync, never instead of it. A
source that fails this rule is a "Won't implement" row on the source
roadmap, not a code review.
- **Terms of service.** Read the platform's developer / API terms and its
acceptable-use policy (web fetch; cite the URL and the date read).
Record in the review: (a) whether a user-run local application may retrieve
and store the user's own data with the credential type the source uses,
(b) any restriction on caching, retention, or derived data, (c) required
attribution or branding (declare it via `attribution` on the descriptor and
a `TRADEMARKS.md` row), (d) rate limits, quota, and any cap on users per
registered app — the source's `defaultSyncInterval`, paging, and
`retryAfterMs` handling must respect them and the docs must state the cap,
(e) whether the source relies on an unofficial protocol, reverse
engineering, or scraping. Unofficial access is not an automatic reject,
but it is a decision the maintainers make explicitly: surface it as a
blocking finding with the relevant clause quoted, so the PR description
records the decision.
- **Consent scope.** OAuth scopes and API permissions are the minimum the
source needs. A scope requested "for later" is a finding.
## 2. Encapsulation — nothing source-specific outside the package
The `defineSource()` descriptor is the contract. If a consumer needs
something source-specific, the descriptor grows a field and consumers read it
through the registry. Verify:
- Grep the source id, provider id, and display name across `packages/core`,
`packages/gateway` (including `packages/gateway/portal/`),
`packages/collector`, `packages/cli`, `packages/cli-shared`,
`packages/types`, `ios/`, `android/`, and `extension/`. Triage: ignore
comments, doc strings, preview or sample fixtures, and tests that
enumerate every source; the legitimate code hits are generic registries
(`DEVICE_HOSTED_SOURCE_TYPES` for phone sources, `snapshot-emitters.json`,
universes, docs). Anything else that branches, lists, or formats on the
name is a finding — a hard-coded provider list in a CLI command counts.
- No source-specific unit noun, icon, color, label, URL construction,
URL parsing, schema, or normalization in shared code. Each of these has a
descriptor field (`unitName`, `icon`, `name`, `urlPatterns`,
`urlCanonicalizer`, `ownedWebDomains`, `selfIdentity`, `analyticsSchemas`,
`documentEventProfile`, …). Use the field; do not add a case downstream.
- No new gateway route, worker, or migration that exists only for this
source. If the source needs a capability the gateway lacks, the review
asks for a generic capability on the descriptor, not a special path.
- The reverse direction too: the package imports only `@omnesis/core`,
`@omnesis/source-sdk`, `@omnesis/types`, `@omnesis/config` — never
`@omnesis/gateway` or `@omnesis/collector`, and never a relative path into
another package.
## 3. Descriptor and identity
- Exactly one default export: `defineSource()`, `defineProvider()`, or
`defineStructuredSource()`. A platform hosting several data kinds behind
one authentication is one `defineProvider()` with several sources, not
several packages. A single source can declare `authenticate(session)`;
shared credential state and a shared account context belong on
`defineProvider()` even when it currently hosts only one source (see § 4).
- `id` is a stable kebab-case base id with no account suffix; the composed
runtime id is `<id>:<accountId>` and the descriptor id is what
`urlPatterns`, `unitName`, universes and docs are keyed by. A descriptor
id that differs from the `sourceId` prefix (`whatsapp-messages` vs
`whatsapp:<phone>`) silently loses its unit noun in the portal — flag it.
- `name`, `description`, `unitName` (plural noun: "emails", "workouts")
are set on every source entry. `primaryCount` is set when the source emits
both documents and analytics rows, or the headline count shows the wrong
plane.
- `experimental: true` until the source has run against real data for long
enough to trust; the collector then hides it from the add-source picker
unless `OMNESIS_EXPERIMENTAL=1`. An experimental source is documented only
on `website/docs/experimental.html`.
- `supportedPlatforms` is set for a source that reads an OS-specific local
store; omitted for a cross-platform one.
- `singleInstance: true` only when a host genuinely admits one instance
(one local database). Its scope is per host, not global.
- `discover()` may return IDs or `AccountDescriptor`s. IDs are stable keys;
labels, subjects, tenants and aliases describe them, never rename or merge
them. Test legacy ID-only discovery and descriptor refresh on an existing
source. An assigned account absent from authoritative discovery is refused
without preventing an available sibling from loading.
- `defaultSyncInterval` for any platform with a quota or rate limit.
- `conversational: true` only for message streams that render audio inline;
it decides how audio is routed, so a wrong value misroutes voice notes.
- Logger component is `source:<id>` or `provider:<id>`, one-line template
literals. No `console.*` (the § 0 grep lists them; ESLint does not).
- Tunables (page sizes, backoffs, thresholds) live in `omnesisConfigSchema`
or the descriptor, not as literals scattered through the package.
- A source that asks the operator for a setting declares `config:` with
`config.object({...})`, not a hand-written `params:` array. The two are
mutually exclusive and `defineSource` refuses both. The declaration is
what produces the form clients render, the validator the host runs, and
the type `create()` receives — a hand-written array produces only the
first, so the other two get written again by hand and drift.
- Every setting the factory reads is declared. A value read through a cast
over `sourceConfig.params` is a setting no form can produce and no
validator checks; it exists only in a hand-edited file.
- A setting the operator cannot answer while adding the source is
`advanced: true` — declared, parsed and typed, but off the add form.
- A path setting says what has to be true of it as data: `mustExist`,
`mustContain` with `mustContainKind`, and `containsHint` for the one
branch only the source can explain. It does not expand the path itself —
the host resolves a declared path before `create()` sees it, so a source
that calls its own expansion is a source whose factory can disagree with
its own validator.
- `check` is for what data cannot express, and its presence should be
arguable. A blank value that means "detect it" also needs
`checkWhenEmpty`.
- Every optional setting carries `help` saying what leaving it blank does.
The label says what the field is called; only `help` answers the question
the operator actually has.
- Schema defaults, patterns, select options and list element types are valid
at definition time. Required whitespace fails; numeric coercion does not turn
whitespace, booleans, arrays or objects into a number. Test host checks on
each list element, including the derived form validator. Text lists split on
commas/newlines except paths (newlines); use `separator: "newline"` for globs
or other values whose commas are literal. Round-trip installed config strings.
### 3.1 Evolution and host compatibility
- Declare `contract.apiVersion` separately from state version and output
revision. Current-generation packages use 2; omitted means 1. Test refusal
by a host that cannot load the generation, not just definition-time validation.
- `requires` names the capabilities correctness depends on. Scoped claims need
`snapshot-sessions`; versioned migrations, legacy classification, `maxBytes`,
and `onUnreadable: "stop"` need `state-envelope`, even for state version 1.
Verify each required behavior against `HOST_CAPABILITIES` and a real host
code-path test. Never advertise reserved `connection-identity` without opaque
connection allocation and alias resolution actually working.
- A missing capability refuses creation, reaches setup-failure reporting, and
leaves supported siblings operational. Provider and entry requirements merge;
a provider-level state decoder must never migrate a child's bookmark.
- `outputRevision` describes output meaning, not cursor shape. It does not
schedule reprocessing; prove the explicit re-emission/backfill mechanism if
output changes must affect already indexed records.
- `contract.state.decode` accepts every settled, pending, empty-store and legacy
state the source can produce. Supply a complete major migration chain and
classify bare installed cursors with `legacyVersion` where needed. Minor
additions tolerate missing fields. Test real installed cursor fixtures through
`resolveSourceState` and `withVersionedState`, not just the decoder.
- `onUnreadable` reflects replayability: `rebootstrap` only where upstream can
replenish the history; `stop` otherwise. A downgrade or failed migration must
not overwrite the prior cursor. Check operator remediation and repeated ticks.
- `maxBytes` measures the encoded envelope. Test realistic large state, and a
failing ceiling through the wrapper and runner with the prior cursor intact.
Class instances retain getters, private-field receivers and lifecycle hooks.
## 4. Auth and credentials
- `authType` matches the real flow (`oauth | qr | api-key | link-widget |
local`). `link-widget` also needs `widgetOrigins` and `widgetRenderer`, or
the portal's Content-Security-Policy blocks the vendor SDK and the widget
never opens.
- `acceptsAuthCode: true` exactly when `authFlow` awaits
`callbacks.receiveCode()` (grep the provider for it); without the flag the
CLI and portal never offer the paste-code fallback and a headless flow
hangs until timeout. An OAuth redirect is built from
`callbacks.publicBaseUrl` with `callbacks.flowId` as `state`, so a
collector behind NAT still completes.
- No fixed local callback port. A second concurrent add of the same provider
must not fail with an address-in-use error.
- Credentials are written only through the shared secret-file helpers in
`@omnesis/core` (`writeProviderAccountCredentials`, `writeSecretJsonFile`),
under the per-account tree, and only **after** the credential has been
verified against the platform. Nothing is persisted for a failed add.
- Refreshed tokens are persisted, not held in a closure; a restart must not
re-prompt.
- A new source declares `authenticate(session)`, not `authFlow`. The old
callback shape still runs, so an existing provider is not broken, but it
cannot ask twice, cannot carry its own instructions, and cannot say whether
a challenge expects an answer.
- Every challenge carries the words that go with it — a `title`, and
`instructions` wherever the operator has to do something outside Omnesis.
A client composing that sentence would need a branch per source, which is
the thing source encapsulation forbids. Grep the portal and CLI for the
platform's name; finding it is the bug.
- `show` for a challenge nobody answers through the host (a pairing code, a
redirect this machine catches on loopback); `ask` for one that comes back
(a gateway-caught redirect, a pasted code, fields, a widget result).
- `canShow(kind)` gates the selected interaction. Undeclared client capabilities
mean legacy redirect/QR display, not support for every challenge. Unsupported
`ask` fails immediately; shown challenges, nonpending questions and redirects
or widgets cannot be answered through the generic answer endpoint. Round-trip
every produced challenge, including an `elsewhere` redirect, through the
subprocess parser and gateway route.
- Field requirements come from their declarations. Validate supplied values as
well as later answers; repeated invalid answers report `credential-rejected`.
Re-authentication must resolve `session.accountId`, never silently persist a
different account. Cancel, timeout and retry information survives the wire.
- Values the operator supplies are asked for, not pre-collected. A
re-authentication whose stored credential is the thing that stopped working
must be able to ask for a new one.
- Failures are `AuthFailure` with a code. In particular a refusal and an
unreachable platform are separate arms: one never clears by retrying, the
other may clear on its own.
- `credentialState()` answers from the stored credential, never from a live
request; a network blip must not park the source in `needs-auth` and fire
the reminder ladder. Report `unknown` when the credential cannot be read —
a locked keyring is not a missing credential — and `connected` with an
`expiresAt` when the platform names a deadline, rather than flipping to
broken on the day it passes. `isAuthenticated()` is the deprecated form.
- `discover()` and `cleanupCredentials()` use `ctx.configDir`, and
`authenticate()` uses `session.host.configDir`;
`create()` uses `host.configDir` (or `host.stateDir` for per-account
state). Never re-derive the default
config directory — an isolated test instance would otherwise read or wipe
the operator's live credentials.
- No token, key, or secret in logs, error messages, or document metadata.
- A throw from `create()`, `discover()`, or `createContext()` is the message
the operator reads as the reason the source was not added: it names the
condition and the fix.
## 5. Sync runtime
- **Cursor.** Opaque, JSON-serialisable, with explicit state evolution (§ 3.1).
Shared handoff cursors must be portable across hosts; member-local file maps
may name that member's own paths. Never mutate a decoded prior cursor before
the page commits. A failure must leave the installed checkpoint usable.
- **Scoped host.** Prefer `CreateOptions.host`, not the collector's gateway
client or global configuration. Analytics reads stay within declared tables;
writes stay on pages. `stateDir` is account-scoped and shared by sibling sources:
use distinct filenames and do not mistake a host-created directory for a
credential. Test removal of the last account with empty state directories.
- **Local probes.** `probeReadAccess` and `probeLocalStores` are fresh read-only
observations honoring cancellation, not a sync, repair, migration or cached
success. Store reports distinguish encrypted/plaintext/absent/locked/
unverifiable without paths, filenames, raw errors or indexed content.
- **Paging.** `syncPage` / `emptySync`; `hasMore` terminates. Nothing caps
page count except the wall-clock timeout, so an always-true `hasMore` burns
an hour per run.
- **Abort.** `opts.signal` reaches every upstream call, in `sync()` and in
`syncStructured()` alike. The § 0 grep shows whether the source reads it
at all.
- **Data cutoff.** `applyDataCutoff` with `CreateOptions.dataCutoff`.
- **Lifecycle.** `dispose()` closes handles. A source holding a live
connection implements `suspend()` / `resume()`; without them a disabled
push source keeps reconnecting and a re-enable stacks a second socket.
- **Resync.** `onResync()` re-marks the source's whole local store for
- emission, so a resync rebuilds the corpus in full rather than from recent
- changes, and fences a late page from a run that timed out.
+ emission, so a resync reads the corpus in full rather than from recent
+ changes, and fences a late page from a run that timed out. It runs for both
+ resync modes: after a from-scratch resync the gateway holds nothing, after
+ an in-place one it holds every record, so the source re-emits each item
+ under the external id it had and must not assume anything was wiped.
- **Change-triggered sync.** `watchPaths` names the database and its `-wal`
sibling; `watchDirectoryPaths` for directories that may be absent at boot.
- **Freshness.** `freshness` is declared only when `sync()` is how data
arrives; on a push or analytics-only source it produces a false stall
warning.
- **Errors.** Failures at the sync boundary are typed `SyncError`s with
`retryable`, `retryAfterMs` for rate limits, and a `SyncRemediation` for
anything the operator must fix by hand (a permission grant, a re-login).
Untyped throws fall through substring matching into `unknown` with no
remedy.
- **Failure scope.** One bad item is skipped and logged; only a transient
failure re-throws the page. A page must never advance the cursor past
input it did not durably emit, and never fail the whole page for one item.
- **Rate limits.** The platform's limits are honoured (`retryAfterMs`,
`defaultSyncInterval`, the shared rate limiter), and the client sends an
identifying User-Agent where the platform asks for one. A rate-limit
`SyncError` names `quota: { kind }` when the platform says what it counted
against: `app` only when every account signs in through one application
credential on the collector (no `perAccount` credentials spec), otherwise
`account`.
- **Writer cost.** A source that emits a burst (a full bootstrap, a resync)
is processed by a single writer thread on the gateway. Page sizes and the
amount of metadata, links, and people per document decide how long that
writer is held; name the consumers whose cost model changes if this source
emits far more of anything per item than its peers.
## 6. Deletions and snapshots
This section owns the deletion contract; § 13 and § 15 refer back to it.
- A source that **knows** an item was deleted names it once, not on every
sync. On the analytics side that lives on the table write, so a source
deleting from several tables says so per table, and each deletion is a
`deletedKeys` record over the columns the TABLE declares as its
`deleteKey` (defaulting to its primary key). Check the declaration, not the
page: a key coarser than the primary key means "every row of this group goes
together", which is right for a table re-read per parent and wrong for one
whose rows are deleted individually.
- A source with no deletion signal sends a **snapshot** (`presentExternalIds`
for documents, `presentKeys` per table write) only on the final page
(`hasMore: false`), assembled with
`SnapshotEnumeration` so a partition it could not read is withheld rather
than reported absent. The gateway treats an omission as a claim to be
corroborated over time, but a snapshot built from a partial read still
starts a deletion clock on items that exist.
- A source with several backing stores may narrow the assertion instead of
withholding it: `presentClaims` (from `SnapshotEnumeration.claims()`) names
each store read in full and what it holds, and every document carries the
`partitionKey` of the store it came from. Check both halves — a claim names
a partition, and a document belongs to one only because the source stamped
it, so a claim on a partition whose documents carry no key asks for
deletions that silently never happen. `presentExternalIds` and
`presentClaims` on one page is refused, not resolved: a complete read
vouches for the whole source, and claims are the degraded cycle's form.
- Partition keys are deterministic and bounded to 256 characters. Prove the
upgrade re-stamps unchanged documents, and state how legacy unpartitioned
documents or vanished partitions eventually reconcile. An unreadable file
keeps last-known IDs while readable siblings may delete; an unknown file or
root stays gapped no matter how many cycles it fails. Test removal/repointing
of configured scope separately from an unavailable mount and cross-root moves.
- Every file emitting a snapshot has a row in
`packages/source-sdk/src/snapshot-emitters.json` whose sentence honestly
says how completeness is guaranteed; the snapshot-contract test reddens
otherwise. Read the sentence and check it against the code — the test
cannot.
- On a replicated source, deletions are reconciled by the lease holder; the
source still emits tombstones and snapshots regardless of the lease.
## 7. Documents and analytics rows
- `externalId` is stable across syncs and across restarts; aggregates use a
composite key. `contentHash` via the shared helper. `sourceCreatedAt` /
`sourceUpdatedAt` are UTC ISO 8601, and `sourceUpdatedAt` moves when the
item is edited upstream — a source that copies the creation time into it
never re-emits an edit. Wall-clock times from the platform are converted
with the shared time-zone helpers, never assumed local.
- Content is markdown (HTML through `htmlToMarkdown`), bounded in size, with
the searchable text first. No secrets or credentials in content or
metadata.
- Relations use `EdgeDeclaration` (thread, parent, attachment, cited
document), not ad-hoc metadata keys.
- Attachments go through `resolveAttachmentConfig(sourceConfig, {
includeAudioTypes })` and the injected `extractAttachment`; no bespoke
allow-list.
- **Analytics reads.** The `analytics.query` handle reaches only the tables
this source declares — its own, plus any shared with a sibling of the same
type — and only as a single SELECT. A source that reads a table it does not
declare is refused by the gateway, so a source relying on another's rows is
a design problem to raise, not a query to fix. The gateway checks which
tables a query names, not which rows it returns, so a table written by
several accounts answers with every account's rows: every read that picks
rows to fetch or write back filters on this account's value of the table's
`sharedDiscriminatorColumn`; a lookup by ids only this account can hold —
drawn from its own rows, or from its own upstream listing where the
platform never reuses an id across accounts — needs no second filter. A
sibling's row enriched and written back is refused, and the refused page
replays on every retry.
- **Table writes.** A page names every table it fills, in the order the host
should write them (`analytics`, one `TableWrite` or a list). A source whose
upstream record fans out writes its children on the same page as their
parent, so one cursor covers all of them — it does not paginate a phase per
table, and there is no way to write outside the page at all. A table named
twice is two writes in order, which is how a keyed set is replaced: the
clear, then the rows. One write carrying both deletes its own rows, because
the host stores a write's rows before it applies its deletions. A page with
nothing to write omits the field rather than naming a table with no rows.
- **One page, or two.** Rows go on one page when a checkpoint between them
would be a lie — a parent and the children it fans out into. Rows that are
merely in hand at the same moment belong on separate pages: each checkpoints,
so a later failure leaves the earlier write standing. A source that writes
its account list before walking each account over the network is doing this
correctly, not working around the contract.
- **Analytics schemas** (structured and hybrid sources): non-empty
`primaryKey`; no `_stream_id` column (the gateway appends it);
`semanticTimeColumn` is a `DATE` or `TIMESTAMPTZ` column or explicitly
`null`, never omitted and never a zone-less `TIMESTAMP`; a `record` spec
with `titleColumns` / `keyColumns` so rows can be cited; `volatile: true`
on fetch stamps and digests so a watch does not fire on every page;
categorical columns carry `description` and `allowedValues` /
`canonicalValues`; `boundDocument` keys equal the primary key as a set;
`sharedDiscriminatorColumn` when sibling sources share a table, or one
source's removal drops the sibling's rows. Fixed schemas evolve
additively; `dynamicColumns: true` only when upstream users can reshape
the table.
## 8. People
- Every document with a human counterpart carries `metadata.people` as
`PersonMention[]`. Without it the source is invisible to person search,
person pages, catch-up, and Watch person predicates. Sources with no human
counterpart legitimately emit none, but a self-authored corpus still emits
`{ role: "author", isSelf: true }`.
- Roles come only from `PERSON_ROLES` (author, sender, recipient,
participant, attendee, mentioned, contact, owner). Map platform roles onto
them; an invented role is stored but scores nothing, so the person never
gains an interaction score. `mentioned` and `contact` are neutral: a
source that emits only those, or only recipients without self as sender,
contributes nothing to the people ranking.
- Identifiers are normalised **by the source**: emails lower-cased, phones
E.164 via `normalizePhone` with the region hint from
`ingestionContext.phoneRegion`, platform handles in `lids` namespaced by
source (`github:<login>`). The gateway does not normalise phones or LIDs;
an un-namespaced handle can fuse two platforms' users.
- Self, one of two designs: a source whose account id **is** a platform
identity (a login, an athlete id, a phone number) emits the account's own
identifier as a normal mention and declares `selfIdentity` on the
descriptor so the gateway resolves it to the self person; a source whose
documents are inherently the user's own (notes, tasks) emits
`isSelf: true`. An `isSelf` mention attaches **no** aliases, so it cannot
be used to add identifiers to self, and a platform-id source that relies
on it instead of `selfIdentity` leaves its own id as a duplicate person.
The `accountPattern` handles an account id carrying more than identity.
- Bots, system senders, no-reply and shared addresses are excluded before
they become people (`isNonIdentifyingEmail`, `isAutomatedSenderAddress`, a
platform-specific bot check).
- Free-text or name-only mentions (a phone number quoted in a body, a
reaction by display name) set `allowPersonCreation: false`.
- Names pass through `cleanPersonName`; never an email or phone in `name`;
no generic labels ("Support Team") as a person.
- The roles the normalizer emits are the roles the source's
`documentEventProfile` declares (§ 11 owns that check).
- Synthetic fixtures reference the universe cast (`cast.json`: `self`,
`emails`, `phones`, `lids`, `extra.<sourceKey>Id`) via the shared cast
helpers, so the new source joins the same identities as its peers instead
of forming an identity island.
## 9. URLs
- `metadata.sourceUrl` opens **that item** in the platform's web or desktop
surface, not a landing page. When no reliable per-item URL exists, omit it:
a plausible-but-wrong URL sends the user to the wrong item and the UI
cannot tell.
- `metadata.appUrl` only when a native scheme genuinely opens that item on a
phone. iOS and Android try it first and fall back to `sourceUrl` when no
installed app handles it, so a scheme that exists on only one platform is
fine; a scheme that opens the app but not the item is not, because the
launch succeeds and the fallback never runs. There is no separate
desktop-app URL field: a native scheme may be the `sourceUrl` only when it
opens the same item in the platform's desktop app; a scheme that works only
on a phone belongs in `appUrl`.
- `urlPatterns`: the external id is **capture group 1**, equal to
`externalId` case-insensitively; the resolver reads group 1 whatever
`idGroup` says, so a pattern must work with `idGroup` unset. Patterns
compile under RE2, ≤ 300 chars, ≤ 50 per source. Without patterns,
inbound links to this source's items are dropped permanently and the
reference graph never learns the edge. A test in the package extracts
`externalId` from an emitted `sourceUrl` through the declared patterns.
- `ownedWebDomains`: bare lowercase hosts the source fully covers, so the
browser extension stops capturing that site. Nothing broader.
- `urlCanonicalizer` when the item URL has several equivalent flavours; the
canonical form is for dedup and lookup only, and its hosts must not
collide with another source's rules (the gateway refuses the whole
declaration bundle). Check the generic normaliser does not strip the part
of the URL that carries identity (`?ref=`, a slash-less fragment).
- `urlHub` / `urlTargetRole` stay unset unless the source is a bag of links
(bookmarks, history, captured pages).
## 10. Icons and presentation
- `icon` is declared with `sfSymbol`, `color`, `bgColor`, and a renderable
image (`url` or `imageDataUri`) — the web portal and Android cannot draw
SF Symbols. Missing → a generic document glyph in the portal, the timeline
on both phones, and blank space in the CLI. The gateway rasterises to a
64 px PNG data URI and drops anything over its size cap silently, so the
asset must be a real SVG or PNG under the cap.
- Per-instance `icon` / `label` on the instance when the look depends on the
account (institution, browser). A source whose accounts differ that way must
also declare its **family** (`meta.family`) — the type's own name and glyph.
A family is never assembled from an account row, so a source that declares
none is shown by its raw type id wherever a client groups the corpus by
type.
- The mark's owner and strategy (hot-linked vs bundled) has a row in
`TRADEMARKS.md`; a bundled glyph from an icon set has its attribution in
`THIRD_PARTY_NOTICES.md`; `attribution.itemFooter` carries any byline the
platform's terms require.
- iOS presently maps unit nouns and human names through switches in
`ios/Sources/Omnesis/UI/SourcesView.swift`; a source absent from them
shows "docs" and its raw type id. Flag it, and prefer the generic fix over
extending the switch.
- The rendered check (portal sources page and timeline) is a § 17 item.
## 11. Watch and the ontology
- Every document-emitting source declares `documentEventProfile`
(`documentTypes`, `personRoles`, `metadataFields` with descriptions).
Without it the source syncs and searches normally but the gateway lists it
as unwatchable and every watch naming it is refused — nothing fails at
build or sync time, and no repo-wide test requires the declaration.
- The declared types, roles, and fields are what the normalizer writes,
proven by a parity test over the real sync path. A declared value nobody
emits compiles a watch that can never fire; an undeclared field is absent
from the watch journal.
- `identifiesPeople: true` on any field whose values can name a human.
- A source that rolls several messages into one document (a chat's day)
marks its message-count field `semantics: "message-count"`, and its group
flag, if it has one, `semantics: "group-conversation"`. Without them the
agent's `count_messages` counts each document as one message.
- Two sources must be distinguishable by profile alone (the catalog hides
source names).
- `documentTemporalProjections` use only the closed field set
(`$semanticTime`, `scheduledAt`, `dueAt`, `endsAt`, `timeZone`, `status`);
`temporalProjection` on analytics tables anchors on `$semanticTime` and
references declared columns of the right type.
- `semanticallyIndexed` reflects reality; a semantic-match watch on an
unindexed source is refused.
## 12. Multiple instances of the same source
The user may connect several accounts of one source (two mailboxes, two
phone numbers). Verify nothing structurally prevents it:
- Every namespace derives from `accountId` / `sourceId` / `providerId`
passed to `create()` — no module-level singletons, caches, fixed file
names, fixed ports, or fixed external-id namespaces.
- Local stores live under `configDir/<fileKey>/<accountId>/`; credentials
under the per-account tree; `cleanupCredentials(accountId)` removes only
that account.
- `discover()` returns per-account ids; `authFlow()` returns the account it
actually authenticated.
- Two instances can sync concurrently: no shared lock file, no shared
temporary path, no process-wide rate limiter that starves one account.
- Pausing or removing one account leaves its sibling untouched
(`packages/collector/src/e2e/multi-account.e2e.test.ts` is the reference).
## 13. Multi-device mode
`multiDevice` declares how several collectors may serve one account:
`exclusive` (default), `handoff`, `replicated`, `partitioned`. The mode is
pinned when the source row is created and cannot be changed later except
from `exclusive`, so it must be right at first ship.
- The chosen mode matches where the data lives. Cloud-authoritative with an
account-side cursor → `handoff` (no built-in source ships it yet; it
requires lease-aware syncing and portable cursors). The same data visible
on every device via platform sync → `replicated`, which obliges external
ids to be **identical for the same item on every device** — verify, do not
assume; host-local uuids make it `exclusive`. Each device observes a
genuinely distinct stream → `partitioned`. Picking `replicated` for
per-device streams makes two hosts overwrite each other's rows and delete
each other's documents on every snapshot.
- `partitioned`: no device or stream key inside external ids (the gateway
owns the stream); host-local parameters carry `scope: "member"`.
- `replicated`: `replicaVersionPolicy: "source-updated-at"` only if
`sourceUpdatedAt` is monotone canonical UTC; deletions follow § 6.
- The e2e suites for the mode under `packages/collector/src/e2e/`
(`partitioned-universe`, `replicated-universe`, `handoff-universe`,
`source-membership`, `replica-*-dispute`) cover the source's shape, or a
provider-specific acceptance test is added.
## 14. Phone-pushed sources
A source whose data is pushed by the iOS or Android app rather than synced
by a collector has extra obligations on both sides of the wire:
- The descriptor sets `execution: "external"` (the collector schedules no
sync; the portal hides it from Add source and inerts the sync-interval
knobs). `pushBased: true` is the deprecated spelling. `gatewayHosted` is
different: it marks a source whose data arrives at the gateway itself, as
the browser extension's pages and the Hermes and OpenClaw transcripts do.
- The type is in `DEVICE_HOSTED_SOURCE_TYPES` (`packages/types/src/device.ts`)
so the `write:<type>` scope exists; otherwise its batches get 403 and the
phone's whole push queue stalls behind them.
- Both apps declare the hosted-source contract (iOS `Source.swift`, Android
`DeviceCapabilities.kt`) with the same `multiDevice` mode as the
descriptor, and `packages/collector/src/e2e/mobile-source-modes.e2e.test.ts`
asserts it.
- Push sources register under `<type>:local`: two phones share one row
unless the mode is `partitioned` or `replicated`. Decide deliberately.
- Permission health uses `mobile-permission-health.ts` states with `impact`
and `remediation` once unhealthy; the app's activation flow requests the
permission and reports it.
- Snapshots emitted from Swift or Kotlin are not covered by the
snapshot-contract test; review their completeness by hand against § 6.
- Battery and background execution: the push runs from a background task
with bounded work per wake; a resync from a phone is refused for push-only
types (it would wipe data the phone cannot replay), so the app must be
able to re-send its full history on its own.
## 15. Touchpoints outside the package
Most of these are silent when forgotten. Check each:
| Touchpoint | Why | If forgotten |
| --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| `packages/collector/package.json` dependency on `@omnesis/provider-<name>` | the only registration step; the collector imports every `@omnesis/provider-*` it depends on | source never appears (silent) |
| `package-lock.json` entries for the package | `npm ci` in CI | install fails (loud) |
| root `tsconfig.json` `references` | `tsc --build` coverage | package never type-checked (silent) |
| package `tsconfig.json`; a `tsconfig.tests.json` only if the main one excludes `*.test.ts` | every suite must be in some typecheck program | `scripts/typecheck-lanes.test.mjs` reddens when a suite is in none (loud) |
| `package.json`: `@omnesis/provider-<name>`, `license: AGPL-3.0-or-later`, src-pointing `main`, declared runtime deps | release transform, phantom-deps test | publish breaks (loud) |
| SPDX header on every file (`scripts/add-license-header.mjs`) | license hygiene | pre-commit hook (loud) |
| `knip.json` collector `ignoreDependencies` | dep-only registration reads as unused | knip reddens (loud) |
| synth twin `packages/providers-synth/<name>/` (also in root tsconfig, knip, collector deps) | demo gateway, every synthetic e2e sweep | source missing from demos and e2e (silent) |
| `evals/universes/default/` fixtures + `universe.json` seed; `e2e-minimal` fixtures and its build script | universes | seeded-without-fixture is loud; missing entirely is silent |
| golden corpus snapshot regenerated; `synth-pipeline.e2e.test.ts` non-one-to-one list if a fixture entry yields ≠ 1 document | e2e | snapshot diff / count assertion (loud) |
| `snapshot-emitters.json` row (§ 6) | snapshot discipline | snapshot-contract test (loud) |
| `DEVICE_HOSTED_SOURCE_TYPES` (phone sources) / `packages/gateway/src/internal-source-descriptors.ts` (gateway-hosted) | § 14 | 403s / never advertised |
| `privacy/pii-allowlist.json` path-scoped entries for invented fixture identities | PII scan | commit hook (loud) |
| `website/docs/sources.html` row (or `experimental.html` while experimental); `setup.html` / `apps.html` if setup steps exist; any per-app user cap stated | public docs | nothing checks it (silent) |
| `TRADEMARKS.md`, `THIRD_PARTY_NOTICES.md` (§ 10) | marks | undocumented mark use |
| source roadmap master issue: row moves to Implemented | process | roadmap drifts |
| `CHANGELOG.md` | release notes | none |
## 16. Tests
From the § 0 test run, confirm the package's test files cover, next to the
code as `*.test.ts`:
- **Bootstrap** — first sync over a populated fixture yields the expected
documents (and rows).
- **Incremental** — a second sync with the first cursor yields nothing; one
added item yields only that item.
- **Normalization** — pure raw-row → `DocumentInput` tests over
representative fixtures, including people, URLs, timestamps, and the
edge declarations.
- **Error handling** — auth failure, permission failure, rate limit, a
malformed item, and a platform outage each surface as the right typed
error without poisoning the cursor.
- **Unchanged upstream is a no-op** — `expectUnchangedUpstreamIsNoOp` /
`runSyncCycleContract` from `@omnesis/source-sdk/testing` for any source
with more than one phase.
- **Contract parity** — the `documentEventProfile` parity test (§ 11), the
`urlPatterns` ↔ `externalId` test (§ 9), and a complete-`SourceIcon`
assertion (§ 10).
- **Multi-instance** — two `create()` calls with different account ids do
not share state.
- **Installed-state and steady-state** — migrate bare and enveloped cursors at
each pending phase, sync several unchanged or impaired cycles, and assert
preserved IDs, no duplicate rows and no false deletion evidence. Exercise
real create/update/delete/move paths and a permanently unreadable sibling.
- **Cross-boundary failures** — auth through subprocess/client parsing; hybrid
documents plus multiple table writes with failures between planes; unchanged
checkpoint until every required write succeeds, including replay.
- **Multi-device** — the mode's obligations (§ 13) where non-exclusive.
- Push sources test their push path; the app-side logic runs in the fast
native lanes.
- A spawned-gateway e2e on `SyntheticE2EHarness` when the source's synth
twin exists (the per-source suites under `packages/collector/src/e2e/`
are the pattern).
- Fixtures are invented from scratch: reserved domains, fictional phone
ranges, invented names, nothing paraphrased from anyone's real data.
- Tests use unique temp paths and ports ≥ 17601.
## 17. Manual validation to hand back to the author
Some checks cannot be automated. The report ends with a list the author
confirms before merge, each with the exact steps:
1. Add the source through the portal and through `omnesis sources add` on a
fresh install; the auth flow completes both with the browser redirect
and, where `acceptsAuthCode` is set, with a pasted code.
2. Add a **second** account of the same source while the first is syncing.
3. Open three documents from the portal's "Open in source" and confirm each
lands on the right item on desktop.
4. On iOS and on Android, tap "Open in source" on the same documents, once
with the platform's app installed and once without; confirm the item
opens or the web fallback does, never a silent no-op.
5. The icon, name, and unit noun render in the portal sources page, the
portal timeline, and the mobile timeline. When a synthetic universe
seeds the source, `scripts/shot-portal.sh sources` against an isolated
synthetic gateway captures the portal half; read the PNG.
6. Search by a person the source emitted (`from:` / a person page) returns
its documents; the person is the same record as in a sibling source.
7. Create a watch that names the source and one of its declared roles or
fields; it compiles and fires on a new item.
8. Disable, re-enable, pause, remove, and re-add the source; then resync it
from the portal; the corpus matches the platform afterwards and nothing
reconnects twice.
9. Stop the collector for longer than the platform's token lifetime and
restart it; the source resumes without re-authenticating.
10. For a non-exclusive multi-device mode, host the source from two
collectors (or two phones) and confirm the union is right and detaching
one loses nothing.
---
## Execution rules
- When delegating, sub-agents review; they do not edit. Collect the findings, de-duplicate,
and rank: **blocking** (viability, terms, encapsulation, data loss,
credential handling, wrong multi-device mode, missing registration),
**must fix before merge** (silent gaps: profile, icon, people, URLs,
snapshot discipline, tests), **should fix** (hygiene, tunables, docs
wording), **manual validation** (§ 17).
- For each finding give the file and line, the contract clause it violates
(quote the doc comment or the page section), and what a user would see.
- Fix mechanical findings directly on the branch — missing registrations,
headers, lockfile, knip, a missing profile whose values are evident from
the normalizer — and say what you changed. Leave design decisions
(mode choice, terms-of-service risk, unofficial protocol use) as findings
for the author and maintainers.
- Re-run the baseline signals from § 0 after fixes and the authoritative affected
gate for the current tree. Run additional provider E2E coverage where the
changed behavior needs it; a narrowed diagnostic is not the broader gate.
Respect repository admission for all heavy checks; a queued job is not failed.
- Finish with the ranked report, the list of files you changed, and the
manual validation list. Do not declare the source ready: that is the
author's and maintainers' call once § 17 is done.