AGENTS.md · git:20261110.9ee4336 · 2026-11-10 · sha256 bf19fb0e13110be4
AGENTS.md git:20261110.9ee4336A
Immutable. This exact content never changes and is served at /api/v1/blob/bf19fb0e13110be4.
# Agent & Contributor Guide
This file is the operational handbook for AI agents and human contributors working in this repo. For end-user docs see `README.md`; for the contributor onboarding flow see `CONTRIBUTING.md`. This file documents project-wide rules, the release flow, the plugin system, the session lifecycle, the schema migration framework, and the beta dogfood mechanics.
> **State as of 0.9.x:** total-recall is a .NET 8 NativeAOT plugin (C# imperative shell + F# functional core). **Shell: PowerShell (win32) — use PowerShell syntax for all bash commands.** The TypeScript implementation that lived in `src/` through 0.7.x was stripped during the 0.7.2 → 0.8.0 cutover. Anything in this file that mentions `dist/`, `bun`, `tsup`, `vitest`, `publish.yml`, `bin/start.cjs`, or `src/db/schema.ts` is either gone or renamed — see the strip series in `CHANGELOG.md` (commits `87975a7` → `7a8c437`).
---
## Quick Reference
### Project Overview
.NET 8 NativeAOT MCP server plugin (C# imperative shell + F# functional core) with npm packaging for Claude Code / Copilot CLI / Cursor / Kiro / OpenCode / Codex CLI marketplace distribution. Three-tier memory (Hot/Warm/Cold with a **sticky** flag — new entries default to warm and *earn* promotion to hot by access; sticky pins, set via `memory_pin`/`memory_unpin`, are always injected first and immune to decay/compaction) + hierarchical KB, all local by default (SQLite + sqlite-vec + bundled ONNX), optionally synced to Cortex (sticky pins are local-only, never synced). The single binary also exposes a third surface: a built-in local web UI (`total-recall ui`) serving an embedded React SPA with six sections (Dashboard, Memory, Knowledge Base, Usage, Insights, Config) via `TotalRecall.Web`.
### Layer Diagram
```
TotalRecall.Host (C#) ← AOT entry point + composition root
├── TotalRecall.Server (C#) ← MCP JSON-RPC over stdio; 49 handlers (one file each)
├── TotalRecall.Web (C#) ← embedded minimal-API + React SPA (web UI), dispatches to ToolRegistry
├── TotalRecall.Cli (C#) ← CLI commands (Spectre.Console), including `total-recall ui`
├── TotalRecall.Infrastructure (C#) ← SQLite/Postgres, ONNX embedder, importers, migrations
└── TotalRecall.Core (F#) ← Pure functions: tokenizer, decay, ranking, parsers, chunker
npm wrapper layer (zero-dep Node):
bin/start.js (launcher) ← dispatches on argv: `serve` (or no args) runs the MCP
bootstrap shim (IS the MCP server from the harness's view; comes up instantly;
provisions [sha256-verified via release provisioning.manifest.json] + spawns +
proxies the NativeAOT engine; supervises/restarts on crash; never drops the MCP
connection). Every OTHER subcommand (`ui`, `reindex-embeddings`, `config`,
`dump-catalog`, …) is run via bin/shim/direct.js — provision then spawn the engine
directly with inherited stdio (NO shim/handshake), because those emit human-readable
stdout, not MCP JSON-RPC. Shim modules: bin/shim/{jsonrpc-stdio,state,server-surface,
provisioner,engine-proxy,orchestrator,direct}.js. catalog.json (repo root) seeds
tools/list before the engine is ready; generated by
`dotnet run --project src/TotalRecall.Host -- dump-catalog`.
scripts/fetch-binary.js ← download helper (shared by postinstall + provisioner)
scripts/postinstall.js, scripts/verify-binaries.js
pi/ ← pi harness extension (`pi install`; `package.json` `pi` block): extension.js (MCP
registration, session-start text, floor hook), floor.js (spawns the provisioned engine), userServer.js (follows a hand-configured
mcp.json server's TOTAL_RECALL_* env so the floor reads the same store),
command.js (`/total-recall`). Plain ESM, no build; tests `pi/*.test.js` run in `test:shim`.
codex/ + .codex-plugin/ ← Codex CLI host (`codex plugin add`): plugin.json overlay, hooks.json + hook.js (node
SessionStart/UserPromptSubmit hooks), bootstrap.cjs → mcp.json (generated by build-mcp.js), manifest.test.js
(also the seven-file version-sync guard). Reuses pi/floor.js. Tests run in `test:shim`.
```
> **BuildSpa is opt-in.** `TotalRecall.Web` compiles the React SPA only when `-p:BuildSpa=true` is passed (e.g. in release publish). Default `dotnet build` and all tests are Node-free — the binary embeds a placeholder page instead. See `src/TotalRecall.Web/AGENTS.md` for the full build pipeline.
### WHERE TO LOOK
| Task | Location |
|------|----------|
| Add/modify MCP tool | `src/TotalRecall.Server/Handlers/<ToolName>Handler.cs` |
| Handler wiring | `src/TotalRecall.Server/ServerComposition.cs` → `BuildRegistry()` |
| Pinned-block rendering (Server + CLI) | `src/TotalRecall.Infrastructure/Memory/PinnedBlockRenderer.cs` |
| Pinned floor (throttle / state / hook backend) | `src/TotalRecall.Infrastructure/Memory/PinnedFloorDecider.cs`, `PinnedFloorState.cs`; `src/TotalRecall.Cli/Commands/PinnedFloorCommand.cs` |
| pi harness support (extension, floor runner, `/total-recall`) | `pi/extension.js`, `pi/floor.js`, `pi/command.js`; manifest = `package.json` `pi` block |
| Codex CLI host (hooks, MCP launcher, manifest) | `codex/hook.js`, `codex/hooks.json`, `codex/bootstrap.cjs` → `codex/mcp.json` (`node codex/build-mcp.js`), `.codex-plugin/plugin.json` |
| Project-scoped pin injection | `src/TotalRecall.Infrastructure/Memory/ProjectResolver.cs`, `ProjectKey.cs`, `PinnedScope.cs` |
| Handler interface | `src/TotalRecall.Server/IToolHandler.cs` |
| Wire-format DTOs | `src/TotalRecall.Server/JsonContext.cs` |
| Schema migration | `src/TotalRecall.Infrastructure/Storage/Schema.cs` → migrations array |
| Storage interfaces | `src/TotalRecall.Infrastructure/Storage/` (`IStore`, `IVectorSearch`, etc.) |
| Embedding (ONNX) | `src/TotalRecall.Infrastructure/Embedding/` |
| Importers (host tools) | `src/TotalRecall.Infrastructure/Importers/` |
| Cortex sync | `src/TotalRecall.Infrastructure/Sync/` |
| Core domain types | `src/TotalRecall.Core/Types.fs` |
| Pure logic (decay, ranking) | `src/TotalRecall.Core/*.fs` |
| Migration guard | `src/TotalRecall.Server/AutoMigrationGuard.cs` |
| Error handling | `src/TotalRecall.Server/ErrorTranslator.cs` |
| Exception logging | `src/TotalRecall.Infrastructure/Diagnostics/ExceptionLogger.cs` |
| AOT entry point | `src/TotalRecall.Host/Program.cs` |
| Benchmark queries | `eval/benchmarks/smoke.jsonl` (22), `eval/benchmarks/retrieval.jsonl` (139) |
| Plugin manifests | `.claude-plugin/plugin.json`, `.copilot-plugin/plugin.json`, `.cursor-plugin/plugin.json`, `.codex-plugin/plugin.json` |
### Key Commands (PowerShell — this repo runs on win32)
```powershell
# Build
dotnet build src/TotalRecall.sln
# Test (all, skip integration)
dotnet test src/TotalRecall.sln --filter "Category!=Integration"
# Test (single project)
dotnet test tests/TotalRecall.Server.Tests/TotalRecall.Server.Tests.csproj
# AOT publish (local RID)
dotnet publish src/TotalRecall.Host/TotalRecall.Host.csproj -c Release -r win-x64 -p:PublishAot=true
# npm (pulls sqlite-vec native libs needed by csproj copy targets)
npm ci
# Shim unit tests (Node, independent of .NET)
npm run test:shim
# Verify binary present
total-recall status
```
### Supported RIDs
`linux-x64`, `linux-arm64`, `osx-arm64`, `win-x64` — Intel Mac (`osx-x64`) not shipped.
---
## Build & Release
### .NET 8 NativeAOT, no `dist/`
The build artifact is a per-platform AOT-published binary at `src/TotalRecall.Host/bin/Release/net8.0/<rid>/publish/total-recall` (Unix) or `total-recall.exe` (Windows). There is no `dist/` directory, no `tsup` build, no `npm run build` script. CI publishes binaries via `.github/workflows/release.yml` and ships them inside the npm tarball under `binaries/<rid>/`.
The .NET SDK is pinned by `global.json` at the repo root (`{"sdk":{"version":"10.0.100","rollForward":"latestFeature"}}`). The pin exists because GitHub-hosted macOS runners ship .NET 10 preview pre-installed and we need every CI matrix leg to use the same SDK regardless of runner pre-installs. .NET 10 SDK builds the `net8.0` target framework cleanly.
### ONNX model is fetched at build time (not committed)
The embedding model (`models/bge-small-en-v1.5/model.onnx`, ~133 MB fp32) is **not** committed to the repo and is not in Git LFS. Contributors run `sh scripts/fetch-bge-small.sh` once to fetch + sha256-verify it from a pinned HuggingFace revision into `models/bge-small-en-v1.5/`. Release builds run the same script and bundle the verified model into the per-RID release tarball, so plugin users get offline embeddings with no runtime HuggingFace download. At runtime the .NET `ModelManager` validates the bundled model and throws a clear error if it is absent — it does NOT download (see `src/TotalRecall.Infrastructure/Embedding/ModelManager.cs`).
After swapping the local embedder, the startup guard detects the embedder-fingerprint mismatch and acts per `embedding.on_model_change` (default `auto`): on **sqlite**/**cortex** it schedules a **background, batched, resumable** re-embed (an in-process worker started by `ServerComposition`, never on the MCP boot path — a synchronous re-embed of a large DB would blow the stdio startup timeout into a boot loop) and keeps serving with degraded local retrieval until it finishes; progress + degraded state surface via `session_start`/`status` (`backgroundTasks.reindex`). `warn` runs degraded without re-stamping; `block` is the legacy fail-fast refusal. On **postgres**, `auto` raises an explicit "not supported" error (re-ingest fresh / use `warn`). The `total-recall reindex-embeddings` command runs the same coordinator offline (foreground, also resumable) and is still needed for `warn`/`block` users and manual re-embeds. See `EmbedderMigration` (returns a decision) → `ReindexCoordinator` → `EmbeddingReindexer.RunBatched`.
### Version sync — seven files, one version (STANDING RULE)
total-recall is a multi-host plugin (Claude Code, Copilot CLI, Cursor, OpenCode, Codex CLI, Hermes, …). Each host reads its own plugin manifest, and every manifest carries its own `version` field. They MUST all match the `package.json` version on every release. Historical drift incidents:
- `.copilot-plugin/plugin.json` was stuck on `0.1.0` for many releases — Copilot CLI users saw `0.1.0` reported even when npm was at 0.7.2
- `.claude-plugin/plugin.json` was stuck on `0.7.2` through the entire TS→.NET cutover (beta.1 → beta.3) until the build agent caught it during the beta.4 audit
- `hermes-plugin/plugin.yaml` was stuck on `1.4.0` through the 2.0.0 release — the v2.0.0 sync repaired the plugin.json drift but missed the one YAML manifest (caught during the 2.1.0 release)
**On every release you MUST bump the version in ALL of these to the same value:**
1. `package.json`
2. `package-lock.json` (top-level `version` field AND `packages[""].version` — npm keeps both in sync; the safest edit is a `replace_all` of the old version string)
3. `.claude-plugin/plugin.json`
4. `.copilot-plugin/plugin.json`
5. `.cursor-plugin/plugin.json`
6. `.codex-plugin/plugin.json` (Codex ignores `.claude-plugin/plugin.json` once this file exists and does not fall back to its version)
7. `hermes-plugin/plugin.yaml` (the one non-JSON manifest — a plain-text grep for the old version string is the only way it shows up)
`.opencode/` uses `INSTALL.md` (no versioned manifest) so it is exempt, but any version references in that doc should still be reviewed.
**Version surfaces that are NOT bumped in-repo** (do not "fix" these during a release):
- The AOT binary's `--version` output and the MCP `serverInfo.version` are stamped at release time by `.github/workflows/release.yml` from the git tag (`-p:Version` / `-p:InformationalVersion`). The repo contains no hardcoded value for them — tagging `vX.Y.Z` is the bump.
- `hermes-plugin/mcp_client.py` `clientInfo` (`hermes-mcp-client 0.1.0`) identifies the MCP client implementation, not the plugin; `protocolVersion` is the MCP spec date.
- `global.json` pins the .NET SDK; `Version="..."` attributes in `.csproj` files are NuGet package references.
The fastest audit is a repo-wide grep for the OLD version string after bumping — anything still matching (outside `CHANGELOG.md` history and the exclusions above) was missed.
When agents dispatch subagents to bump versions or cut releases, this list MUST be included in the prompt. Never assume "I'll just bump package.json" — every release must sync all seven.
`codex/manifest.test.js` (run by CI's shim-tests job) now fails when `package.json`, both `package-lock.json` fields, the four plugin manifests and `hermes-plugin/plugin.yaml` disagree. A follow-up in `docs/TODO.md` ("Plugin Version Single Source of Truth") tracks making the sync automatic rather than checked.
### Release flow
1. Bump version in all seven files above to the same value.
2. Update `CHANGELOG.md` with the new version's `### Fixed` / `### Added` / `### Changed` sections.
3. Commit with a message like `release(beta.N): bump to 0.x.y-beta.N; …` or `release: v0.x.y; …`.
4. Tag with `git tag -a vX.Y.Z -m "..."` (annotated tag with a release-note body in the tag message — `gh release` displays it).
5. Push the branch first (`git push origin rewrite/dotnet`), wait for `.github/workflows/dotnet-ci.yml` to go green.
6. Only then push the tag (`git push origin vX.Y.Z`), which fires `.github/workflows/release.yml`. The 4-job matrix builds AOT binaries for `linux-x64`, `linux-arm64`, `osx-arm64`, and `win-x64`, stages them in `binaries/<rid>/`, then the publish job downloads the four artifacts, runs `prepublishOnly` (`scripts/verify-binaries.js`), `npm publish`es with the right dist-tag (`beta`/`rc`/`latest` resolved from the version string by inline shell logic), and attaches per-RID `.tar.gz` archives to a GitHub Release.
7. Verify the publish landed: `npm view @strvmarv/total-recall dist-tags`, then `gh release view vX.Y.Z --json assets`.
The single CI workflow that runs on every push/PR is `.github/workflows/dotnet-ci.yml`. The release workflow only runs on `v*` tag pushes. There is no `publish.yml` — that was the legacy TS publish workflow and was deleted in commit `7a8c437`.
### Code-signing (not yet shipped)
Windows binaries are not yet Authenticode-signed. Defender's mid-extract scan can hold file handles long enough that npm's temp-then-rename install path fails with `EPERM` on Windows hosts (see "Beta dogfood mechanics" below for the workaround). Authenticode signing is tracked in `docs/TODO.md`.
---
## Plugin System
### How marketplace installs work
The marketplace is a separate git repo (`strvmarv/total-recall-marketplace`) containing a `marketplace.json` that lists plugins and their sources. Claude Code clones the marketplace repo into `~/.claude/plugins/marketplaces/<name>/`, reads the `marketplace.json`, and resolves each plugin entry's `source` field to fetch the plugin content. **Source types supported:**
- `source: github` (or `source: url`) → Claude Code does `git clone` into `~/.claude/plugins/cache/<marketplace>/<plugin>/<version>/`. No npm install. No postinstall hooks. Pure file-tree fetch.
- `source: npm` → Claude Code does `npm install <package>@<version>` into a temp dir, then renames the temp dir into the cache. Triggers `postinstall` lifecycle hooks. Hits Windows Defender mid-extract `EPERM` rename failures more often than the github path.
Both paths converge on `bin/start.js` because `.mcp.json` always invokes `node ${CLAUDE_PLUGIN_ROOT}/bin/start.js`. The shim is itself the MCP server: it comes up instantly, answers `initialize`/`ping`/`tools/list` (serving `catalog.json` before the engine is ready), provisions (sha256-verified via `provisioning.manifest.json`) and spawns the engine, then proxies all traffic. The MCP connection never drops.
### `.mcp.json` invocation
```json
{
"mcpServers": {
"total-recall": {
"command": "node",
"args": ["${CLAUDE_PLUGIN_ROOT}/bin/start.js"],
"cwd": "${CLAUDE_PLUGIN_ROOT}"
}
}
}
```
`bin/start.js` is zero-dep Node. For the `serve` invocation (what `.mcp.json` runs) it acts as the MCP server from the harness's perspective — non-serve subcommands like `total-recall ui` bypass the shim entirely (see the launcher dispatch above). It comes up instantly and handles provisioning in-process:
1. **Engine binary present and sha256-verified** (`.verified.json` marker exists) → shim spawns the engine and begins proxying immediately. The common, instant path.
2. **Binary missing or unverified** (first launch after a plugin update on the git-clone path) → shim answers `initialize`/`ping`/`tools/list` immediately, returns a structured `{ "status": "not_ready", "phase": "provisioning", ... }` result for any `tools/call`, and provisions in the background (downloads `total-recall-<rid>.tar.gz` from GitHub Releases, sha256-verifies against the release `provisioning.manifest.json`, extracts, writes `.verified.json`). Once ready the engine is spawned, the shim begins proxying, and it emits `notifications/tools/list_changed`. **The MCP connection never drops** — the harness retries tool calls naturally and self-heals without user intervention.
`ensureBinary()` (shared with `scripts/postinstall.js`, where blocking is fine — install time has no MCP timeout) extracts via `tarExecutable()`: on Windows it pins to `%SystemRoot%\System32\tar.exe` (bsdtar), because a GNU `tar` earlier on PATH (Git for Windows / msys) treats the `C:\...` destination as a remote host and fails ("Cannot connect to C:").
This background-fetch path exists because the **git-clone install path** fetches the source tree without `binaries/` (we never commit prebuilt binaries to git — the npm tarball ships them, but git-source installs don't go through npm). When a Claude Code marketplace entry uses `source: github`, the installed tree has `bin/start.js` and `scripts/fetch-binary.js` but no `binaries/`, so the background fetch provisions it on first launch.
### Codex CLI host
Codex installs the plugin from the same marketplace as Claude Code, then prefers `.codex-plugin/plugin.json` over `.claude-plugin/plugin.json`: its `hooks` path replaces `hooks/hooks.json` and its `mcpServers` path replaces the root `.mcp.json`, so Codex gets `codex/hooks.json` and `codex/mcp.json` while every other host keeps its files. Rules learned the hard way (Codex 0.162.1, Windows): (1) hook commands run through PowerShell, so the bash `run.sh` hooks cannot be reused, and only the **braced** `${PLUGIN_ROOT}` is substituted by Codex (an unbraced `$NAME` reaches the shell unexpanded); (2) a bundled MCP server's command, args and env get **no** variable substitution and a sanitized environment (19 variables on Windows), and a relative `cwd` resolves to the plugin root — so `codex/mcp.json` is generated from `codex/bootstrap.cjs` (`node codex/build-mcp.js`; `--check` is tested), which finds the newest installed `total-recall` under `$CODEX_HOME || ~/.codex` `/plugins/cache/` and imports its `bin/start.js` with **no** `cwd` (the engine resolves project scope from the working directory it inherits); (3) per Codex's docs (not independently observed) hook output is capped at ~2,500 tokens unless the handler sets `additionalContextLimit` (6000 for the instructions, 16000 for the floor; the 19 KB instructions were observed delivered unspilled with the key set); (4) per Codex's docs (the interactive flow was not observed) plugin hooks need a one-time `/hooks` trust review; (5) `codex exec` cannot answer MCP approval prompts, so users need `default_tools_approval_mode = "approve"` for the server (documented, not shipped). The Codex manifest carries its own `version` (the seventh release-sync file). `SessionEnd` is deliberately not mapped (3 s clamp, no user-visible output).
### Removing a plugin install (for testing)
To fully uninstall for a clean reinstall test, remove all of:
- `~/.claude/plugins/cache/strvmarv-total-recall-marketplace/total-recall/<version>/` (per-version cache for github-source installs)
- `~/.claude/plugins/cache/total-recall/` (flat cache layout for npm-source installs — added in 0.8.x by Claude Code)
- `~/.claude/plugins/marketplaces/strvmarv-total-recall-marketplace/` (marketplace metadata cache, may not exist depending on Claude Code version)
- Entries in `~/.claude/settings.json`: `enabledPlugins["total-recall@..."]` and `extraKnownMarketplaces["strvmarv-..."]`
- `~/.total-recall/` (the SQLite database, models cache, config — only delete this if you want a fresh state, not a fresh plugin install)
---
## Beta dogfood mechanics
Beta tags (`v0.8.0-beta.N`) are published to the npm `@beta` dist-tag, which is **separate from `@latest`** (currently still on `0.7.2` TypeScript through the cutover). Public users who run `/plugin update total-recall` against the upstream marketplace get whatever the upstream marketplace's `marketplace.json` resolves to — and **the upstream marketplace is never pointed at a beta tag**, so public users never get a beta unless they explicitly opt in.
Beta dogfood is per-tester via a **local marketplace override**. Each tester clones the marketplace repo locally, edits the local clone's `marketplace.json` to pin the beta ref, registers the local clone as a marketplace in their personal `~/.claude/settings.json`, and `/plugin update`s. Public users are unaffected.
### Setup (per tester, one-time per machine)
```bash
# 1. Clone the marketplace repo
git clone https://github.com/strvmarv/total-recall-marketplace.git ~/dev/total-recall-marketplace
cd ~/dev/total-recall-marketplace
# 2. Edit .claude-plugin/marketplace.json — replace the total-recall entry's
# "source" block with the github source pinned to the beta ref:
#
# "source": {
# "source": "github",
# "repo": "strvmarv/total-recall",
# "ref": "v0.8.0-beta.7"
# }
#
# Why github source (not npm source): the npm install path on Windows
# trips EPERM on the temp-dir rename when Defender holds a handle on a
# freshly-extracted .exe inside the temp dir. The github source is a
# pure git clone with no rename window, and bin/start.js downloads the
# binary on first launch via fetch-binary.js — the download path doesn't
# trigger the same Defender lock.
# 3. Register the local clone in ~/.claude/settings.json. Find the
# "extraKnownMarketplaces" entry for strvmarv-total-recall-marketplace
# and replace its "source" block with a path source:
#
# "extraKnownMarketplaces": {
# "strvmarv-total-recall-marketplace": {
# "source": { "source": "path", "path": "/home/<you>/dev/total-recall-marketplace" }
# }
# }
#
# Keep the existing "enabledPlugins" entry unchanged.
# 4. Fully quit and restart Claude Code so it re-reads settings.json.
# 5. In a fresh CC session: /plugin update total-recall
# 6. Verify in a terminal:
total-recall --version
# Expected: total-recall 0.8.0-beta.<N> (NOT 0.7.2 or 0.1.0)
total-recall status
# Expected: clean tier output, no DllNotFoundException, no TypeInitializationException
```
### Bumping to a new beta
```bash
cd ~/dev/total-recall-marketplace
# Edit .claude-plugin/marketplace.json — bump the "ref" value to the new tag.
# In Claude Code: /plugin update total-recall
# (No need to touch settings.json again — the path source picks up
# marketplace.json edits on every plugin operation.)
```
### Reverting to public stable
Restore the original `extraKnownMarketplaces` entry's source block to:
```json
"source": { "source": "github", "repo": "strvmarv/total-recall-marketplace" }
```
Restart Claude Code, `/plugin update total-recall`. You're back on stable.
### Windows EPERM workaround
If `/plugin update` fails on Windows with `EPERM: operation not permitted, rename '...temp_npm_...' -> '...total-recall'`, the install actually succeeded — only the final rename failed. Recovery from a WSL prompt:
```bash
# Find the leftover temp dir
ls /mnt/c/Users/<you>/.claude/plugins/cache/temp_npm_*
# Rename it manually (WSL bypasses the Win32 file lock)
mv /mnt/c/Users/<you>/.claude/plugins/cache/temp_npm_<id> \
/mnt/c/Users/<you>/.claude/plugins/cache/total-recall
# Restart Claude Code
```
The root cause is Windows Defender mid-scanning the freshly-extracted `total-recall.exe` at the moment of rename. Switching the marketplace to `source: github` avoids the issue entirely (the github path doesn't have the rename window). Tracked in `docs/TODO.md`.
---
## Session Lifecycle
### What happens on `session_start`
1. **Migration guard** — `AutoMigrationGuard` (`src/TotalRecall.Server/AutoMigrationGuard.cs`) inspects the database file in read-only mode (`InspectDbFormat`) and dispatches via a 5-state state machine: `NotPresent`, `EmptyFile`, `TsFormat`, `PartialNetEmpty`, `PartialNetPopulated`, `NetMigrated`. Handles the partial-state cliff where both `total-recall.db` and `total-recall.db.ts-backup` exist by sidelining the suspect file to `<dbPath>.failed-migration-<utc>` (never deletes anything).
2. **Import sync** — scans Claude Code, Copilot CLI, Cursor, Cline, OpenCode, and Hermes memory dirs via the `IImporter` collection in `src/TotalRecall.Infrastructure/Importers/`. Deduplicates via content hash in `import_log`.
3. **Warm sweep** — if last sweep was more than `warm_sweep_interval_days` ago, moves old unaccessed warm entries to cold. The sweep also runs earned **warm→hot promotion** (an entry with `access_count >= compaction.promote_min_access` AND `decay_score >= compaction.promote_threshold`); sticky entries are excluded from eviction. Tracked via `compaction_log` with `reason = 'warm_sweep_decay'`.
4. **Project docs auto-ingest** — detects `README.md`, `CONTRIBUTING.md`, `CLAUDE.md`, `AGENTS.md`, and `docs/` in cwd. Ingests into a `<project>-project-docs` KB collection. Deduplicates via `import_log`.
5. **Smoke test** — if `_meta.smoke_test_version` differs from current package version, runs a 22-query benchmark from `eval/benchmarks/smoke.jsonl`. Pass threshold: `exactMatchRate >= 0.8`. Writes version to `_meta` on completion. Result returned as `smokeTest` field.
6. **Sticky + hot tier assembly** — sticky (pinned) entries are injected first, verbatim, under a `## Pinned directives (always follow)` header (never truncated; their token cost reduces the hot budget), then current non-sticky hot entries as injectable context. Enforces token budget by evicting lowest-decay entries to warm. Sticky entries are never scanned by decay, the warm sweep, or compaction, and are unbounded (exempt from the hot per-entry char cap). The block is rendered by `PinnedBlockRenderer.Render` (`src/TotalRecall.Infrastructure/Memory/PinnedBlockRenderer.cs`) — the single source of truth shared by session start, `session_refresh`, and the per-turn pinned-floor hook. (This rendering was previously inlined as `SessionLifecycle.BuildPinnedBlock`; `SessionLifecycle` now calls the extracted renderer.) **Project scoping** (when `tiers.pinned.project_scoping = true`, the default): `ProjectResolver` detects the current git repo from cwd (pure filesystem walk; no `git` subprocess) and reduces the `[remote "origin"]` URL to a lowercase `owner/repo` slug (or bare folder name when no remote is configured). Pins tagged with a matching `project` value inject only in that repo; untagged pins are global and inject everywhere. When no repo is detected, only global pins inject (fail-closed). Policy is centralised in `PinnedScope.OptsFor` (`src/TotalRecall.Infrastructure/Memory/PinnedScope.cs`).
7. **Tier summary** — counts entries across all tiers and KB collections, returned as `tierSummary` in the response.
8. **Hint generation** — `GenerateHints()` surfaces up to 5 high-value warm memories: corrections and preferences (priority 1), frequently accessed entries with `access_count >= 3` (priority 2), and warm→hot promotion candidates (priority 3). Each hint is truncated to 120 chars. No LLM calls — DB queries only.
9. **Session continuity** — `GetLastSessionAge()` returns human-readable relative time since last compaction event (proxy for last session). Returns `null` for first-time users.
10. **Config snapshot** — captures current config as a named snapshot (`"session-start"`), sets `ctx.ConfigSnapshotId` for the session so retrieval events and compaction are tagged to this config state.
11. **Regression detection** — compares current session metrics against previous config snapshot. Alerts if miss rate increased by ≥ `regression.miss_rate_delta` (default: 0.1) or latency increased by ≥ `regression.latency_ratio` (default: 2.0x). Skipped if fewer than 2 snapshots or insufficient events. Result returned as `regressionAlerts` field.
### Pinned-directive floor (per-turn re-injection)
`session_start` injects the pinned block once, at the top of context. In a long session it drifts up the transcript and the model stops honoring it. The **pinned floor** re-asserts it near the live edge on an adaptive throttle.
- **Hook:** `hooks/user-prompt-submit/run.sh` is a `UserPromptSubmit` hook (wired in `hooks/hooks.json`). It is a thin wrapper: it picks the host (`claude-code` default; `cursor`/`copilot-cli` from env), short-circuits to `{}` for Cursor (whose `beforeSubmitPrompt` is block-only), and otherwise forwards stdin to the CLI. It MUST never fail a prompt — on any non-zero exit or empty output it emits `{}` and exits 0.
- **CLI backend:** `total-recall pinned-floor --host <claude-code|copilot-cli|cursor|kiro|gemini-cli>` (`src/TotalRecall.Cli/Commands/PinnedFloorCommand.cs`). Reads the host hook payload (`session_id`, `transcript_path`) from stdin, decides via the throttle, and on an inject turn renders the pinned block live from the DB (via `PinnedBlockRenderer`) and emits host-correct output. For `claude-code`/`copilot-cli`/`gemini-cli` this is a JSON envelope (`hookSpecificOutput.additionalContext`); for `kiro` it is plain text (Kiro appends raw STDOUT verbatim to context, no JSON envelope). Fail-safe: any exception prints `{}` (JSON-envelope hosts) or empty string (`kiro`) and exits 0.
- **Throttle:** `PinnedFloorDecider.Decide` (`src/TotalRecall.Infrastructure/Memory/PinnedFloorDecider.cs`) is a pure, I/O-free function. Re-injects when EITHER trigger trips since the last injection: `floor_every_n_turns` turns elapsed OR ~`floor_growth_tokens` of transcript growth (estimated from transcript byte size at 4 bytes/token). The first turn of a session seeds state and skips (session_start already injected). Degenerate config (≤ 0) is clamped, never injected every turn.
- **State:** `PinnedFloorState` (`src/TotalRecall.Infrastructure/Memory/PinnedFloorState.cs`) persists per-session throttle state as JSON under `<dataDir>/floor-state/<sanitized-session>.json`. Reads are fail-safe (any error → fresh unseeded state); files older than 3 days are pruned on each run.
- **Config:** the Pinned tier was merged into Hot as a sticky flag in 4.0; `[tiers.pinned]` persists as a deprecated alias whose `floor_*` / `project_scoping` fields still drive sticky re-injection (`max_content_chars` no longer applies). Fields — `floor_enabled` (default `true`), `floor_every_n_turns` (default `6`), `floor_growth_tokens` (default `6000`), `project_scoping` (default `true`). Set `floor_enabled = false` to disable the per-turn floor; set `project_scoping = false` to inject all pins everywhere regardless of repo. Loaded into `Core.Config.PinnedTierConfig` (`FloorEnabled` / `FloorEveryNTurns` / `FloorGrowthTokens` / `ProjectScoping`).
- **Per-host status:** Claude Code active (`UserPromptSubmit` → `additionalContext`); Copilot CLI wired the same way but upstream currently ignores the returned `additionalContext`; Cursor uses session-start injection + skill-guided `session_refresh` as a layered fallback; Kiro active (`UserPromptSubmit` command hook → `pinned-floor --host kiro` plain-text STDOUT; `UserPromptSubmit` agent hook → retrieval nudge; `SessionStart` agent hook → `session_start` tool call; `Stop` agent hook → capture; `always`-mode steering → persistent `session_start` instruction). Known limitation: token-growth throttle inactive for Kiro (payload lacks `transcript_path`); floor runs on turn-count only. Gemini CLI active (`BeforeAgent` command hook → `pinned-floor --host gemini-cli` `hookSpecificOutput.additionalContext`; `GEMINI.md` → persistent `session_start` instruction). Token-growth throttle active (payload includes `transcript_path`). pi active via the `pi/` extension (`before_agent_start` returns a hidden `total-recall-context` message carrying the session-start text once per `session_start`/`session_compact` plus the floor output; `session_start` reason `reload` is skipped when the transcript already holds the message): the extension spawns the engine binary directly (`pi/floor.js`, gated on `existsSync` + `payloadIntact`, NOT on `.verified.json`, which only the shim's verified download writes; silent until the engine is provisioned), passes `{session_id, cwd}` with no `transcript_path` so the floor is turn-count only (pi creates its session file only after the first message, which would misfire the growth trigger on turn 2), registers the MCP server with no `cwd` so the engine's project detection sees pi's directory and with `timeout: 120` (pi's 60 s default is shorter than a cold first `session_start`), follows a hand-configured `total-recall` `mcp.json` entry (`pi/userServer.js`: plain `TOTAL_RECALL_*` env values are passed to the floor, dynamic/remote/disabled entries silence it), re-queues the instructions on `session_tree` when the active branch no longer holds them, appends a note that the CLI is run as `node <root>/bin/start.js` (`total-recall` is not on PATH after `pi install`), and skips every hook for subagent children (`PI_SUBAGENT_ID`) and `--no-session` runs. `/total-recall` forwards to the model and works interactively or over RPC, not in `pi -p`. Measured on Windows (win-x64, 10 runs each): a floor call costs about 140 ms whether it seeds, skips or injects (median 137-138 ms, max 161 ms), and the published package is about 378 MB unpacked (`npm view` 4.2.3: 377,586,264 bytes, 89 files). Cline/Windsurf/Zed/VS Code: MCP-tools-only (no hooks, no pinned floor); Windsurf has a rules file for session-start instruction.
- **Codex CLI:** active via `.codex-plugin/plugin.json` → `codex/hooks.json` → `codex/hook.js user-prompt-submit`, which runs the provisioned engine directly (`pi/floor.js`, dynamic import so a broken file cannot fail the hook) as `pinned-floor --host claude-code` with payload `{session_id, cwd}`. **No `transcript_path` is forwarded, so the floor is turn-count only:** a Codex rollout grows ~9 KB per turn of fixed records plus ~5 KB per sampling round plus tool output and the injected block itself (measured: ~38 KB per turn including 19 KB of re-delivered instructions on resume), so the engine's 24 KB growth trigger would fire on most prompts. Verified on Codex 0.162.1: the floor fires on exactly the 7th prompt of a session (default every 6 turns) regardless of tool activity, and the session id is stable across `codex exec resume`. Hooks inherit Codex's full environment while the bundled MCP server gets a sanitized one, so `codex/mcp.json` forwards the store-determining variables (`env_vars`); `STRIP_STORE_ENV` in `hook.js` is the unused fallback.
### Config persistence
`config_set` writes to `~/.total-recall/config.toml` via `Tomlyn`. Changes are merged with existing user config and take effect immediately in the current session. Before writing, `config_set` auto-creates a config snapshot named `pre-change:<key>` so retrieval metrics from before and after can be compared with `eval_compare`.
### Eval system
`eval_report` returns: precision, hit rate, miss rate, MRR, latency, breakdowns by tier and content type, top misses (lowest scoring queries), false positives (high score but unused), and compaction health (total compactions, preservation ratio, semantic drift). Data comes from `retrieval_events` and `compaction_log` tables. Accepts optional `config_snapshot` param to filter events by a specific config snapshot ID, and `days` param (default: 7). Outcomes are resolved at read time: a retrieval marked used is a hit, one explicitly marked unused (or one left un-acted past the grace window) is a miss, and a recent un-acted retrieval is still pending and excluded. Scored to the `assistant` query source by default (override with `source`); the grace window defaults to 60 minutes (override with `grace_minutes`). This is why the metrics are driven by `memory_feedback` — without feedback, recent retrievals stay pending and only age into misses after the grace window.
`eval_compare` compares retrieval metrics between two config snapshots. Required param: `before` (snapshot name or ID). Optional: `after` (default: `"latest"`), `days` (default: 30). Returns summary deltas, per-tier and per-content-type breakdowns, and query-level diff showing regressions (used→unused) and improvements (unused→used). Warns if either snapshot has no retrieval events.
`eval_snapshot` manually creates a named config snapshot. Returns `{ id, name, created }`. Useful for tagging a baseline before config experiments.
`eval_benchmark` measures the local embedder's retrieval quality against the bundled corpus. It runs against an **isolated, ephemeral SQLite database** (created, seeded, searched, and deleted per run) — never the live store — so it neither mutates real data nor surfaces real memories/KB in the per-query results.
`eval_grow` lists pending benchmark candidates auto-captured from retrieval misses (in `benchmark_candidates`) and lets you accept/reject them. Accepted entries get appended to `eval/benchmarks/retrieval.jsonl`. **Accept guard:** candidates are real captured queries/content, so an accepted entry whose query or surfaced content contains a secret, PII (always-on generic detection: emails, API tokens, private keys), or a configured internal term (`eval.grow.sensitive_terms`, default empty, matched case-insensitively) is blocked from the public corpus and left pending for review. Sensitive candidates are flagged in both the CLI table and the web UI grow list.
`memory_feedback` (assistant-only; registered in sqlite + cortex modes, not Postgres) records whether a retrieval was actually used. The host passes the `retrievalId` from a prior `memory_search` (`{ retrievalId, results }`) or `kb_search` (top-level `retrievalId`) response, plus `used` (defaults to `true`); it updates the outcome on the matching `retrieval_events` row so `eval_report` and the Insights health score can score real hits/misses. It is deliberately **not** in the web UI `ToolAllowlist` — feedback is an assistant-loop signal. Note: in cortex mode `kb_search` routes to the remote backend before local telemetry runs, so its `retrievalId` is `""`; calling `memory_feedback` with an empty/unknown id is a harmless no-op (`{ updated: false }`).
`insights` runs entry-level analysis of the local memory store (works in every backend mode — user memories are stored locally and locally embedded even under cortex): near-duplicate clusters (same-tier cosine over the live set hot+warm, capped by `limit`), pin-promotion candidates (high `access_count`, not yet sticky), retrieval gaps, a recall-vs-threshold curve (`Metrics.Compute` run at several thresholds over one load of `retrieval_events`), and a self-explaining health score with a four-part breakdown (retrieval/capture/pinned/KB). It powers the web UI Insights page. Near-duplicate detection is same-tier in v1.
### `ToolContext` and the composition root
`ToolContext` (in `src/TotalRecall.Server/`) carries session state through all tool handlers: `Store`, `Config`, `Embedder`, `SessionId`, and `ConfigSnapshotId`. The `ConfigSnapshotId` is set by `session_start` and used by `memory_search` (for retrieval event logging) and the compactor (for compaction logging). New tools that call `LogRetrievalEvent` should pass `ctx.ConfigSnapshotId`.
The composition root in `src/TotalRecall.Host/Program.cs` wires up all dependencies (storage, embedder, importers, MCP server, migration guard) and is the AOT entry point. The 51 MCP handlers live in `src/TotalRecall.Server/Handlers/` — one file per handler.
---
## Database Migrations
Schema changes are handled by a sequential migration framework in `src/TotalRecall.Infrastructure/Storage/Schema.cs`. The `MigrationRunner` runs each migration function inside a transaction, indexed by `_schema_version`. On startup, it checks the current schema version and runs any newer migrations.
Schema is managed by sequential migrations (currently 16; Migration 16 added the pinned tables). For the authoritative migration table, see `src/TotalRecall.Infrastructure/AGENTS.md`.
To add a schema change:
1. Add a new function to the migrations array in `Schema.cs` (do **NOT** modify existing migrations — they're frozen).
2. The function receives the `SqliteConnection` and runs inside a transaction.
3. Use `CREATE TABLE IF NOT EXISTS` and `ALTER TABLE ... ADD COLUMN` as needed.
4. The version number is the array index + 1.
5. Add a test in `tests/TotalRecall.Infrastructure.Tests/SchemaTests.cs` covering the new migration's pre/post-state.
---
## Diagnostics & error reporting
`src/TotalRecall.Infrastructure/Diagnostics/ExceptionLogger.cs` provides `LogChain(prefix, ex)` that walks the entire `InnerException` chain and writes each level with indented `-> <Type>: <Message>` formatting, then the outer stack trace. AOT-safe (uses the first-class `InnerException` property, no reflection).
**Use it at every catch boundary** that can plausibly hit a static-ctor failure or P/Invoke failure (migration guard, server composition, CLI commands that touch the embedder or open the DB). A bare `Console.Error.WriteLine(ex.Message)` at one of these boundaries hides `DllNotFoundException` for missing native libs under a `TypeInitializationException` and produces unactionable error output. The 0.8.0-beta.4 beta tester was blocked for ~30 minutes by this exact failure mode before `ExceptionLogger.LogChain` was retrofitted across 10 boundary catches in 0.8.0-beta.5.
---
## Project-wide standing rules
These are non-negotiable rules that apply to every commit, every PR, every subagent dispatch:
1. **No `Co-Authored-By: Claude ...` (or any AI co-author) trailers** in commit messages. Project history is attributed solely to human authors. When dispatching subagents that include `git commit` instructions, the prompt MUST explicitly say "do NOT add any Co-Authored-By trailer."
2. **Spec and plan documents in `docs/superpowers/specs/` and `docs/superpowers/plans/` are NEVER committed.** They live in the working tree only. The brainstorming and writing-plans skills both default to committing them — that default is overridden on this project. Do not auto-commit them.
3. **Seven-file version sync on every release** (see "Version sync" section above).
4. **Never delete anything destructively.** This applies broadly: never `git reset --hard` without confirmation, never `git push --force` without confirmation, never delete user data. The `AutoMigrationGuard` follows this principle: it sidelines suspect database files to `<dbPath>.failed-migration-<utc>` instead of deleting them.
## Deferred Items
See `docs/TODO.md` for the post-cutover follow-up backlog: code signing, multi-platform CI matrix, version-sync automation, doc scrubs, MCP bootstrap shim follow-ups, and more.