CLAUDE.md git:20261017.bc83944B

Immutable. This exact content never changes and is served at /api/v1/blob/378546d83e36ca83.

# Session Cartographer

Hooks are the foundation — they produce JSONL event logs. Everything else is an independent lens on that data.

```
Hooks (produce JSONL)
  ├── /remember (CLI search, bash + awk)
  ├── /standup (peer view: concurrent sessions and file contention)
  ├── /carto (web UI, Node + React)
  ├── /wrapup (strategic session-end preservation)
  ├── /trustmap (auto mode environment config)
  └── Qdrant indexer (semantic search)
```

- **`/remember`** — Claude uses this to recover context from past sessions. Agent's primary history tool.
- **`/remember --project <name>`** (no query) — Orient on a project or family before diving in. Replaced `/focus`, now a Claude Code built-in; `skills/focus` is a retirement stub.
- **`/standup`** — The peer view. `/remember --project` answers what happened in a project; this answers who else is in it right now: sessions active in the window, their projects and commits, and the files two sessions are both editing. `--commit <sha>` names the session behind a commit that landed underneath you. Groups the events the hooks already write by session; captures nothing new, writes nothing.
- **`/carto`** — Opens the Explorer web app for the human. Not an agent tool.
- **`/wrapup`** — End-of-session synthesis. Captures decisions, discoveries, and unfinished threads as a milestone event. Agent-initiated.
- **`/trustmap`** — Derives auto mode's `autoMode.environment` from the corpus. The same question Claude Code's setup wizard answers by rescanning the machine, answered instead from events already extracted — usage-weighted, cross-provider, and re-runnable so an update proposes only the delta.
- **CLI** (`cartographer-search.sh`) — Standalone search, no install needed. Used by all skills.

## Project Structure

```
scripts/
  cartographer-search.sh        — CLI search: BM25 (awk) + semantic + RRF fusion
  cartographer-project.sh       — Resolves a directory to its real project (worktree → parent repo); CLI face of cartographer_project() in hooks/common.sh, used by skills
  cartographer-common.sh        — Sources hooks/common.sh into scripts/ from either layout, so shell writers share cartographer_append
  jsonl-lock.js                 — Rewrite lock + state file shared by appenders and in-place rewriters; replaceJsonl (tail-carrying temp+rename), appendJsonl, snapshot/drain CLI
  repair-torn-rows.js           — Dry-run-default split of rows torn by concurrent appends, settled by copies in the other logs
  cartographer-facts.js         — Facts client: POST /api/facts over HTTP or spool, renders counts, writes no telemetry
  bm25-search.awk               — BM25 scorer (two-pass TF-IDF, pure awk)
  transcript-to-turns.awk       — Turn-group transcripts (user→next-user boundary) before BM25
  embed-events.js               — Batch index JSONL events into Qdrant
  qdrant-collection.js          — The one definition of a bootstrapped collection: creation plus the project/timestamp payload indexes; CLI `ensure-indexes` for skills and shells
  qdrant-auth.js                — Which Qdrant API key a request carries (env, else start-services' key file for loopback URLs); qdrantFetch adds the api-key header
  qdrant-auth.sh                — Shell face of the same rules (sourced by index-event.sh, cartographer-search.sh, start-services.sh); qdrant_curl
  qdrant-curl.sh                — CLI face of qdrant_curl, for skills
  semantic-search.js             — Query Qdrant by vector similarity
  index-event.sh                — Real-time single-event indexing (called by hooks)
  backfill-git-history.sh       — Import git commits into event logs
  backfill-memories.sh          — Index Claude Code memory files
  backfill-app-sessions.js      — Import desktop-app/Cowork session metadata (titles + VM-session prompts)
  retro-index.sh                — Backfill historical transcripts into Qdrant (turn-grouped)
  catch-up-transcripts.sh       — Checkpointed/cooldown transcript refresh used by SessionStart
  infer-codex-project.js        — Derive Codex repo attribution from tool workdirs when cwd is generic
  hermes-source.js              — Hermes Agent adapter: SQLite sessions + workspace artifacts → milestone rows and turns (read-only; needs a policy file)
  reconstruct-history.js        — Deep transcript analysis for backfill (turn-grouped)
  classify-prompt-intent.js     — Rule-based prompt-intent classifier (17 categories, zero-dep)
  backfill-prompt-intents.js    — Tag already-indexed turns with prompt_intent (payload-only, no re-embed)
  prompt-intent-report.js       — Corpus-wide intent distribution + per-bucket sampling (retuning tool)
  hit-rate-report.js            — Joins served-log.jsonl + access-ledger.jsonl: search hit rate by rank/source/project
  abandonment-report.js         — Read-only probe of no-access recall calls: reads transcripts to split unmarked use, good/bad abandonment and no-need, reports MRR with implicit use credited; gated on reproducing hit-rate-report.js (docs/ABANDONMENT.md)
  session-digest.js             — Compact per-session panel (tempo, commits, files, recall, dirty repos); used by /wrapup
  day-digest.js                 — One local day across all sessions, by project, commits checked against git (landed/moved/rewritten/missing, git-only); `session-digest.js --day`, --md receipt, --json contract
  digest-parts.js               — Commit parsing, edit-path resolution, repo state shared by both digests
  cartographer-standup.js       — Concurrent-session roster + CONTENTION (files two sessions both edit) + --commit attribution; used by /standup
  cartographer-collision.js     — PreToolUse edit hook body: warns when a peer session touched the file; also runs the history note below
  file-history.js               — Opt-in (CARTOGRAPHER_FILE_HISTORY_ON_EDIT=1) first-edit note: diagnoses naming the file, last peer commit, a wrapup; once per (session, file)
  non-projects.js               — isNonProject(): workspace-root and worktree directory names that are not projects (filters project labels, never file paths)
  trust-digest.js               — Derives infrastructure actually touched (orgs, LAN hosts, buckets, CLIs) for auto mode's autoMode.environment; used by /trustmap
  build-profile.js              — Derives .carto/profile.md: standing summary of projects, preferences, decisions, work shape, cadence
  sentinels.js                  — isResolved()/firstResolved(): the one definition of "field carries no real value"
  recall-join.js                — The one served/access join: "marked used" = exact (call_id, event_id) result_used row; shared by Internals, session-digest.js, and the Memory Desk's Recall view
  session-windows.js            — Shared session time-window construction (enrich + repair consume it)
  session-match.js              — Strict orphan→session matching (project + proximity, refuses ambiguity)
  backfill-investigations.js    — Normalize /investigate hypotheses from .carto/events into the searched log
  repair-orphan-sessions.js     — One-time recovery of pre-0.5.0 milestones stamped session_id "unknown"
  repair-transcript-paths.js    — Rewrites Codex transcript_paths stale from archiving (dry-run default, --write)
  resolve-transcript.sh         — Single resolver: recorded path → archive basename → session-id hunt
project-registry.json             — Project aliases for multi-repo families (used by search + orientation)
plugins/session-cartographer/
  skills/remember/SKILL.md      — /remember skill (Claude's context recovery tool)
  skills/focus/SKILL.md         — retirement stub pointing to /remember orientation
  skills/standup/SKILL.md       — /standup skill (peer view: who else is working, file contention, commit attribution)
  skills/carto/SKILL.md         — /carto skill (launches Explorer web app for humans)
  skills/wrapup/SKILL.md        — /wrapup skill (strategic session-end preservation)
  skills/trustmap/SKILL.md      — /trustmap skill (derive/update auto mode's autoMode.environment from the corpus)
  scripts/remember-search.sh    — Legacy keyword-only search (superseded by cartographer-search.sh)
  hooks/
    hooks.json                  — Hook registrations, including background transcript catch-up
    log-research.sh             — WebFetch/WebSearch → research-log.jsonl + changelog.jsonl
    log-session-milestones.sh   — Compactions, session ends, agent stops (with git context)
    log-tool-use.sh             — Edit/Write/Bash + git commits with classification (opt-in)
explorer/
  server/
    index.js                    — Express API (:2526), SSE stream, search proxy
    bm25.js                     — BM25 scorer (JS port for API path)
    search.js                   — Hybrid search: BM25 + Qdrant proxy + RRF
    jsonl.js                    — Resilient JSONL reader with fs.watch; logPositions/readAppended serve durable append cursors
    facts.js                    — Deterministic aggregates over the warm corpus: census, tempo, delta (folds, no index)
    facts-contract.js           — Facts verbs, bounds, cursor encode/decode, request/response validation
    event-time.js               — One definition of "when did this happen" (shared by ranking and facts)
    project-filter.js           — One definition of "is this event in scope" (matches bm25.js substring behaviour)
    memory-recall.js            — Recall calls for the Memory Desk (/api/memory/recall, /recall/call): on demand, read-only, unjoined marks listed not guessed
    memory-day.js               — /api/memory/day: runs scripts/session-digest.js --day as a child and returns its JSON or Markdown verbatim, so the Day view cannot disagree with the CLI or the pulse
  src/                          — React 19 + Vite + Tailwind UI (:2527)
docs/
  FACTS.md                      — /api/facts spec: census/tempo/delta, why folds not indexes, delta cursor rules
  RANK_FUSION.md                — BM25 + RRF scoring architecture
  SCORING.md                    — Score interpretation guide
  SETUP.md                      — Install, Qdrant, cold start backfill, disk usage
  MIGRATION_TURNS.md            — Existing-user guide for turn-based transcript migration
  CUSTOM_HOOKS.md               — How to log your own events
  EXPLORER_SPEC.md              — Explorer implementation spec
  companion_explorer_spec.md    — Explorer product spec
  CHANGELOG_SPEC.md             — Event log format
  landscape-survey.md           — 30+ Claude Code memory projects compared
  LONGMEMEVAL.md                — What a LongMemEval run would/wouldn't validate; ingestion + reader config
tests/private/                  — Gitignored: test cases, fixtures, benchmarks
```

## Implementation Constraints — READ THESE

- **BM25 in awk is intentional.** `bm25-search.awk` is the CLI search scorer. Zero dependencies (no Node, no jq). Do not port to Python. The JS port in `explorer/server/bm25.js` exists separately for the API path.
- **Turn-grouping is an indexing-layer concern, not a query-layer concern.** `transcript-to-turns.awk` preprocesses transcripts into one document per turn (user prompt + assistant responses up to the next user prompt) and is used by `retro-index.sh` + hooks when ingesting into Qdrant. Event IDs are deterministic (`turn-<sid>-<idx>`). Do not run it per-query — the CLI keyword `--transcript` fallback is plain per-line grep by design; semantic search (Qdrant) owns turn-coherent transcript recall. See `docs/MIGRATION_TURNS.md` for historical context.
- **Field extraction uses a fallback chain** (`summary → description → prompt → url → query → event_id → milestone`) across diverse JSONL schemas. Do not hardcode a single field.
- **Event summaries are single-line.** Writers flatten `\n`/`\t` at the source (`tr` in hooks, gsub in the semantic TSV emitter, `index-event.sh`). The pipeline is TSV/line-based: a multi-line summary splits a TSV row, and the fragments mis-parse as rank/key/timestamp — rank coerces to 0 and outranks every real result. The fusion awk drops rows with an empty key or non-numeric rank as a backstop. Pretty-printed JSON must never be appended to the JSONL logs.
- **Transcripts are first-class citizens in RRF.** They compete equally with event log results. Do not append them at the bottom.
- **`LC_ALL=C` on grep and awk** prevents multibyte errors on unicode in JSONL.
- **Transcript search uses `find -exec grep {} +`** to batch file matching in one process. Do not revert to per-file subprocess loops (1,839 files = 40x slower).
- **A hot hook reads its payload with one jq, and the rows it writes are pinned to a capture.** `log-tool-use.sh` runs on every Bash/Edit call and `log-session-milestones.sh` on every turn; on 2026-10-07 a fork census put a plain `npm test` at 64 processes and ~230 ms (19 jq, 12 awk, 8 head, 5 tr, 5 grep) and the per-turn Claude Stop no-op at 24 processes and ~165 ms, nearly all of it process starts (~13 ms a jq, ~28 ms a git, ~52 ms the python3 that URL-encoded one path). Now 7 and 2. The shape that holds it: every field comes out of one `jq … | @sh` into `eval set --`, including the last changelog row for the parent lookup and `now | todate` for the timestamp; everything the Bash branch reads from the command text (heredoc stripping, the three git invocations, the noise verdict, the write harvest, the 500-byte summary) comes out of one `LC_ALL=C awk` (`AWK_COMMAND_FACTS`); both rows of a pair come out of one jq; the project is resolved only on the branch that writes, through `cartographer_repo` (one git call for toplevel and project). Byte cuts go through `cartographer_bytes`/`cartographer_head_c`, which slice under `LC_ALL=C` the way `head -c` did; a python interpreter is not a dependency of a hook. The contract is byte-identical rows: `tests/fixtures/hook-rows/expected.txt` is what the pre-rewrite hooks wrote for the payloads in `run.sh` (ids, timestamps, shas and the temp path masked), and `tests/unit/hook-rows-golden.test.js` diffs the live hooks against it. Change what a row says and that capture is regenerated deliberately, from the old hooks or by hand, never by copying the new output over it.
- **Every JSONL append is one `write(2)`, and every in-place rewrite goes through `scripts/jsonl-lock.js`.** bash's builtin `printf` writes through a 1024-byte line buffer on macOS (8 KB on glibc), so a longer row left in several writes and concurrent sessions spliced rows together: six lines of `session-milestones.jsonl` on 2026-10-16, invisible on Linux CI, skipped silently by every reader. `cartographer_append` in `hooks/common.sh` is the one shell append. A row ≤1023 bytes keeps the builtin (no process, 99.5% of tool-use rows), and a longer one goes through one `cat` of a here-string. Do not hand-roll `printf … >> log` or `jq … >> log` in a hook or script, and do not let awk write JSONL through its own buffer (4 KB, it tears too). Node writers use `appendJsonl`. Rewriters read with `snapshotJsonl` and write with `replaceJsonl`, which takes `<log>.lock`, marks `<log>.rewrite` held with a generation, carries every row appended since the read, renames, and drains `<log>.spool`. Appenders re-read the state after writing and write again unless the next generation still holds. A grace period alone lost rows at load 160 and 434, so delivery is at least once by design: a rare row appears twice, and readers fold it by event_id. Truncate-and-write, or a plain `mv` over a log, loses the rows that sessions append meanwhile. `tests/unit/jsonl-append-concurrency.test.js` and `jsonl-lock.test.js` hold both properties, each with a control that must fail, and run on macOS in CI. A torn line is counted (`logs.malformed` in both health endpoints), never skipped silently.
- **The SessionStart note reads `unresolved[]`, picks by parsed time, and keeps its gate in the trial log.** `surface-focus-on-start.sh` (opt-in, `CARTOGRAPHER_FOCUS_ON_START=1`) renders through `hooks/start-note.jq`: the newest `/wrapup` for the `cartographer_project`, the checkout's state, and live peers from `cartographer-standup.js --roster`. Every wrapup since 2026-08-20 carries `unresolved[]` (326 of 326 measured 2026-10-15), so that is the hand-off field. A parallel `open_threads` would be a second copy to drift, and only older wrapups fall back to their `Unfinished:`/`Open:` clause. "Newest" is by parsed timestamp, because 6 wrapups carry `+00:00` offsets and file order is not time order. The note shows when the wrapup id differs from the last shown row's `wrapup_id` in `focus-on-start-trial.jsonl`, or a peer is live, and every evaluation logs a row with its `session_id`. It replaced the co-occurrence lenses, which cost ~1.8 s per start (two full graph builds, 22 processes) and drew no measurable follow-up across 41 shows. Do not put a graph build back on this path: the new note costs ~130 ms and 9 processes, ~100 ms of it node starting the roster.
- **Hooks call `index-event.sh` for real-time Qdrant indexing.** Qdrant remains optional, but failures are recorded in `.carto/index-errors.jsonl` and return nonzero so backfills do not checkpoint incomplete sessions.
- **Explorer binds to 127.0.0.1 only.** Never 0.0.0.0. Binding does not stop DNS rebinding, so every listener (the API, headless Turbo, the UI host) also refuses a non-loopback `Host`, a foreign `Origin` and `Sec-Fetch-Site: cross-site` through `explorer/server/loopback-guard.js`. Until 2026-10-16 only the UI host checked, and :2526 served 1.7 MB of `/api/memory/state` to a forged Host; `api-loopback-guard.test.js` and `turbo-loopback-guard.test.js` now send one to each port. Path traversal protection on transcript endpoints. Rendered content is never parsed as HTML, and there is no DOMPurify: markdown is emitted as React elements, nothing uses `dangerouslySetInnerHTML`, the Memory Weather view's `innerHTML` templates are static strings, and every corpus URL that becomes an `href` (Memory Desk markdown links, `EventCard`'s event `url`) passes `safeArtifactLink` (`http(s)`, `mailto`, `#` only); a refused one renders as text. React 19 rewrites only a `javascript:` href, so it is not the guard; `event-card-links.test.js` fails if an `EventCard` href skips the allowlist. A render path that emits raw HTML must bring its own sanitizer. No browser code calls Qdrant (the API proxies it), so `start-services.sh` runs it with CORS off, because Qdrant's default reflects any `Origin`. CORS off stops a web page reading :6333, not writing to it — Qdrant 1.12.1 parses a preflight-free `text/plain` POST as JSON, so only an API key closes that, and the launcher provisions one: generated on first start into `$CARTOGRAPHER_QDRANT_STORAGE/api-key` (mode 600), written into its mode-600 config as `service.api_key`, and sent as the `api-key` header by every client through `scripts/qdrant-auth.js` or `qdrant-auth.sh`, which read that file (loopback URLs only) or `CARTOGRAPHER_QDRANT_API_KEY`. A new Qdrant client goes through one of the two, or it fails against a keyed Qdrant without erroring; `qdrant-api-key.test.js` lists every client and checks the header arrives, and `qdrant-start-services-key.test.js` sends the blind write to a launcher-started Qdrant and expects 401.
- **Ports:** 2526 (API), 2527 (UI), 6333 (Qdrant), 8890 (embeddings).
- **`project-families.json` is gitignored.** Run `generate-families.sh` to bootstrap from event logs.
- **Enrichment scripts modify `changelog.jsonl` in place.** Back up before running on large datasets.
- **Only five logs are searched:** `changelog.jsonl`, `research-log.jsonl`, `session-milestones.jsonl`, `tool-use-log.jsonl`, `prompt-history.jsonl`. Anything written elsewhere is write-only. `/investigate` wrote 64 diagnoses to `.carto/events/` before anyone noticed. A new event writer must append to one of the five, use `event_id`/`timestamp`, put its searchable text in `summary`, and call `index-event.sh` — miss any one and the record is unreachable. The fifth is **derived, not hook-written**: `build-prompt-history.js` projects `~/.claude/history.jsonl` into it. That source is Claude Code's file and is never written to; it is also not read directly by either engine, because its rows carry no `event_id` and both scorers would then mint a positional key that moves on every append.
- **Never treat a sentinel as an identity. Use `scripts/sentinels.js`.** The event pipeline spells absence three ways — `""`, `"unknown"`, and `null` — and `/wrapup` alone has written all three. `"unknown"` is truthy and equal to itself, so `if (sid)` passes and `groupBy(sid)` merges every unattributed record into one phantom entity. Nothing errors; the numbers are just wrong. That phantom cost a 54% overstated recovery rate during the 0.5.0 repair. Any code that groups, matches, or keys on `session_id`, `provider`, or `project` must go through `isResolved()` rather than re-deriving the set inline (six sites had already diverged).
- **`project` is resolved from the git *common* dir, never from `--show-toplevel`.** Inside a worktree the toplevel basename is the throwaway worktree directory, so a session in `widget-web/.claude/worktrees/confident-yalow-e1cdc6` files under that name and every record orphans the moment the worktree is pruned — ~5,700 events, unrecoverable once the directory is gone, because the repair works by asking git to resolve the recorded cwd. `cartographer_project()` in `hooks/common.sh` is the definition; `scripts/cartographer-project.sh` is its command-line face for skills, which are markdown and cannot source a shell library. Fixing this in the hooks was only half of it: the three skills that write to `session-milestones.jsonl` each carried their own copy of the old derivation for four more months, and produced milestones that contradicted themselves — project `confident-yalow-e1cdc6` beside a digest reading `{widget-web: 118}`, since the digest is built from hook events. Agent control rooms (Tessera, Kangentic) put a worktree behind every task, so any new writer must call the shared resolver rather than spell a fourth copy.
- **Parse an edit summary with `scripts/edit-paths.js`; resolve it yourself.** The edit hook writes one line per edit in two shapes — `Modified: /abs/app.js` from Edit/Write, and `Modified: a.js,b.js (via bash)` when auto mode edits through `sed -i`, a heredoc, or `open(p,'w')`. Reading that back with a bare `/^Modified:\s*(.+)$/` fails three ways at once and errors on none of them: the ` (via bash)` marker rides into the key, a multi-file row becomes one fabricated filename, and the hook's detector — which reads shell *source text* and is loose by construction — puts JS property access in the panel. Measured on session `3c5f5a20`, the digest's hottest-files panel claimed 76 files touched and led with `errors.push` ×11, with `docs/HANDOFF.md` and `docs/HANDOFF.md,docs/PRODUCT.md` sitting in it as separate entries; 58% of the 2,889 candidates the hook has emitted corpus-wide are not paths. Parsing is shared. **Resolution deliberately is not**: `memory.js` serves file contents over HTTP and must confine itself to `CORPUS_ROOT`, while `session-digest.js` only names a file and would otherwise drop real edits to `~/.claude` memory files — pass the resolver that matches the risk, and report what missed (`files_unresolved`) rather than dropping it, since a candidate can miss because a real file was since deleted.
- **The session-id env var is `CLAUDE_CODE_SESSION_ID`.** `CLAUDE_SESSION_ID` is a legacy name Claude Code never sets. Reading only the legacy name left every served row unattributed and delta serving dormant for the whole life of the feature. Any new consumer must read the chain `CARTOGRAPHER_SESSION_ID → CLAUDE_SESSION_ID → CLAUDE_CODE_SESSION_ID → CODEX_SESSION_ID`, taking the first *resolved* value (`sentinels.js`), not the first non-empty one: a `${a:-$b}` chain let `CARTOGRAPHER_SESSION_ID=unknown` shadow a real id and become the served rows' session and a shared delta-serving list. Hermes exports none of the four. Its wrappers opt in with `CARTOGRAPHER_SESSION_ID`/`CARTOGRAPHER_PROVIDER` (docs/SETUP.md); `HERMES_SESSION_ID` only infers `provider: "hermes"` and is never read as a session, because that would switch on delta serving for gateway sessions that stay open for weeks.
- **Test harnesses must unset the session vars.** Delta serving is real now; a harness that inherits a live session id silently loses repeat results and fails passing tests.
- **Retrieval telemetry is exact for new calls.** Served rows and `--touch` records share `call_id`; purpose, session, and provider are carried alongside it. Served, search-call and access rows also carry `session_source` (the chain variable that supplied the session), and a sessionless served or call row says `attribution_status: "no_session"` rather than leaving a bare `""`. Both writers — the awk path and `turbo-search-client.js` — emit these; `telemetry-attribution.test.js` drives each with the variables its caller actually has. `access-ledger.jsonl` is append-only. Do not rewrite it in place, and keep the CLI/API activation implementations in sync. Legacy inference is report-only and opt-in.
- **`.carto/profile.md` is derived, never hand-authored.** `build-profile.js` rebuilds it from the corpus; hand edits are lost on the next run. CLAUDE.md owns hand-written context — duplicating it into the profile creates two sources of truth that drift. Two filters are load-bearing: backfilled git history contains other authors' commits (a profile without the owner filter describes everyone whose repo you ever cloned), and `project` is cwd-derived, so the home directory and workspace root show up as the busiest "projects" unless excluded.
- **`trust-digest.js` emits identifiers, never arguments.** Commands reduce to their leading word, URLs to their host. The output is meant to be pasteable into a settings file without a secret review, and a full command line from `tool-use-log.jsonl` cannot make that promise. Two heuristics there are load-bearing and were both wrong on the first pass: command tokens are resolved against `PATH` (splitting compound lines also splits the inside of inline `node -e`/`python -c` payloads, so `const` and `then` outranked `adb`), and hostnames are shape-checked (summaries clip at ~200 chars, so a URL near the end yields fragments like `huggingfa` that read as single-label internal hosts). Both replaced a guess with a fact — do not revert either to a stoplist.
- **`.carto/` is gitignored, and that default must survive.** Cartographer's event logs are agent-session transcripts, which auto mode's classifier treats as sensitive data belonging in no repo. A project that wants its history versioned deliberately un-ignores its own path; the default for a repo that merely *runs* cartographer stays "don't commit the logs."
- **Codex archives sessions; it does not delete them.** A finished Codex session moves from `~/.codex/sessions/<y>/<m>/<d>/` to the flat `~/.codex/archived_sessions/`, so every `transcript_path` the hooks stamp goes stale on archive while the data stays on disk. Consumers that stat the recorded path and stop report "transcript missing" for a fully recoverable transcript — a silent recall failure, which is worse than an error. `scripts/resolve-transcript.sh` is the single resolver (recorded path → archive basename → session-id hunt across all stores); scan roots carry `CARTOGRAPHER_CODEX_ARCHIVED_DIR` alongside the sessions dir. Do not reintroduce a bare `find ~/.codex/sessions`.
- **Every Explorer view names the agent that produced the row.** Roughly half this corpus is Codex (54,826 of 110,752 changelog events; 50,533 of 86,055 tool-use rows), and the Explorer shipped provider-blind for months while `/remember` and the Memory Desk handled both — `/api/sessions` folded the field away and no classic view read it, so a mixed corpus rendered as one agent with nothing reporting a problem. `explorer/server/sessions.js` owns the fold and attributes through `isResolved()`; `computeFacets` carries a `providers` dimension (`?fa=codex`); `AgentBadge` is the one palette, outlined rather than tinted, and measured at 8:1 on every surface it sits on. It measures 7.0–7.5:1 on the Memory Desk's selected row, so the agent appears there as plain text; the badge and selected rows are measured from computed styles in `tests/browser/memory-entry.cjs`. The classic views' type and project badges and facet pills take their hues from `explorer/src/lib/palette.js` under the same outlined rule, and the harness measures every classic ground the same way. Transcript derivation is **per-provider** — the pre-port fallback could only ever build `~/.claude/projects/<encoded>/<sid>.jsonl`, which no Codex session can match, so Codex rows silently derived nothing (7 of 28 openable in a 7-day window; 10 after the port, the other 18 genuinely absent from disk). List views use `codexSessionIndex()`, one directory walk for the whole window (1,426 files, 17 ms); the per-transcript ladder with its `find` stays in `/api/transcript`, where the user asked for one specific session. A second copy of provider attribution exists client-side (`explorer/src/lib/provider.js`) because the Timeline groups raw events itself and the server module sits on a shebanged Node script — `tests/unit/carto-provider-coverage.test.js` asserts the two agree, which is the only thing keeping that pair honest.
- **Hermes comes from SQLite, and four of its adapter's rules were measured, not chosen.** `scripts/hermes-source.js` reads `<hermes home>/state.db` and `profiles/*/state.db` read-only, and emits the same turn document the two awk adapters do (asserted against the Codex adapter's output in `tests/unit/hermes-source.test.js`). (1) **Exclusion is two-level.** A project, cwd or path match drops the whole session; a term match replaces only the *line*. On the reference corpus the excluded terms appeared in 105 of 106 cron runs and 66 of 67 daily notes, nearly all as the agent's own scope rules, so a mention-means-drop filter would have silently deleted the agent. Excluded project names are also redacted, because a run report that names what it skipped is itself the disclosure. (2) **A cron prompt is never indexed, and neither are tool results.** A scheduled agent's prompt can embed this corpus's own pulse, and indexing it feeds Cartographer's summaries back into Cartographer. (3) **The novelty gate is tuned per document type, not left at 0.85.** Turns run with it off (`PE_GATE_REJECT=2.0`, as `retro-index.sh` does), or a grown turn re-indexed under its own id matches itself at ~1.0 and is rejected. Artifact sections run at 0.97: at the default, 748 of 835 sections were rejected, the newest daily note entirely, because notes written from one template score 0.85–0.93 against each other while saying different things; true repeats clustered at 0.97–1.0. Write-time exit 0 includes gate rejections, so check `.carto/index-rejects.jsonl`, not the exit code. (4) **Single-shot sessions are not conversations.** 171 of 212 `desktop` sessions were one prompt answered once, so a session needs a second prompt or a tool call. Policy lives in the user's `hermes.json` (`--print-config-path`), never in the repo, and the adapter refuses to run without one. Drill-down is `hermes-source.js --show <session>`; an artifact row carries its markdown file as `transcript_path`.
- **Codex reads `hooks/codex-hooks.json`; Claude Code reads `hooks/hooks.json`.** Claude Code finds its file by convention, and Codex loads the file its manifest names. The Codex file is the Claude file minus the events Codex does not fire (today `PostToolUseFailure`), because a Codex that rejected an unknown event would switch off every Codex hook and nothing in a Claude session would show it. A new hook goes in both files unless its event is Claude-only; `tests/unit/provider-adapters.test.js` fails if they drift or if the Codex file names an event outside the list read from the Codex binary.
- **`--get` uses `rg` when available, `grep` as fallback.** BSD grep over the four event logs costs ~1s per invocation, which turns a five-id fetch into three seconds and teaches an agent to avoid the cheapest verification step in the system. `rg` does the same pass in 0.02s.
- **The noise trim runs on relevance, and decay is a switch, not a curve.** Both engines trim at 10% of the top RRF × salience score *before* any age weighting. Run after decay, as the API did until 2026-10-16, the trim cut old records for their age: 55 of 240 known items were absent from the pool, not merely ranked low. The CLI had no trim, so the engines also disagreed about the pool. Fusion is rank-based: rank 1 scores 1/61, rank 10 scores 1/70, and a rank-1 partial match in another ladder also scores 1/61. An old known item therefore survives the page only if its age factor stays within a few percent of 1. Measured on 204k events (240 known items, 40 recency cases, semantic off), known-item S@10 by age <7d/7-30d/30-90d/90d+ was exp λ=0.001 0.98/0.85/0.62/0.15; power-law t^-0.5 0.98/0.77/0.62/0.30; ACT-R base-level d=0.05 0.80/0.53/0.60/0.72, trading the week for the archive. So when ≤ `CARTOGRAPHER_DECAY_LOOKUP_MAX` (10) candidates hold every query token, those candidates skip decay: 0.98/1.00/0.98/1.00 on the API, with recency unchanged. On the CLI, measured on paired cases, past 90 days went 0.23 → 1.00. Everything else decays as before. Skipping decay keeps a named record on the page, not first: a recent 0.7–0.8-salience record sharing part of the query outranks a 0.25–0.5 one holding all of it. So the named candidates are also multiplied by `CARTOGRAPHER_DECAY_LOOKUP_LIFT` (3) after decay, as a score change so `--min-score` still cuts a sorted list. Live known S@1 went 148 → 221 of 240, S@10 234 → 240, no case lower, recency and topical identical (docs/SCORING.md, "Lifting named records"). "Holds every token" is presence-only (`full` on `scoreBM25` results, column 10 of `bm25-search.awk`), so the engines agree although their idf differs. `tests/unit/age-curve-gate.test.js` proves its fixture reproduces the loss before gating, and `scripts/age-curve.js` runs the same curve on the live corpus. Change λ, the trim, salience, the exemption or the lift only with both. A different browse curve also has to pass the topical kind (`age-curve.js --truth`, on the live corpus and `--as-of` the labels' date), because both gate metrics reward steeper decay. On 2026-10-17 every power-law form tested, ACT-R base-level included, lost topical P@10 there (0.40-0.42 against exp's 0.48), so exp stays (docs/SCORING.md). The truth scorer is `scripts/truth-score.js`, shared with `eval-search.js`; its word-overlap match needs two shared words, because at one it credited any recent row sharing a word with a short label, which is exactly what a steeper curve puts on the page.
- **A window is bound at the source, never applied to a ranking.** Every ladder truncates — `FUSION_DEPTH` in both engines, `max_results` in `bm25-search.awk`'s `END` block — so filtering afterwards keeps the globally best matches and only then asks which fall inside `--since`. On a six-figure corpus a day is ~1% of events, so the ladder arrives at fusion nearly empty while every stage reports success. All three legs now bound early: `scoreBM25` filters before its slice, the awk filters in **pass 2 only**, and `semanticSearch` puts a `timestamp` range clause in the Qdrant query. Two things are load-bearing. The awk's pass 1 owns `ndocs`, `avgdl` and `df` — narrowing it recomputes IDF over a handful of documents and silently reweights every surviving score, so in-window scores must stay byte-identical to an unwindowed run. And Qdrant compares RFC3339 payload strings **chronologically**, not lexicographically (verified on 1.12.1: `2026-03-17T21:21:04-07:00` is included by `gte 2026-03-18T00:00:00Z` and excluded by `lt` on the same boundary), which matters because ~2% of payloads carry non-UTC offsets — do not "simplify" it to a string compare, and do not drop the client-side `windowed()` backstop, which still catches a pool that arrives unbounded from a server without range support.
- **Project scope is a case-insensitive substring everywhere, and Qdrant equality is not it.** `projectMatcher` in `explorer/server/project-filter.js` is the one definition: a family name selects its repositories, so `widget` means `widget-web`. Qdrant's `match: {value}` is exact equality, so the semantic leg once scoped to a literal string that matched nothing while the keyword leg scoped to the whole family — measured at 0 semantic rows against 9,943 indexed points for `--project widget`, with `/api/recall` worst hit because it performs no registry expansion at all. Resolve the spec against the project values actually present and emit a `should` of exact matches (8–13 ms over 128k resident docs, under 1% of the request budget — computed per call, because caching it buys ~1% and costs a staleness bug the first time a new project appears). Do not "fix" this by dropping the filter and post-filtering: that reintroduces the truncation starvation above. The registry is precision and convenience, not the mechanism — an unregistered prefix must still scope both ladders alike.
- **The collection carries a `keyword` index on `project` and a `datetime` index on `timestamp`, and `scripts/qdrant-collection.js` is the only place that knows it.** The semantic leg filters every query on those two fields, and Qdrant evaluates an unindexed filter by scanning payloads. Measured 2026-10-07 on 157k points (Qdrant 1.12.1): a project-filtered search cost 200-420 ms server time and a timestamp range 100-170 ms, against 3-6 ms unfiltered. That was the dominant cost in the semantic stage of `/api/recall` and the reason some recalls blew the 4000 ms Turbo budget. With the indexes the same filtered searches cost 4-16 ms with identical top-50 results; each build took ~1.1 s once. They had been created by hand, so nothing in the repo made them and a fresh install or `--reindex` lost them while every stage reported success. Now every bootstrap path goes through the module: `embed-events.js` creates them with the collection, and the three places that already check collection existence (`index-event.sh`'s health check, the Explorer's `/api/health`, Turbo at boot) read `payload_schema` and create only what is missing, so an upgraded install gets them from its first hook event without a reindex. Re-creating an existing index is a Qdrant no-op (200 `acknowledged`, ~15 ms), but the hook path runs per event, so it is read-first and the common case writes nothing. A refused build (datetime needs Qdrant 1.8+) is recorded in `index-errors.jsonl` and backed off for an hour; it never fails the event, because an unindexed collection is slow while an unindexed event is lost.
- **Ranking answers "what is relevant." Facts answer "what is true." Do not ask one for the other.** A ranker handed a census question — "what happened yesterday" — has no relevance gradient to work with, so it returns *an* answer with no way for the caller to tell it is not *the* answer. Measured: a scheduled agent's daily pulse surfaced 1 event from a window holding 736 events, 22 sessions, and 20 commits across four repos. `/api/facts` (`explorer/server/facts.js`) owns counting; `/api/recall` owns relevance. A new question is a new verb there, not a cleverer query string.
- **The facts verbs are folds, not indexes, and that was a measurement not a preference.** Loading 127k events costs 881 ms — paid once, held resident by Turbo. A full linear fold over all of them costs 12–22 ms, under 2% of the request budget. Precomputing aggregates would buy nothing and add an invalidation path. The measured exception is extraction-derived facts (file paths out of free-text summaries: 368 ms), which is also where the extraction heuristic is most likely to be confidently wrong — see the `trust-digest.js` note above. Add an index only when a fold is actually too slow, and say which measurement forced it.
- **A fact carries the `event_id`s it counted.** Every bucket ships a bounded sample so any number can be checked with `--get`; `tempo` samples the day it scores, because the number the caller acts on is the one that has to be checkable. A deterministic answer that is silently wrong is strictly worse than a slow one. Never let an unresolved id occupy a sample slot — nulls filled the slots in arrival order and returned `[null, null]` for buckets holding perfectly fetchable ids.
- **`delta` is a cursor over log positions, never a `since` timestamp.** The corpus is backfilled: `backfill-git-history.sh`, `retro-index.sh`, and `catch-up-transcripts.sh` append events dated months in the past, so "timestamped after my last run" and "arrived since my last run" are different sets and only the second means *new*. Arrival order lives in the append-only logs, not in the resident array (loaded file-by-file at spawn, newest-first only for post-spawn arrivals). `logPositions`/`readAppended` in `jsonl.js` are the one implementation; they reuse `boundaryHash` because a byte offset alone cannot tell an append from an in-place repair, and a repair that grows the file makes the shifted tail read as fresh appends. A rewritten source is reported as `stale` and contributes nothing rather than yielding a confident wrong diff.
- **Facts write nothing.** `scripts/cartographer-search.sh` remains the single writer of served and access telemetry; a facts endpoint logging its own activity would reintroduce the double-counting the recall contract was shaped to prevent. Facts are projections of the five logs, not events — no sixth log, so the rule above is untouched.
- **A z-score over daily event counts is labelled, not trusted.** Counts are Poisson-ish and zero-inflated; below a baseline mean of ~10 the normal approximation fails and the score explodes. Real runs produced z=44.9 against a mean of 2.33 and z=60.1 against 0.22 — arithmetically right, practically useless, since both only mean "idle, then not idle". Report the score with `z_status`, never suppress or silently rescale it. The current UTC day is never scored: comparing a two-hour day against complete days reads as a collapse every time.
- **An HTTP status is an answer, not an outage.** Both Turbo clients once treated a 4xx as a transport failure and retried on the file spool — which reaches the same process, so it re-ran the rejection and emitted a composite error naming two failures that did not exist. Only an unreachable service earns the spool. **A client's own timeout is not an outage either**: the service has the request and is still running it, so a spool retry ran the same query twice at once. That was every recall fallback in the 30 days to 2026-09-26 (6 of 260): a 1.7–2.7 s semantic stage outran the 1.5 s budget, the rerun outran the spool's 3 s, and the caller got a 13–89 s portable search. A sandbox's refused connect fails in ~90 ms, which is how the two are told apart. The recall budget (`TURBO_TIMEOUT_DEFAULT_MS`, 4000 ms) covers the slow tail, not the median, because missing it costs the portable CLI. Exit non-zero either way on the recall path: `cartographer-search.sh` reads any non-zero exit as "fall back to the portable CLI", and on a contract rejection that fallback is still correct.
- **A CLI that prints and then calls `process.exit()` waits for stdout first.** On macOS Node writes to a pipe asynchronously, so an exit straight after a large write ends the process before the pipe has taken everything: the reader gets a prefix cut mid-string, with exit code 0. The MCP server reads every CLI it spawns through a pipe, and its `standup` tool failed to parse at byte 65,536 once a checkout had a few hundred modified files. Falling off the end of the module is safe, because Node waits for pending writes. An explicit exit first awaits `process.stdout.write('', resolve)`, whose callback runs only after every earlier write has completed (`exitWhenFlushed` in `cartographer-standup.js` and `cooccurrence-graph.js`; `wrapup-coverage.js` and `session-digest.js --day` await the write itself). `tests/unit/cli-pipe-drain.test.js` reads through a reader that stalls after the first chunk, and a new large-output CLI belongs in it. The size that matters is the smallest buffer a reader can bring, so treat 8 KB as the bound: measured 2026-10-16, a shell pipe and a Node-spawned socket both cut a single write at 65,536 bytes, but standup's cuts under the MCP server included one at 8,192, and a Node parent's socketpair starts with an 8,192-byte send buffer (`net.local.stream.sendspace`). The explicit exits left in `hit-rate-report.js` (no-data message), `trust-digest.js --template` (a fixed 2.2 KB) and `infer-codex-project.js` (one name) print far less than that; their large outputs fall off the end.
- **A file review diff is bounded by the session's commits, never by HEAD.** `reviewRange()` in `explorer/server/memory.js` picks base = last commit at or before the session's first event and head = last commit within a grace of its last event, falling to the working tree only when the session is in flight or committed nothing for that file. `sessionBounds()` folds over the whole warm corpus, not the desk window, because a 90-day view and a 1-hour view must bound the same session the same way. Boundaries are commit *times*, not authorship: a session whose first act commits a previous session's leftovers inherits them, and the response says so in `range` caveats rather than hiding it. Do not "simplify" back to `git diff HEAD` — that answers "what is uncommitted", a different question the panel is not asking.
- **Diff-shape quadrant labels:** bootstrap, construct, surgical, rework. Not "dangerous."

## Two Search Paths

1. **CLI** (`cartographer-search.sh`): bash + awk BM25. Used by `/remember` skill. No server needed.
2. **API** (`explorer/server/`): JS BM25 + Express. In-memory index, but `scoreBM25` scans every document: 20–31 ms median per query over ~203k events (measured 2026-10-16). Used by the Explorer UI (`/carto`) and Turbo. Proxies Qdrant for semantic search.

Both use BM25 (k1=1.2, b=0.75) and RRF (k=60), but they are not one ranker, and the same query can rank differently on each:
- **IDF scope.** The CLI scores each log file on its own, so N, avgdl and df are per file. The API indexes the deduplicated corpus once.
- **Dual-written events.** The hooks write many events to changelog and to a domain log. The CLI fuses both copies under one key and sums their RRF credit; the API deduplicates by `event_id` before scoring and credits one.
- **Trim.** Both engines drop results under 10% of the top RRF × salience score, before decay (since 2026-10-16). `--min-score` on the CLI is a further absolute floor.
- **Time.** The awk `ts_to_epoch` (`bm25-search.awk`, `cartographer-search.sh`) reads a timestamp's clock fields as UTC and ignores its offset, so a `-07:00` row sits seven hours early in decay and in `--since`/`--before`. The API parses the offset.

## How Claude should use this

When a user says "remember X" or needs context from a past session, use `/remember`. The skill runs the search, returns ranked results with transcript paths. **Read the transcript** to recover full context — the search result is the map, the transcript is the territory.

When a user says "explore" or wants to browse history visually, use `/carto` to start the web app and open the browser. That's a human tool — don't try to scrape it.

At the end of a productive session — or when a user says "wrap up" — use `/wrapup` to synthesize decisions, discoveries, and open threads into a milestone event. The milestone gets indexed and becomes searchable via `/remember`.

## Testing

**Read [docs/TESTING.md](docs/TESTING.md) before writing a test here.** Every real
defect in the recall path returned a confident, well-formed, wrong answer with no
stage reporting failure, so "it didn't error" and "it returned results" are both
worthless as signals — assert on *composition* (which ladders contributed, whether
a count survives an independent scan, whether an id fetches back).

- `env -u CLAUDE_SESSION_ID -u CLAUDE_CODE_SESSION_ID -u CODEX_SESSION_ID -u CARTOGRAPHER_SESSION_ID node --test tests/unit/*.test.js` — portable unit suite; use Node 22 to match CI and unset session variables so delta serving does not eat repeat results.
- `npm run build --prefix explorer` and `node tests/browser/memory-entry.cjs` — Explorer build and isolated headless Chromium regression; see the testing guide for dependencies.
- `bash tests/release-smoke.sh`, `bash tests/source-marketplace-smoke.sh` — packaging.
- `tests/private/` is gitignored and absent from a fresh checkout. Anything documented against it does not run for anyone who clones.
- **A fixture must prove it exercises the defect.** These bugs are absences, so a test asserting only "results came back" passes against the broken code. Add the second assertion — that in-window rows really do fall past `FUSION_DEPTH`, that the keyword and semantic ladders agree on scope.
- **Hook tests clear the trial gates.** Claude Code applies settings.json `env` to its Bash tool, so a developer running a trial has `CARTOGRAPHER_RECALL_ON_PROMPT`/`_FILE_HISTORY_ON_EDIT`/`_FOCUS_ON_START` in the suite's environment. Spread `withoutTrialGates(process.env)` (`tests/unit/helpers/trial-gates.js`) instead of `process.env`; `trial-gate-isolation.test.js` derives the gates from the hooks and fails a test file that runs one without it.
- **Pin the semantic leg off** with `process.env.CARTOGRAPHER_SEMANTIC = '0'` at the top of the file, before any import, or a live Qdrant leaks corpus ids into fixture expectations and the suite passes only where the service is down.
- **No MCP-based browser testing.** Use the checked-in headless browser harness and manual visual review; don't puppet Chrome via desktop-control MCP.