AGENTS.md · diff

git:20261017.1753942 to git:20261017.bb68e19

1 added, 1 removed. Audit A to A.

# Session Cartographer
Hooks are the foundation — they produce JSONL event logs. Everything else is an independent lens on that data.
```
Hooks (produce JSONL)
├── /remember (CLI search, bash + awk)
├── /carto (web UI, Node + React)
├── /wrapup (strategic session-end preservation)
└── Qdrant indexer (semantic search)
```
- **`/remember`** — Codex uses this to recover context from past sessions. Agent's primary history tool.
- **`/remember --project <name>`** (no query) — Orient on a project or family before diving in. Replaced `/focus`, now a Claude Code built-in; `skills/focus` is a retirement stub.
- **`/carto`** — Opens the Explorer web app for the human. Not an agent tool.
- **`/wrapup`** — End-of-session synthesis. Captures decisions, discoveries, and unfinished threads as a milestone event. Agent-initiated.
- **CLI** (`cartographer-search.sh`) — Standalone search, no install needed. Used by all skills.
## Project Structure
```
scripts/
cartographer-search.sh — CLI search: BM25 (awk) + semantic + RRF fusion
bm25-search.awk — BM25 scorer (two-pass TF-IDF, pure awk)
transcript-to-turns.awk — Turn-group transcripts (user→next-user boundary) before BM25
embed-events.js — Batch index JSONL events into Qdrant
semantic-search.js — Query Qdrant by vector similarity
index-event.sh — Real-time single-event indexing (called by hooks)
record-wrapup.sh — Durable wrapup write + verified index receipt
wrapup-coverage.js — Derived material-session coverage + pending queue
cartographer-standup.js — Peer-session roster + project/file contention (/standup)
backfill-git-history.sh — Import git commits into event logs
backfill-memories.sh — Index Codex memory files
backfill-app-sessions.js — Import desktop-app/Cowork session metadata (titles + VM-session prompts)
retro-index.sh — Backfill historical transcripts into Qdrant (turn-grouped)
catch-up-transcripts.sh — Checkpointed/cooldown transcript refresh used by SessionStart
infer-codex-project.js — Derive Codex repo attribution from tool workdirs when cwd is generic
reconstruct-history.js — Deep transcript analysis for backfill (turn-grouped)
classify-prompt-intent.js — Rule-based prompt-intent classifier (17 categories, zero-dep)
backfill-prompt-intents.js — Tag already-indexed turns with prompt_intent (payload-only, no re-embed)
prompt-intent-report.js — Corpus-wide intent distribution + per-bucket sampling (retuning tool)
hit-rate-report.js — Joins served-log.jsonl + access-ledger.jsonl: search hit rate by rank/source/project
project-registry.json — Project aliases for multi-repo families (used by search + orientation)
plugins/session-cartographer/
skills/remember/SKILL.md — /remember skill (Codex's context recovery tool)
skills/focus/SKILL.md — retirement stub pointing to /remember orientation
skills/carto/SKILL.md — /carto skill (launches Explorer web app for humans)
skills/wrapup/SKILL.md — /wrapup skill (strategic session-end preservation)
scripts/remember-search.sh — Legacy keyword-only search (superseded by cartographer-search.sh)
hooks/
hooks.json — Hook registrations, including background transcript catch-up
log-research.sh — WebFetch/WebSearch → research-log.jsonl + changelog.jsonl
log-session-milestones.sh — Compactions, session ends, agent stops (with git context)
log-tool-use.sh — Edit/Write/Bash + git commits with classification (opt-in)
explorer/
server/
index.js — Express API (:2526), SSE stream, search proxy
bm25.js — BM25 scorer (JS port for API path)
search.js — Hybrid search: BM25 + Qdrant proxy + RRF
jsonl.js — Resilient JSONL reader with fs.watch
src/ — React 19 + Vite + Tailwind UI (:2527)
docs/
RANK_FUSION.md — BM25 + RRF scoring architecture
SCORING.md — Score interpretation guide
SETUP.md — Install, Qdrant, cold start backfill, disk usage
MIGRATION_TURNS.md — Existing-user guide for turn-based transcript migration
CUSTOM_HOOKS.md — How to log your own events
EXPLORER_SPEC.md — Explorer implementation spec
companion_explorer_spec.md — Explorer product spec
CHANGELOG_SPEC.md — Event log format
landscape-survey.md — 30+ Codex memory projects compared
tests/private/ — Gitignored: test cases, fixtures, benchmarks
```
## Implementation Constraints — READ THESE
- **BM25 in awk is intentional.** `bm25-search.awk` is the CLI search scorer. Zero dependencies (no Node, no jq). Do not port to Python. The JS port in `explorer/server/bm25.js` exists separately for the API path.
- **Turn-grouping is an indexing-layer concern, not a query-layer concern.** `transcript-to-turns.awk` preprocesses transcripts into one document per turn (user prompt + assistant responses up to the next user prompt) and is used by `retro-index.sh` + hooks when ingesting into Qdrant. Event IDs are deterministic (`turn-<sid>-<idx>`). Do not run it per-query — the CLI keyword `--transcript` fallback is plain per-line grep by design; semantic search (Qdrant) owns turn-coherent transcript recall. See `docs/MIGRATION_TURNS.md` for historical context.
- **Field extraction uses a fallback chain** (`summary → description → prompt → url → query → event_id → milestone`) across diverse JSONL schemas. Do not hardcode a single field.
- **Event summaries are single-line.** Writers flatten `\n`/`\t` at the source (`tr` in hooks, gsub in the semantic TSV emitter, `index-event.sh`). The pipeline is TSV/line-based: a multi-line summary splits a TSV row, and the fragments mis-parse as rank/key/timestamp — rank coerces to 0 and outranks every real result. The fusion awk drops rows with an empty key or non-numeric rank as a backstop. Pretty-printed JSON must never be appended to the JSONL logs.
- **Transcripts are first-class citizens in RRF.** They compete equally with event log results. Do not append them at the bottom.
- **`LC_ALL=C` on grep and awk** prevents multibyte errors on unicode in JSONL.
- **Transcript search uses `find -exec grep {} +`** to batch file matching in one process. Do not revert to per-file subprocess loops (1,839 files = 40x slower).
- **Hooks call `index-event.sh` for real-time Qdrant indexing.** Qdrant remains optional, but failures are recorded in `.carto/index-errors.jsonl` and return nonzero so backfills do not checkpoint incomplete sessions. Novelty rejects are recorded separately in `.carto/index-rejects.jsonl`; synchronous callers opt into a JSON receipt with `CARTOGRAPHER_INDEX_RECEIPT=1`.
- **Turbo opt-in is user-global and provider-neutral.** Claude Code and Codex read the same `~/.config/session-cartographer/config.json`; do not add separate agent settings as competing sources of truth. Ordinary recall may use HTTP or the sandbox-safe private file transport, while control operations retain the portable CLI path.
- **Turbo awareness is guidance, not automatic retrieval.** When the shared preference is active, SessionStart may remind the agent to use `remember` for tasks that depend on prior work. Do not turn that into unconditional startup search; measure deduplicated session exposures against explicit result use.
- **Explorer binds to 127.0.0.1 only.** Never 0.0.0.0. Binding does not stop DNS rebinding, so every listener (the API, headless Turbo, the UI host) also refuses a non-loopback `Host`, a foreign `Origin` and `Sec-Fetch-Site: cross-site` through `explorer/server/loopback-guard.js`; `api-loopback-guard.test.js` and `turbo-loopback-guard.test.js` send a forged Host to each port. Path traversal protection on transcript endpoints. Rendered content is never parsed as HTML, and there is no DOMPurify: markdown is emitted as React elements, nothing uses `dangerouslySetInnerHTML`, and every corpus URL that becomes an `href` (Memory Desk markdown links, `EventCard`'s event `url`) passes `safeArtifactLink` (`http(s)`, `mailto`, `#` only); a refused one renders as text. React 19 rewrites only a `javascript:` href, so it is not the guard; `event-card-links.test.js` fails if an `EventCard` href skips the allowlist. A render path that emits raw HTML must bring its own sanitizer. No browser code calls Qdrant (the API proxies it), so `start-services.sh` runs it with CORS off, because Qdrant's default reflects any `Origin`. CORS off stops a web page reading :6333, not writing to it — Qdrant 1.12.1 parses a preflight-free `text/plain` POST as JSON, so only an API key closes that, and the launcher provisions one: generated on first start into `$CARTOGRAPHER_QDRANT_STORAGE/api-key` (mode 600), written into its mode-600 config as `service.api_key`, and sent as the `api-key` header by every client through `scripts/qdrant-auth.js` or `qdrant-auth.sh`, which read that file (loopback URLs only) or `CARTOGRAPHER_QDRANT_API_KEY`. A new Qdrant client goes through one of the two, or it fails against a keyed Qdrant without erroring; `qdrant-api-key.test.js` lists every client and checks the header arrives, and `qdrant-start-services-key.test.js` sends the blind write to a launcher-started Qdrant and expects 401.
- **Ports:** 2526 (API), 2527 (UI), 6333 (Qdrant), 8890 (embeddings).
- **`project-families.json` is gitignored.** Run `generate-families.sh` to bootstrap from event logs.
- **Enrichment scripts modify `changelog.jsonl` in place.** Back up before running on large datasets.
- **Retrieval telemetry is exact for new calls.** Served rows and `--touch` records share `call_id`; purpose, session, and provider are carried alongside it. `access-ledger.jsonl` is append-only. Do not rewrite it in place, and keep the CLI/API activation implementations in sync. Legacy inference is report-only and opt-in.
- - **The noise trim runs on relevance, and decay is a switch, not a curve.** Both engines trim at 10% of the top RRF × salience score *before* any age weighting. Run after decay, as the API did until 2026-10-16, the trim cut old records for their age: 55 of 240 known items were absent from the pool, not merely ranked low. The CLI had no trim, so the engines also disagreed about the pool. Fusion is rank-based: rank 1 scores 1/61, rank 10 scores 1/70, and a rank-1 partial match in another ladder also scores 1/61. An old known item therefore survives the page only if its age factor stays within a few percent of 1. Measured on 204k events (240 known items, 40 recency cases, semantic off), known-item S@10 by age <7d/7-30d/30-90d/90d+ was exp λ=0.001 0.98/0.85/0.62/0.15; power-law t^-0.5 0.98/0.77/0.62/0.30; ACT-R base-level d=0.05 0.80/0.53/0.60/0.72, trading the week for the archive. So when ≤ `CARTOGRAPHER_DECAY_LOOKUP_MAX` (10) candidates hold every query token, those candidates skip decay: 0.98/1.00/0.98/1.00 on the API, with recency unchanged. On the CLI, measured on paired cases, past 90 days went 0.23 → 1.00. Everything else decays as before. "Holds every token" is presence-only (`full` on `scoreBM25` results, column 10 of `bm25-search.awk`), so the engines agree although their idf differs. `tests/unit/age-curve-gate.test.js` proves its fixture reproduces the loss before gating, and `scripts/age-curve.js` runs the same curve on the live corpus. Change λ, the trim, salience or the exemption only with both. A different browse curve also has to pass the topical kind (`age-curve.js --truth`, on the live corpus and `--as-of` the labels' date), because both gate metrics reward steeper decay. On 2026-10-17 every power-law form tested, ACT-R base-level included, lost topical P@10 there (0.40-0.42 against exp's 0.48), so exp stays (docs/SCORING.md). The truth scorer is `scripts/truth-score.js`, shared with `eval-search.js`; its word-overlap match needs two shared words, because at one it credited any recent row sharing a word with a short label, which is exactly what a steeper curve puts on the page.
+ - **The noise trim runs on relevance, and decay is a switch, not a curve.** Both engines trim at 10% of the top RRF × salience score *before* any age weighting. Run after decay, as the API did until 2026-10-16, the trim cut old records for their age: 55 of 240 known items were absent from the pool, not merely ranked low. The CLI had no trim, so the engines also disagreed about the pool. Fusion is rank-based: rank 1 scores 1/61, rank 10 scores 1/70, and a rank-1 partial match in another ladder also scores 1/61. An old known item therefore survives the page only if its age factor stays within a few percent of 1. Measured on 204k events (240 known items, 40 recency cases, semantic off), known-item S@10 by age <7d/7-30d/30-90d/90d+ was exp λ=0.001 0.98/0.85/0.62/0.15; power-law t^-0.5 0.98/0.77/0.62/0.30; ACT-R base-level d=0.05 0.80/0.53/0.60/0.72, trading the week for the archive. So when ≤ `CARTOGRAPHER_DECAY_LOOKUP_MAX` (10) candidates hold every query token, those candidates skip decay: 0.98/1.00/0.98/1.00 on the API, with recency unchanged. On the CLI, measured on paired cases, past 90 days went 0.23 → 1.00. Everything else decays as before. Skipping decay keeps a named record on the page, not first: a recent 0.7–0.8-salience record sharing part of the query outranks a 0.25–0.5 one holding all of it. So the named candidates are also multiplied by `CARTOGRAPHER_DECAY_LOOKUP_LIFT` (3) after decay, as a score change so `--min-score` still cuts a sorted list. Live known S@1 went 148 → 221 of 240, S@10 234 → 240, no case lower, recency and topical identical (docs/SCORING.md, "Lifting named records"). "Holds every token" is presence-only (`full` on `scoreBM25` results, column 10 of `bm25-search.awk`), so the engines agree although their idf differs. `tests/unit/age-curve-gate.test.js` proves its fixture reproduces the loss before gating, and `scripts/age-curve.js` runs the same curve on the live corpus. Change λ, the trim, salience, the exemption or the lift only with both. A different browse curve also has to pass the topical kind (`age-curve.js --truth`, on the live corpus and `--as-of` the labels' date), because both gate metrics reward steeper decay. On 2026-10-17 every power-law form tested, ACT-R base-level included, lost topical P@10 there (0.40-0.42 against exp's 0.48), so exp stays (docs/SCORING.md). The truth scorer is `scripts/truth-score.js`, shared with `eval-search.js`; its word-overlap match needs two shared words, because at one it credited any recent row sharing a word with a short label, which is exactly what a steeper curve puts on the page.
- **Diff-shape quadrant labels:** bootstrap, construct, surgical, rework. Not "dangerous."
## Two Search Paths
1. **CLI** (`cartographer-search.sh`): bash + awk BM25. Used by `/remember` skill. No server needed.
2. **API** (`explorer/server/`): JS BM25 + Express. In-memory index, but `scoreBM25` scans every document: 20–31 ms median per query over ~203k events (measured 2026-10-16). Used by the Explorer UI (`/carto`) and Turbo. Proxies Qdrant for semantic search.
Both use BM25 (k1=1.2, b=0.75) and RRF (k=60), but they are not one ranker. The CLI computes IDF per log file, and the API over the deduplicated corpus. The CLI sums RRF credit for both copies of a dual-written event, and the API credits one. Both trim under 10% of the top score before decay. The awk timestamp parser ignores UTC offsets, and the API's does not. CLAUDE.md lists each difference.
## How Codex should use this
When a user says "remember X" or needs context from a past session, use `/remember`. The skill runs the search, returns ranked results with transcript paths. **Read the transcript** to recover full context — the search result is the map, the transcript is the territory.
When a user says "explore" or wants to browse history visually, use `/carto` to start the web app and open the browser. That's a human tool — don't try to scrape it.
At the end of a productive session — or when a user says "wrap up" — use `/wrapup` to synthesize decisions, discoveries, and open threads into a milestone event. The milestone gets indexed and becomes searchable via `/remember`.
## Testing
Read [docs/TESTING.md](docs/TESTING.md) before writing tests. Assert on result
composition and provenance, not merely successful responses.
- `env -u CLAUDE_SESSION_ID -u CLAUDE_CODE_SESSION_ID -u CODEX_SESSION_ID -u CARTOGRAPHER_SESSION_ID node --test tests/unit/*.test.js` — portable unit suite; use Node 22 to match CI.
- `npm run build --prefix explorer` and `node tests/browser/memory-entry.cjs` — Explorer build and isolated headless Chromium regression; install locked root/Explorer dependencies and Chromium first as documented in the testing guide.
- `bash tests/release-smoke.sh`, `bash tests/source-marketplace-smoke.sh` — packaging and available CLI installation checks.
- `tests/private/` is gitignored and optional, not a fresh-clone prerequisite.
- **No MCP-based browser testing.** Use the checked-in headless browser harness and manual visual review; don't puppet Chrome via desktop-control MCP.