git:20261003.b75142d to git:20261016.3f80b5e

12 added, 0 removed. Audit A to A.

# AGENTS.md

Guidance for AI agents and human contributors working in this repository. This file follows the tool-agnostic [AGENTS.md](https://agents.md) convention, so it is read by Claude Code, Codex, Cursor, Copilot, and any agent that honors it. For local toolchain rough edges and the release flow, see [CONTRIBUTING.md](CONTRIBUTING.md).

## Project

token-goat is a Claude Code / Codex CLI companion, written in TypeScript and bundled to `dist/token-goat.mjs`, that reduces token burn through three mechanisms: image shrinking, session-aware read hints, and surgical reads (CLI commands that extract one symbol or section instead of a whole file).

## Build, test, lint

`--cmd-b64` is strictly an internal hook transport flag, generated programmatically by token-goat hooks. Agents and interactive users must pass ordinary plain shell commands directly, using positional arguments or `--cmd`; never synthesize base64 manually, because character corruption can change commands and file paths.

```bash

npm install

npm test # full test suite

npm run test:guards # fast I/O-free structural guards (tests/guards)

npm run test:matrix # built-bundle command matrix (slow, pre-push/CI tier)

npm run typecheck # tsc --noEmit

npm run lint # ESLint

npm run docs:arch # sync CLAUDE.arch.md with src/ modules

npm run docs:arch:check # verify zero architecture drift

```

Tests run in two tiers:

- **`pre-commit` (fast, ~2s)** — lint + typecheck + `npm run test:guards`. The guards are pure-introspection invariants in `tests/guards/` (no bundle build, no DB, no git fixtures). They catch the structural bug class — an implemented-but-unregistered or unfunctional command — *before the commit lands*.

- **`pre-push` / CI (full)** — the entire suite, including the built-bundle command matrix (`tests/command_matrix_e2e.*.test.ts`), which indexes a real fixture and runs every registered command against the shipped `dist/token-goat.mjs`, asserting real output.

Both tiers derive their command set from one source, [tests/registry.ts](tests/registry.ts) (`allCommandNames()`), so a newly registered command is automatically in scope for the guard and the matrix — there is no second list to forget. **Every user-facing command must appear in the matrix**; a registered command with no matrix case fails the coverage gate by design.

**Tests must be deterministic.** Do not dismiss intermittent failures as environmental noise or waive them with retries. Find and remove the source of timing, shared-state, ordering, or resource-contention flakiness; a test is complete only when it passes reliably in its real full-suite execution mode.

## The indexer and worker are the critical path — prioritize them

The indexer (`src/parser.ts`: parse, then write symbols/refs/sections, plus embeddings) and the worker that drives it (`src/worker.ts`: drain the dirty queue, reindex changed files) are the core of the product. Every surgical-read command — `symbol`, `read`, `skeleton`, `outline`, `semantic`, read-dedup — returns nothing if this pipeline is broken. Treat it as the **highest priority for tests, bug-fixing, and improvements**, above hooks, image-shrinking, formatting, and CLI ergonomics, when triaging or planning work.

Test it on the *real* wiring. A release once shipped with the worker draining the queue into a default **stub** callback, so nothing ever wrote to the `symbols` table and the parser was tree-shaken out of the built bundle — yet the suite was green, because every worker test injected its own callback (exercising the loop, never the production default path), the parser was only unit-tested in isolation, and nothing ran the built bundle. So any change touching the worker or indexer needs:

- an **end-to-end test on the real default path** (drain → index → `symbols` populated → a known symbol resolves), not a mock-callback test, and

- a **smoke test against the built `dist` bundle**, to catch code that gets tree-shaken out of the shipped artifact.

### The index must stay bounded

Symbol *bodies* are the one thing in the index that can grow superlinearly, and doing so takes the tool down rather than degrading it. `extractJsonSymbols` once stored each top-level key's whole *source line* as its body; minified JSON puts every key on line 1, so a 1.5 MB / 1142-key file stored 1.6 GB. `global.db` reached 2.9 GB, and reindexing that single file held SQLite's writer lock past db.ts's 15s `busy_timeout` — surfacing as `database is locked` and as multi-minute stalls during `token-goat index`.

Three rules follow, and each is enforced by a test:

1. **One write path.** Every parsed symbol reaches the DB through the single `INSERT INTO symbols` in `writeParseResult`, which bounds the body at `MAX_SYMBOL_BODY_CHARS`. Do not add a second insert path — that is how the cap gets bypassed by someone who does not know it exists. Bounding at the choke point rather than in each extractor is what makes a future unbounded-body bug in *any* language incapable of bloating the index.

2. **Elide, never truncate.** An over-cap body is stored `''`. `read_commands.ts`'s `resolveBody` re-slices an empty body from source over `[line_start, line_end]`, so the cap costs nothing on read. A *truncated* body is non-empty, so it defeats that fallback and `read`/`symbol`/`brief` would serve a prefix as though it were the whole symbol while `line_end` still advertised the full range — bounded disk bought with a silent correctness regression.

3. **Amplification is a test, not a hope.** Stored bytes for a file must stay ≤ 4× its size, measured through the real pipeline.

Guards: [tests/guards/symbol_body_bound.test.ts](tests/guards/symbol_body_bound.test.ts) (pre-commit, source introspection) and [tests/index_amplification_guard.test.ts](tests/index_amplification_guard.test.ts) (pre-push/CI, real files → real DB; reports `1200.0x` on the pre-fix parser). Runtime: `token-goat doctor` warns past 1 GB and points at `token-goat reclaim-index --rebuild`.

The failure mode to watch for is the **injected-seam trap**: a test always supplies the dependency that the shipping path omits, so the production default is never exercised and the suite stays green over a broken feature. It applies wherever behavior hides behind an injectable callback or a default parameter.

+ ## No forever ghost or orphan processes

+ Nothing token-goat runs, and nothing its tests or scripts run, may outlive its owner. Shells killed on a timeout left the `find /` they had started walking the disk for hours, and the worker daemons of 2.9.30, which never exited on idle, piled up (one report: 19 processes, about 75 CPU-hours). Three rules follow.

+ - **Bound every walk.** No code, test or script walks a filesystem root or the home directory without a depth or count limit. The `hook` command refuses a shell search that starts at a root or the home directory.

+ - **Reap every tree you spawn, on every path.** A child is ended with its whole tree, on success, failure, timeout and signal alike: in `finally` or `afterEach`, never only on the happy path. In `src/`, start a child through `spawnOwned` in `src/process_tree.ts`, which ends the tree (freeze then kill on Linux and macOS, `taskkill /T` on Windows) when the caller asks, when token-goat exits, and on INT, TERM or HUP. `src/bash_runner.ts` and the image-text helper in `src/image_ocr.ts` use it. In tests, the shell oracle in `tests/helpers/shell_argv.ts` tags every process of a run with a `TG_SHARGV_OWNER` token in its environment and, after a failed run, kills everything carrying that token, so a subshell a timeout orphaned is still found.

+ - **A detached helper carries a death condition tied to its owner.** A child started `detached` or released with `.unref()` must say, in a single-line comment on the line itself or the line above, who owns it and what ends it: `// owner: <who> ... dies when <condition>`. `tests/guards/background_spawns_name_their_death.test.ts` fails a site without that comment. Two residents are the deliberate exceptions, because each outlives the process that started it: the worker daemon ends when its pid file or data directory is gone or names another process, after six hours idle (`TG_WORKER_IDLE_EXIT_MS` changes that), or on `worker stop`; the hook server ends after 30 minutes idle, when its key file is deleted, or when its build is replaced.

+ Cleanup of strays matches by command line and launcher path, never by process name: `upgrade` and `doctor --fix` end the worker daemons and hook servers of the same install through `src/stray_residents.ts`, and leave every other process alone. Kill only a process tree you started, never one you did not.

+ None of this covers token-goat being killed outright (SIGKILL, or TerminateProcess on Windows): no handler runs then, so a child started before that can survive.

## Security boundaries

Six invariants govern redaction, fencing, persisted keys, untrusted config, and path confinement, each established by a defect that shipped past a green suite. Read them before changing any of those surfaces: `token-goat section "CLAUDE.arch.md::Security Boundaries"`, or the same heading in [CLAUDE.arch.md](CLAUDE.arch.md). A change touching one needs a test that fails when the invariant is removed, not just a green suite.