AGENTS.md git:20261025.fe7e603A

Immutable. This exact content never changes and is served at /api/v1/blob/e973a654fb59861e.

# AGENTS.md

Guidance for AI agents and human contributors working in this repository. This file follows the tool-agnostic [AGENTS.md](https://agents.md) convention, so it is read by Claude Code, Codex, Cursor, Copilot, and any agent that honors it. For local toolchain rough edges and the release flow, see [CONTRIBUTING.md](CONTRIBUTING.md).

## Project

token-goat is a Claude Code / Codex CLI companion, written in TypeScript and bundled to `dist/token-goat.mjs`, that reduces token burn through three mechanisms: image shrinking, session-aware read hints, and surgical reads (CLI commands that extract one symbol or section instead of a whole file).

## Build, test, lint

`--cmd-b64` is strictly an internal hook transport flag, generated programmatically by token-goat hooks. Agents and interactive users must pass ordinary plain shell commands directly, using positional arguments or `--cmd`; never synthesize base64 manually, because character corruption can change commands and file paths.

```bash
npm install
npm test            # full test suite
npm run test:guards # fast I/O-free structural guards (tests/guards)
npm run test:matrix # built-bundle command matrix (slow, pre-push/CI tier)
npm run typecheck   # tsc --noEmit
npm run lint        # ESLint
npm run docs:arch   # sync CLAUDE.arch.md with src/ modules
npm run docs:arch:check # verify zero architecture drift
```

Tests run in two tiers:

- **`pre-commit` (fast, ~2s)** — lint + typecheck + `npm run test:guards`. The guards are pure-introspection invariants in `tests/guards/` (no bundle build, no DB, no git fixtures). They catch the structural bug class — an implemented-but-unregistered or unfunctional command — *before the commit lands*.
- **`pre-push` (focused)** — lint, typecheck, then the tests the pushed range touches (the failed, changed and guard tests, the built-bundle matrix when `src/` changed, then importers of the changed modules nearest first) within a time budget, listing what it defers to CI; any doubt, or `PREPUSH_FULL=1`, runs the full suite (see CONTRIBUTING.md, "Lefthook").
- **CI (full)** — the entire suite, including the built-bundle command matrix (`tests/command_matrix_e2e.*.test.ts`), which indexes a real fixture and runs every registered command against the shipped `dist/token-goat.mjs`, asserting real output.

Both tiers derive their command set from one source, [tests/registry.ts](tests/registry.ts) (`allCommandNames()`), so a newly registered command is automatically in scope for the guard and the matrix — there is no second list to forget. **Every user-facing command must appear in the matrix**; a registered command with no matrix case fails the coverage gate by design.

**Tests must be deterministic.** Do not dismiss intermittent failures as environmental noise or waive them with retries. Find and remove the source of timing, shared-state, ordering, or resource-contention flakiness; a test is complete only when it passes reliably in its real full-suite execution mode.

## The indexer and worker are the critical path — prioritize them

The indexer (`src/parser.ts`: parse, then write symbols/refs/sections, plus embeddings) and the worker that drives it (`src/worker.ts`: drain the dirty queue, reindex changed files) are the core of the product. Every surgical-read command — `symbol`, `read`, `skeleton`, `outline`, `semantic`, read-dedup — returns nothing if this pipeline is broken. Treat it as the **highest priority for tests, bug-fixing, and improvements**, above hooks, image-shrinking, formatting, and CLI ergonomics, when triaging or planning work.

Test it on the *real* wiring. A release once shipped with the worker draining the queue into a default **stub** callback, so nothing ever wrote to the `symbols` table and the parser was tree-shaken out of the built bundle — yet the suite was green, because every worker test injected its own callback (exercising the loop, never the production default path), the parser was only unit-tested in isolation, and nothing ran the built bundle. So any change touching the worker or indexer needs:

- an **end-to-end test on the real default path** (drain → index → `symbols` populated → a known symbol resolves), not a mock-callback test, and
- a **smoke test against the built `dist` bundle**, to catch code that gets tree-shaken out of the shipped artifact.

### The index must stay bounded

Symbol *bodies* are the one thing in the index that can grow superlinearly, and doing so takes the tool down rather than degrading it. `extractJsonSymbols` once stored each top-level key's whole *source line* as its body; minified JSON puts every key on line 1, so a 1.5 MB / 1142-key file stored 1.6 GB. `global.db` reached 2.9 GB, and reindexing that single file held SQLite's writer lock past db.ts's 15s `busy_timeout` — surfacing as `database is locked` and as multi-minute stalls during `token-goat index`.

Three rules follow, and each is enforced by a test:

1. **One write path.** Every parsed symbol reaches the DB through the single `INSERT INTO symbols` in `writeParseResult`, which bounds the body at `MAX_SYMBOL_BODY_CHARS`. Do not add a second insert path — that is how the cap gets bypassed by someone who does not know it exists. Bounding at the choke point rather than in each extractor is what makes a future unbounded-body bug in *any* language incapable of bloating the index.
2. **Elide, never truncate.** An over-cap body is stored `''`. `read_commands.ts`'s `resolveBody` re-slices an empty body from source over `[line_start, line_end]`, so the cap costs nothing on read. A *truncated* body is non-empty, so it defeats that fallback and `read`/`symbol`/`brief` would serve a prefix as though it were the whole symbol while `line_end` still advertised the full range — bounded disk bought with a silent correctness regression.
3. **Amplification is a test, not a hope.** Stored bytes for a file must stay ≤ 4× its size, measured through the real pipeline.

Guards: [tests/guards/symbol_body_bound.test.ts](tests/guards/symbol_body_bound.test.ts) (pre-commit, source introspection) and [tests/index_amplification_guard.test.ts](tests/index_amplification_guard.test.ts) (pre-push/CI, real files → real DB; reports `1200.0x` on the pre-fix parser). Runtime: `token-goat doctor` warns past 1 GB and points at `token-goat reclaim-index --rebuild`.

The failure mode to watch for is the **injected-seam trap**: a test always supplies the dependency that the shipping path omits, so the production default is never exercised and the suite stays green over a broken feature. It applies wherever behavior hides behind an injectable callback or a default parameter.

## No forever ghost or orphan processes

Nothing token-goat runs, and nothing its tests or scripts run, may outlive its owner. Shells killed on a timeout left the `find /` they had started walking the disk for hours, and the worker daemons of 2.9.30, which never exited on idle, piled up (one report: 19 processes, about 75 CPU-hours). Three rules follow.

- **Bound every walk.** No code, test or script walks a filesystem root or the home directory without a depth or count limit. The `hook` command refuses a shell search that starts at a root or the home directory.
- **Reap every tree you spawn, on every path.** A child is ended with its whole tree, on success, failure, timeout and signal alike: in `finally` or `afterEach`, never only on the happy path. In `src/`, start a child through `spawnOwned` in `src/process_tree.ts`, which ends the tree (freeze then kill on Linux and macOS, `taskkill /T` on Windows) when the caller asks, when token-goat exits, and on INT, TERM or HUP. `src/bash_runner.ts` and the image-text helper in `src/image_ocr.ts` use it. In tests, the shell oracle in `tests/helpers/shell_argv.ts` tags every process of a run with a `TG_SHARGV_OWNER` token in its environment and, after a failed run, kills everything carrying that token, so a subshell a timeout orphaned is still found.
- **A detached helper carries a death condition tied to its owner.** A child started `detached` or released with `.unref()` must say, in a single-line comment on the line itself or the line above, who owns it and what ends it: `// owner: <who> ... dies when <condition>`. `tests/guards/background_spawns_name_their_death.test.ts` fails a site without that comment. Two residents are the deliberate exceptions, because each outlives the process that started it: the worker daemon ends when its pid file or data directory is gone or names another process, after six hours idle (`TG_WORKER_IDLE_EXIT_MS` changes that), or on `worker stop`; the hook server ends after 30 minutes idle, when its key file is deleted, or when its build is replaced.

Cleanup of strays matches by command line and launcher path, never by process name: `upgrade` and `doctor --fix` end the worker daemons and hook servers of the same install through `src/stray_residents.ts`, and leave every other process alone. Kill only a process tree you started, never one you did not.

None of this covers token-goat being killed outright (SIGKILL, or TerminateProcess on Windows): no handler runs then, so a child started before that can survive.

## Security boundaries

Six invariants govern redaction, fencing, persisted keys, untrusted config, and path confinement, each established by a defect that shipped past a green suite. Read them before changing any of those surfaces: `token-goat section "CLAUDE.arch.md::Security Boundaries"`, or the same heading in [CLAUDE.arch.md](CLAUDE.arch.md). A change touching one needs a test that fails when the invariant is removed, not just a green suite.