AGENTS.md · diff

git:20260912.e3c5f41 to git:20260913.f0ecd45

17 added, 44 removed. Audit A to A.

# AGENTS.md

Operational guidance for AI assistants (and humans) working **on** this repository. This

is the SOTA-skills library — Markdown skills that an AI assistant reads to build and

audit software. There is no application to run; changes are edits to Markdown held to a

few hard invariants. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full conventions.

This file is the single source of truth for every agent: tools that follow the

[AGENTS.md standard](https://agents.md) (Codex, Cursor, Copilot, …) read it directly,

while `CLAUDE.md` (Claude Code) and `GEMINI.md` (Gemini CLI) are symlinks to it — edit

only this file, never the symlinks.

## Landing a change

`main` is a protected branch and **direct pushes are rejected for everyone** (admin

enforcement is on). Every change goes through a pull request:

1. `git checkout -b <branch>`

2. make the edit, then run `./scripts/check-invariants.sh` (and optionally

`pre-commit run --all-files`). `pre-commit install` sets up **both** stages —

pre-push re-runs the invariants, the first moment the **diff-based** ones

(11, 14) have a commit to read

3. push the branch and open a PR

4. all four required checks must pass, then squash-merge — invariants, secret scan,

shell lint, and the negative-control harness that proves the gates can still fail

## Invariants (enforced in pre-commit and CI)

- `scripts/check-invariants.sh` runs **29 checks** and fails the build on any of them. One

- line each below. The *rationale* — and the real incident behind every one — lives in the

- script's own header, at the point of use, and the practical "what this means for your

- PR" version is in [CONTRIBUTING.md](CONTRIBUTING.md#the-invariants-enforced).

- | # | The build fails when… |

- |---|---|

- | 1 | a **skill** file (`skills/*/SKILL.md`, `skills/*/rules/*.md`) exceeds **500 lines** |

- | 2 | a `skills/*/rules/*.md` doesn't end with `## Audit checklist`, **or carries more than one** — the last-heading test passes on a duplicate, and five files shipped that way with six stranded bullets in the block a reader has already scrolled past |

- | 3 | an internal-name denylist hits (the library must stay generic) |

- | 4 | a `SKILL.md` `description` exceeds **1024 chars** (spec cap — loaders silently skip it), is unquoted YAML containing `: `, or either `name`/`description` contains an **XML tag**; also a reserved word (`anthropic`, `claude`) in `name` |

- | 5 | `VERSION`, `plugin.json` and the CHANGELOG top entry disagree, or a tag is ahead of `VERSION` |

- | 6 | a count-bearing surface drifts from a recount of `skills/` (the social-preview pill and README alt are **"N+" floors**) |

- | 7 | a skill is missing from the router's routing table (`skills/sota/SKILL.md`) **or** the library map (`skills/sota/rules/04`) — two files since the map was offloaded |

- | 8 | a relative Markdown link to a `*.md` target doesn't resolve |

- | 9 | `CHANGELOG.md` carries more than one `## [Unreleased]`, or it isn't the top entry |

- | 10 | a `rules/*.md` isn't referenced by its own `SKILL.md` — written, capped, checklist-ed, and never loaded |

- | 11 | `LAST-VERIFIED` moves without a sweep. Escapes: a sweep-shaped diff (≥ 20 skill files) or naming it in the CHANGELOG. The only **diff-based** check; skips with a note when there's no merge base |

- | 12 | an `assets/*.png` is older than the `*.html` it renders — the README embeds the *image*, never the source, so an un-rendered fix reaches nobody. Escape: `[no-render]` in the commit subject |

- | 13 | a scoreboard row in `evals/results/RESULTS.md` leaves its `Samples` cell empty |

- | 14 | a **release** (VERSION changed) carries no `**Front door checked:**` line in its CHANGELOG section, or a declared term is missing from `README.md`/`docs/INDEX.md` **or** from the release's own entry |

- | 15 | the **library map** (`skills/sota/rules/04`, offloaded out of the router) omits a `rules/NN` file that exists, or names one that doesn't — checks 7 and 10 both miss this, and `rules/11` went unlisted for two releases |

- | 16 | the hook `README.md` **documents** differs from the one `install.sh` **writes** (`HOOK_CMD`) — the README's is what a reader copies by hand, so a stale block is the version that spreads |

- | 17 | a document that **describes** the checks disagrees with them — a stated count that isn't the script's, or a restated negative-control coverage list that isn't the harness's. Counts inside `"quotes"` are read as history, not claims. **`README.md` joined the checked set 2026-09-10**, and counts spelled out in *words* are read too: it said "Twenty-five invariants" against a script with 29 for four releases, and neither the file nor the form was in scope |

- | 18 | a **`§` section reference** resolves nowhere — invariant 8 reads only `[text](file.md)` links, so ~1,300 prose references went unchecked and broke silently on any renumber or split |

- | 19 | a **check has no known-bad** and no pinned reason it cannot — and the exempt set may not *grow*, since silencing the coverage check by exempting your new check is a one-line move. Runs on every invocation (~50 ms), not behind a flag |

- | 20 | the router's **§AUDIT** section changes without its pin being re-read — §BUILD has been pinned since v1.15.0 and caught drift twice; §AUDIT had nothing, and `run-repo-audit.py` pastes the whole router. Bumping the pin is the forcing function to re-read `sota/rules/01` §5 |

- | 21 | a **CHANGELOG version below the top entry has no git tag** — invariant 5 checks a tag is never *ahead* of VERSION; nothing checked the other way, and v1.30.0 and v1.31.2 both shipped untagged and unreachable. The top entry is exempt: it is tagged after the merge |

- | 22 | a **`- [ ]` checklist bullet is stranded inside a code fence** in a skill file — it renders as sample output, so no auditor reads it. Two landed in `sota-code-security` rules/11 §2.2 in PR #226 and sat there three weeks; invariant 2 tracks fence state but only for the *heading*, and the line count never changed |

- | 23 | a **CHANGELOG version heading has no `[X.Y.Z]:` link reference**, an **orphan ref** whose heading lives in another file, or a ref pointing at the wrong tag — sibling of 21, failing independently of it. A heading with no ref is not a broken link: Markdown renders it as literal text, so invariant 8 never sees it. Four consecutive releases shipped that way. Compares **sets per file**, never counts |

- | 24 | **this file** reaches 200 lines, or `CLAUDE.md`/`GEMINI.md` stop being symlinks to it — it loads into *every* session, which is the whole reason for the cap, and that reason depends on the symlinks. A **cap, not a target** (settled 2026-09-06); it had already been breached twice, at 201 and 202, each time by adding an invariant's own table row |

- | 25 | the number of **eval CLI flags undocumented in `evals/README.md`** rises above the pin. A ratchet, not a rule: `--no-gate-arm` shipped documented in the README index and in `--help` but not in the harness's own front door, while 27 pre-existing generic flags (`--json`, `--build-model`) would make a strict rule open red and get it disabled. Fails closed if the scan finds **no** flags |

- | 26 | the roadmap's **open set disagrees with itself** — the stated count, the header's open list, the ledger rows marked `**OPEN`, or an item cited in the priorities table that is not open. Status lives in **one** place (the row marker); the rest is derived. Three drifts in one session, plus a fourth live when the check was written (header said 8, rows said 5) |

- | 27 | an **ADOPTION-LOG deferral names no revisit trigger** — its own rule said every deferral carries a revisit condition, in prose, and prose drifted: the roadmap said *"the deferred row"*, singular, while three existed and one had resolved the day before elsewhere. The marker's **own cell** is checked, not the row: a first draft passed a row because "revisit" sat in a different cell |

- | 28 | an **`evals/cases/*.jsonl` declares no `SELECTION RULE`** — building a set from the cases a model got wrong measures the selection, not the system. Held at 20 sets, then two were added and one skipped it: `prompt-independence.jsonl`, whose rule lived in the results doc and not in the case file, in the set backing the **+0.509** headline |

- | 29 | a **release changes a skill `description` and declares no routing check** — a description is the entire auto-load classifier, so adding one competes for every neighbour's traffic: `sota-skill-security` (v1.35.0) took `r1_token_count` from `sota-llm-engineering` 3/3 to 0/3 and was live for a day with invariants 4, 7 and 15 all green. Fires only on a release whose extracted description **map** differs from the merge base's; the escape is a `**Routing checked:**` line naming an artifact that must exist and mention the regression set |

+ `scripts/check-invariants.sh` runs **31 checks** and fails the build on any of them. One line

+ each — with the real incident behind every one — in **[docs/INVARIANTS.md](docs/INVARIANTS.md)**,

+ offloaded out of this file on 2026-09-13 because each new invariant cost a line of the

+ always-loaded budget and the cap had been breached by an invariant's own table row — twice landing at 201 and 202, and hitting exactly 200 twice more while editing this session. **The precise total is not recoverable from git**, because the gate catches a breach pre-commit so it never lands; what is recoverable is that every one was a table row. Read it before changing a gate. The

+ full *rationale* lives at the point of use in the script's own header; the practical "what this

+ means for your PR" version is in [CONTRIBUTING.md](CONTRIBUTING.md#the-invariants-enforced).

**Only instruction files are capped** — a file is capped iff an agent loads it *as instructions*:

`skills/*/SKILL.md` and `skills/*/rules/*.md`, nothing else. README, CHANGELOG, `docs/`, `evals/`

and every script are **uncapped**, deliberately (2026-07-15) — navigability there comes from

[docs/INDEX.md](docs/INDEX.md), not a ceiling. **A line-cap claim anywhere that does not say

*skill files* is stale — fix it.** The 500 matches the Agent Skills guidance (*"keep `SKILL.md`

under 500 lines; move detailed reference material to separate files"*) — `rules/*.md` are those.

**This file is the exception**: `CLAUDE.md`/`GEMINI.md` symlink here, so it loads into **every**

session, where the guidance is *"target under 200 lines"* — long always-loaded files reduce

- adherence. Keep it under 200 (**re-check with `awk 'END{print NR}'` each cut**, breached four

- times now, every time by adding an invariant's own row), detail to `CONTRIBUTING.md`.

+ adherence. Keep it under 200 (**re-check with `awk 'END{print NR}'` each cut** — breached only ever by an invariant's

+ own table row, which is why that table now lives in `docs/INVARIANTS.md`), detail to `CONTRIBUTING.md`.

**Every file-list-driven check reports its denominator** (`ok (N rules files)`) and **fails

closed on an empty scope** — `0 checked, 0 failed, exit 0` is the signature of a gate that

verifies nothing (`sota-code-security` rules/11 §2.2). Added 2026-07-30 after checks 2 and 10

printed `ok` over *zero* files; 4 and 8 were retrofitted only on 2026-08-16, so this sentence was

itself false for a while. The script's header carries the three rules it produced: **watch it

fail first, print your denominator, skip rather than guess.**

*Adding a `rules/NN` file?* Invariant 10 checks its `SKILL.md` indexes it, **invariant 15** that

the library map (`skills/sota/rules/04`) lists it — both directions. `skills/sota/SKILL.md` is at

- **420/500** (re-counted 2026-09-12, wrong a SEVENTH time; offloads got it there — **detail belongs in `rules/`,

- imperatives in the router**). Wrong **six** times — **re-count with `grep -c ''`**. Editing the

- router's **BUILD section** moves `ROUTER_BUILD_SHA` and aborts the evals; AUDIT does not. The

+ **under 500 — `grep -c '' skills/sota/SKILL.md` before you assume headroom.** The number used to be

+ written here and was wrong **nine times**, including twice in one day and once in this very sentence,

+ whose two halves disagreed with each other. It is deleted rather than corrected: a figure that rots on

+ every edit, in a file loaded into every session, with a one-command source of truth, is pure liability.

+ **Detail belongs in `rules/`, imperatives in the router.**

+ Editing the router's **BUILD section** moves `ROUTER_BUILD_SHA` and aborts the evals; AUDIT does not. The

gates enumerate via `git ls-files`, so an **unstaged new file is invisible** — `git add` first.

**`scripts/check-negative-controls.sh` proves our gates can still fail.** Its CI job runs it plus

`evals/smoke-runners.py` over **two** subjects — `check-invariants.sh` (part A) and

`verify-setup.sh` (part B). Each probe injects a known-bad and requires *the intended check* to

complain; any other non-zero exit is a **FALSE PASS**. **It reads the COMMITTED tree**

(`git worktree add HEAD`) — commit first, or you test a new script against old docs. Part A mutates

a good tree in a disposable worktree; part B is inverted, building a fully-configured fake machine

- (`CLAUDE_CONFIG_DIR` + throwaway repo + stub `gh`) and removing one thing per probe. **44 probes** (re-run 2026-09-12: `PASS: 44/44`; it said 43 from a 2026-09-09 run)

+ (`CLAUDE_CONFIG_DIR` + throwaway repo + stub `gh`) and removing one thing per probe. **49 probes** (re-run 2026-09-13 after invariants 30 and 31: `PASS: 49/49`; it said 44, and 43 before that)

(deliberately **not** gated — a static count of call sites under-reads, so only running it is

authoritative): invariants **1, 2, 3, 4, 6, 7, 8, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,

- 24, 25, 26, 27, 28, 29** — 26 of 29 — and verify-setup checks 1, 2, 3, 4, 6a, 6b, 7, 8, 9, 9a, 10a, 13.

+ 24, 25, 26, 27, 28, 29, 30, 31** — 28 of 31 — and verify-setup checks 1, 2, 3, 4, 6a, 6b, 7, 8, 9, 9a, 10a, 13.

Only **5, 9, 12** are unprobed, needing a tag or an mtime, and the harness prints that reason. *A

diff-based check is not unprobeable*: 11 and 14 were exempt on that false ground until a probe

**committed** its mutation (2026-09-09). **A probe asserts its own mutation landed** (a stale

literal once printed `NOT CAUGHT: INERT`, accusing a healthy gate) and a catch for the wrong

reason is refused — probe 21 was a FALSE PASS on its first draft. Adding a check? **Invariant 19

already enforces that it has a known-bad.** `--self-test` runs the suite and then this harness.

Separately, `scripts/check-freshness.sh` (run monthly by `.github/workflows/freshness.yml`)

tracks the root `LAST-VERIFIED` stamp — the date of the last full-library re-verification sweep

against primary sources. Update it only after such a sweep; the run goes red past the **6-month**

window. Per-file line-1 markers are retired. Sweep runbook and eval harness:

[docs/MAINTENANCE.md](docs/MAINTENANCE.md) and [evals/](evals/).

Secrets are scanned by **gitleaks** (`.gitleaks.toml` disables only the noisy

entropy-based `generic-api-key` rule, so the security skills' intentional

secret-shaped examples don't false-positive). CI scans the **full git history**

(`gitleaks git` on a `fetch-depth: 0` checkout), not just the working tree, and

**asserts that scope**: a shallow checkout scans 1 commit, reports "no leaks

found", and exits 0, so the workflow fails on a shallow clone rather than trusting

the setting. The pre-commit hook scans each commit locally.

## Conventions that matter

- **Keep it generic.** Never commit personal or company-specific stacks or project

names, and never phrase guidance as an assumption about the reader's setup. Products

appear only as neutral examples ("e.g. PostgreSQL").

Personalization lives in a local `profiles/<you>.md`, which is git-ignored

(`profiles/*` except `profiles/example.md.template`) and must never be

committed.

- **Verify claims.** Fast-moving facts (versions, specs, advisories) are checked

against a primary source and cited; uncertain items are marked

"needs verification", never asserted.

- **No rot-prone version pins.** Skills never claim "the current release is

X.Y" — write "latest stable" and tell the reader to verify at the official

source. Version numbers mark **semantic boundaries only** ("GA since",

"introduced/fixed/removed in", CVE fix versions, spec editions). When a

recommended tool goes EOL/unmaintained, replace it with the maintained

successor (project-recommended target first, then CNCF), keeping a one-line

EOL note for auditors. (Policy since the 2026-07-08 freshness sweep.)

- **Skill anatomy.** `skills/sota-<domain>/SKILL.md` (two-field frontmatter —

`name` + `description`; BUILD/AUDIT workflows; top-10 non-negotiables; a rules

index) plus `rules/NN-topic.md` files, each ≤ 500 lines and ending in an

`## Audit checklist`. Audit findings use the format

`file:line | rule | severity | effort | fix`.

## Pointers

- [docs/INDEX.md](docs/INDEX.md) — **find-it-fast index**: where every topic is

documented, organized by what you're trying to do (start here if lost)

- [docs/CONTEXT-MANAGEMENT.md](docs/CONTEXT-MANAGEMENT.md) — how the library keeps

the model applying rules as context fills (re-injection hook, principle 5,

terminal re-read, gates) + the decay measurement

- [evals/results/RESULTS.md](evals/results/RESULTS.md) — consolidated scoreboard of

every measured number

- [evals/README.md](evals/README.md) — the efficacy harness: what each case set

measures, how to run it, and the **harness conventions** (guards abort rather than

warn; watch a guard fail before trusting it; wait on a terminal artifact, not a log

substring; assert a scripted edit landed; pin anything hand-mirrored from the

library). Read it before changing anything under `evals/` — four harness changes in

one day silently measured nothing while still printing plausible numbers

- **The read-only setup check, in two halves** — `init-gates.sh` sets a repo up; these check the

result, because "configured" and "working" render identically. `scripts/verify-setup.sh` does the

mechanical half (skills reachable **vs the checkout's own count**, whether their descriptions fit the listing budget or arrive name-only, hook installed vs merely

configured, licence under any name, whether CI ever *executed* and ever *rejected*, and **§F: what

your searcher silently skips** — behavioural, INFO-only, the positive control the one thing that

can fail; `--runs N` widens the CI sample, `--reach-only` is what `install.sh` runs);

[docs/VERIFY-SETUP.md](docs/VERIFY-SETUP.md) is the paste-in prompt for the half a script cannot

do — whether the agent file's content is meaningful and whether its claims are still *true*

- [docs/ADOPTION-LOG.md](docs/ADOPTION-LOG.md) — the **external-idea intake

ledger**: every idea from an outside repo, paper or review, with a verdict and

reason (adopted / adopted-with-a-correction / rejected / deferred / superseded).

A recorded rejection stops the idea being re-litigated; `rejected: already

covered` must cite the file:line that covers it. Intake is not only external —

a session *applying* the library, and an unlicensed source whose ideas can be

taken but whose text cannot, both land here on the same terms

- [docs/CONVENTIONS-LEDGER.md](docs/CONVENTIONS-LEDGER.md) — which of this repo's

- conventions are **enforced** (29 invariants + 9 more inside the eval runners) and

+ conventions are **enforced** (31 invariants + 9 more inside the eval runners) and

which are prose, with the three filters a convention must pass to earn a gate

(has it already failed · does it fail silently · is it mechanically checkable).

Read it before proposing a new gate — it argues against gating the ~18 judgment

conventions, because a flaky gate gets disabled and leaves you worse off

- [CONTRIBUTING.md](CONTRIBUTING.md) — full contribution guide and PR checklist

- [RELEASING.md](RELEASING.md) — how to cut a release, including every

version- and count-bearing surface (README, router, manifests, social

preview)

- [docs/MAINTENANCE.md](docs/MAINTENANCE.md) — accuracy sweep runbook + eval harness

- [docs/WHY-IT-WORKS.md](docs/WHY-IT-WORKS.md) — the measured-efficacy case (lift **vs.

an unguided model**, plus a scoped head-to-head vs. named competing libraries) + the

design benefits; keep its numbers in sync with the eval results when they change

- [docs/WHY-COMPLETENESS-RESIDUAL.md](docs/WHY-COMPLETENESS-RESIDUAL.md) — why a

with-library build still drops a cross-cutting rule now and then (a salience /

context-length attention effect, **not** a coverage gap) + the counter-design

- [SECURITY.md](SECURITY.md) — reporting bad guidance or a leaked secret

- [CHANGELOG.md](CHANGELOG.md) — release history (top entry = current version;

also mirrored in `VERSION`); older releases are archived to keep every file

for navigability (CHANGELOG is no longer line-capped, so archiving is now

optional hygiene, not forced): **1.10.0–1.5.0** in

[docs/CHANGELOG-archive.md](docs/CHANGELOG-archive.md) and **1.4.0 and earlier**

in [docs/CHANGELOG-archive-2.md](docs/CHANGELOG-archive-2.md)