ai-raccoon-manual-checklist · diff
v1.0.0 to v1.0.0
4 added, 2 removed. Audit A to A.
---
name: ai-raccoon-manual-checklist
description: >-
Use when hand-verifying a live AiRaccoon build — a pre-flight or release checklist, a manual
smoke test after installing the global tool, a "does this actually work end to end" pass before
shipping, or any question `dotnet test` cannot answer because it needs a real install, a real
server and a real bank. Derives the version and tool surface from the product instead of pinning
them, records the command and the output behind every answer, and writes the filled checklist to
docs/work/checklist/.
version: 1.0.0
author: ai-badger
license: MIT
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [checklist, manual-testing, release, verification, evidence]
related_skills: [ai-raccoon-memory, review-changes]
---
# AiRaccoon manual checklist
The hand-run pass over a live build: the things `dotnet test` cannot answer because they need a
real install, a real server and a real bank.
A checklist is only worth the evidence behind it. The whole design below exists to keep a filled
checklist distinguishable from a plausible-sounding one — read "How this checklist rots" before
adding, removing or "tidying" anything here, because every failure listed there was found in a
real predecessor and each is easy to reintroduce.
## When NOT to use
- Anything `dotnet test` already covers. This pass is for behaviour that only appears in a real
install; duplicating unit or integration coverage by hand buys nothing and goes stale.
- Judging a diff or a PR — that is a code review, not a build verification.
- Debugging one failing symptom. Run the checklist to find out *what* is broken; trace *why*
somewhere else.
## Never
These three protect the person running the checklist, not the checklist:
- **Never write to the user's live bank** at `~/.ai-raccoon`. Read it only through `?mode=ro` /
`PRAGMA query_only=1`. The checklist's own writes go to a scratch data root you create with
`--data-root`.
- **Never bind the default port** (7721) for a step that starts a server — the user's own server
is usually on it, and `--restart` will cycle theirs. Use `--port 0` and read the bound port back.
- **Never mark an item accepted from a plan.** Accepted means you ran the command and read its
output.
## Process
1. Copy `templates/checklist-template.json` to
`docs/work/checklist/<yyyy-mm-dd>-<what-you-are-checking>.json`, creating the directory if
needed. Results live under `docs/work/checklist/` — never the repo root, and never
`.ai-raccoon/`, which is a bank directory, not a reports directory.
2. **Derive the facts the checklist compares against, before running anything**, and write them
into `derived`. Run `python3 scripts/derive-facts.py <repo-root>` (the root defaults to the
current directory); it prints the version, the MCP tool count and the prompt count, and it
fails loudly — exit 1, with the reason and a standing instruction on stderr — when any of them
comes back empty or zero, rather than handing you a confident `0`. A count typed from memory
or copied from a previous run is a second copy of a number that already exists, and it goes
stale silently.
The checklist then asserts that the *running binary* matches what the *tree* says. That
comparison is the point: it is the only step that can catch a build that installed something
other than what you think you are testing.
3. Work each item, filling in every field:
- `command` — the exact command or tool call you ran.
- `evidence` — the output you read, verbatim, trimmed to the deciding lines.
- `observed-result` — what it means.
- `status` — `pass`, `fail`, `skipped` or `substituted`. A `skipped` needs a reason; it is not
a pass. `substituted` means the behaviour was checked by a different instrument than the item
names — an automated test standing in for a path you could not drive live, say. Folding that
into `pass` overstates it and into `skipped` understates it, so it gets its own word, and the
reason must name the instrument. Record any deliberate deviation from the item's stated
method the same way.
- `accepted` + `acceptance-reason` — whether the observed result is acceptable, and why. A
`fail` may still be accepted as a known, tracked defect, as long as the reason names where it
is tracked.
`accepted` starts as `null`, meaning nobody has answered yet. Leave it null until you decide;
a run with any null left in it is unfinished, not a run with no objections.
4. **Never inherit a prior verdict's reason.** Where an item was `partial`, `fail`, `skipped` or
`substituted` last time, re-derive *why* against the source in front of you before reusing the
explanation. A previous run blamed a job's cadence for three event ids never firing; the real
cause was that they log only when there is something to purge, and a fresh bank has nothing.
The outcome matched, so the wrong mechanism survived a release — and it discouraged seeding the
data that would have exercised them. A reason that is right about the outcome and wrong about
the cause is the hardest kind of stale fact to see.
When a run finds that an earlier one was wrong, that belongs in `findings-against-prior-runs`,
not buried in an item. A checklist that can only describe the current build cannot report that
the last checklist lied, and those findings are often the most valuable thing a run produces.
5. Items whose feature no longer exists get **deleted from the template**, not marked skipped. A
step for a removed feature is worse than no step: it either fails forever and gets waved
through, or it quietly "passes" against nothing.
6. Report counts by status, every `fail` with its evidence line, and every finding against a prior
run. A run where some item has no `command` or no `evidence` is not complete — say so instead
of reporting a total.
Items are independent, so lanes can run in parallel against one packed binary. The axis that keeps
them from colliding is the data root: give every lane its own `--data-root`, and nothing else needs
coordinating.
## Making an item able to fail
Most of these steps can be written so that they pass whether or not the feature works. Four
shapes account for nearly all of it:
- **A filter needs a negative control.** Feed clean content alongside the shapes you expect
rejected. A policy that rejects *everything* passes a rejection-only test, and looks healthiest
exactly when it is most broken.
- **A semantic-retrieval query must be one that keyword match cannot carry.** Query for
- "an antique navigation instrument reflecting evening light in a stargazing room" against content
+ "an antique navigation instrument reflecting evening light inside a stargazing room" against content
that says "astrolabe", "lamplight", "observatory" — no literal overlap, so a dead vector leg
actually fails the item. A query sharing words with the stored text passes on BM25 alone and
tells you nothing about the half you meant to test.
- **A keyword query must be judged by its leg.** The vector leg often finds a note by the words
around an identifier, so a keyword item passes on a build whose keyword leg misses. Read
`evidenceByHash` and require an `fts` leg on the hit.
- **An empty list is a weak check.** A queue or candidate list on a fresh bank returns `[]`
whether it works or is broken. Create the thing first, then assert on its content — the score,
the reasons, the identity — not on the shape of the response.
## Scope
Derived per run from the product, not from this list. The headings below are the stable shape of
the pass; the specific steps under each come from what step 2 found and from what is actually
registered in the build in front of you.
- **Build and install** — Release build, pack, force-update the global tool, and `--version`
matches the derived version.
- **Server lifecycle** — the server starts and `--restart` cycles the one it finds, both on a
non-default port.
- **Write path** — a write stores and returns a hash; a rejected write says so, with a reason,
rather than returning a fabricated entry. A note that cites a source file numbers its chunks in
text order, so its opening reports `chunkIndex` 0.
- **Read path** — search returns the written entry, get returns its content by hash, and a
`file#section` anchor resolves its exact chunk. Keyword search finds an identifier that sits
- where a chunk boundary would fall, and a term longer than a whole chunk.
+ where a chunk boundary would fall, and a term longer than a whole chunk. A pasted query of
+ thousands of characters returns in seconds on a bank of real scale, because its keyword terms
+ are capped.
- **Embedding** — a memory row the engine refuses is split from its batch and abandoned after
three attempts. `scripts/poison-embedding-stub.py` is an engine that refuses one marker.
- **Noise filtering** — each *registered* write-path policy rejects what it claims to, and the
rejected content stays retrievable from the noise store. Check which policies are registered
before writing steps for them.
- **Read-path query guard** — the refuse and annotate tiers behave as specified, and any detector
that ships disabled is still disabled until explicitly armed.
- **File watch** — watch status reflects live registrations, and a restart re-digests a file
changed without an event, judged against that file's own fingerprint.
- **Promotion queue** — the promotion list reports candidates accurately.
- **Full MCP surface** — every derived tool and prompt is reachable.
- **Observability** — emitted event ids resolve against the logging event-id reference.
Anchor each item to the decision record that defines the behaviour, so a step whose ADR was
superseded is easy to spot and delete.
## How this checklist rots
Two predecessors were deleted after both drifted the same six ways. Each defence below is here
because its absence already caused a silent failure:
- **Facts pinned by hand.** One asserted `--version → 1.9.1` and "25 tools" while the tree was at
1.12.0 with 26 tools. The pins had been wrong for three releases and nothing noticed, because
the only thing comparing them was a human reading two numbers. → step 2 derives them.
- **Steps for a deleted feature.** Both still tested a noise policy that had been removed by a
later ADR. A step whose subject does not exist cannot pass honestly. → step 4 deletes them.
- **Results written into a bank directory.** Reports landed in `.ai-raccoon/`, the directory name
the product uses for banks. → step 1 fixes the destination.
- **No evidence field.** The template recorded a claim with no room for the command or the output
behind it, so a filled checklist and an invented one were indistinguishable afterwards. →
`command` and `evidence` are required.
- **Booleans that cannot express "skipped".** `checked`/`accepted` flags collapsed three states
into two: an unrun item and a failed one both read as `false`/`false`. → `status` is a tri-state
and `accepted` starts null.
- **Two drifting copies.** Two directories each held a copy, and one had lost its `templates/`
directory entirely, so its own step 1 pointed at a file that was not there. → one copy, and the
template ships beside the skill.
**Retiring a checklist skill means deleting it from every root it can load from**, not from the
one you were looking at. The deletion that removed the two `.ai-badger/skills/learned/` copies
missed `~/.hermes/skills/`, where a third copy was found still installed long afterwards, still
carrying its `templates/`, still pinning an expected version three minors stale. It was the
*first* hit for someone searching for this checklist, and they began executing it; that copy has
since been removed. Enumerate the roots — the project's
`.ai-badger/skills/learned/`, `.claude/skills/`, `~/.claude/skills/`, `~/.hermes/skills/` — and
confirm the removal in each. The stale copy wins whoever searches first, so a copy you did not
delete is not dormant; it is the one in use.
## Gotchas
- **A count that cannot fail is not a count.** Every counting primitive here has a silent zero
in it: `rglob` over a directory that no longer exists yields nothing, `sum(())` is `0`, and
`grep -c` across several files prints one number per file so a bad sum still looks like a
number. None of them raise. `derive-facts.py` closes that by treating zero as a failure rather
than a result — it exits 1 and says which fact it could not derive — and by counting only lines
that are code. Do not "simplify" either rule away: a file that documents `[McpServerTool]` in a
`///` comment already inflated the surface by one for three releases, and the number looked
derived the whole time. Both properties have tests in the framework; change the script there,
never by editing your copy.
- `--port 0` binds an ephemeral port, so the port must be read back from the server's own output.
Assuming a port here is how a checklist step ends up talking to somebody else's server.
- Force-updating a global tool can silently keep the previous build if the pack step failed
earlier in the same run. The version comparison in step 2 is what catches it; do not skip it
because the build "looked fine".
- Read a scratch bank with `sqlite3 -readonly` while its server is running. Once the server
stops, the WAL bank has no `-shm` file and a read-only open fails with "unable to open database
file".
- A scratch `--data-root` must be a path the running user can create. Pointing it inside a
read-only or root-owned directory produces failures that look like product bugs.