autoqa · diff
git:20260721.5cedab0 to git:20260723.afc54a6
57 added, 11 removed. Audit A to A.
---
name: autoqa
description: Self-QA a repo's running application end-to-end — discover how it runs, build a feature×modality test plan, execute it against a live instance, and produce a witnessed pass/fail report. Use when the user says "autoqa", "QA this repo/branch", "verify every feature works", "release readiness check", "test this like a QA engineer would", or another skill needs an automated QA pass before sign-off.
---
# autoqa — QA any repo against its own running app
You are the QA engineer for this repo. Every verdict is **witnessed** — it points at an
artifact (an HTTP response, a page snapshot, a log line, a DB row) that *shows* the result,
not merely a file that exists. The full witness contract is in Hard rules below.
## Phase 0 — RESOLVE
Establish three facts before anything else:
1. **Target repo** — path or URL the user pointed at (ask only if truly absent).
2. **Target instance** — a running deployment to test against (URL/port), or the
instruction to bring one up locally.
3. **Repo config** — look for `AUTOQA.md` at the repo root or under `docs/`. It is the
- repo's pre-answered discovery: how to run, how to auth, where the feature catalog
- lives, known env caveats. **If found, read it now and skip straight to Phase 2.**
- Missing config means full Phase 1 — and a repo you QA repeatedly earns one: write
- `AUTOQA.md` from what Phase 1 taught you and it becomes the config for every later run.
+ repo's reusable **baseline**, not the complete plan: how to run, how to auth, stable
+ catalog/core checks, and known env caveats. **If found, read it now and skip the generic
+ discovery in Phase 1, but never skip Phase 2's diff discovery.** Missing config means
+ full Phase 1 — and a repo you QA repeatedly earns one: write `AUTOQA.md` from what Phase
+ 1 taught you so later runs can start from that baseline.
Done when: repo path, instance URL (or "must boot"), and config-or-none are stated.
## Phase 1 — DISCOVER
Read the repo the way a new engineer would, in this order, stopping when the three questions
below are answered. Full source-priority list and what each source answers:
[references/discovery.md](references/discovery.md).
- **How does it run?** Dockerfile / compose / Procfile / Makefile / CI workflows / README.
- **How do I authenticate?** env samples, auth middleware, dev-token conventions, CLAUDE.md / AGENTS.md.
- **What are the features?** a feature catalog or spec doc if the repo ships one (use it —
it beats inference), else routes/pages/CLI entrypoints enumerated from code.
Done when: you can write down the run command, an auth recipe, and a feature inventory —
each traced to the file that taught you it.
## Phase 2 — PLAN
- First write the **feature inventory** — a numbered list of every feature discovery found.
- Then build the matrix from it: one row per feature, columns = modality, entry point, check,
- pass criterion, witness to capture. The matrix must have ≥1 row per inventory item; the
- report states the coverage arithmetic (`N discovered / M rows / K untested`) so a reader
- sees nothing was silently dropped.
+ Build the plan from the union of two sources:
+ 1. **Baseline inventory** — every feature/check required by `AUTOQA.md`, or Phase 1 when no
+ config exists.
+ 2. **Diff inventory** — cases derived from the actual change under test. Resolve the base
+ from the user's target or PR; otherwise use the merge-base with the repository's default
+ remote branch. Include committed, staged, unstaged, and relevant untracked changes. Read
+ the diff and the acceptance/design docs it changes or cites. Derive behavior-level cases
+ for changed user entry points, APIs/contracts, schemas/migrations, background work,
+ configuration and feature flags, compatibility/fallbacks, failure handling, security or
+ authorization boundaries, concurrency/idempotency, rollout/rollback, and cleanup. Do not
+ mistake a large unit-test list for this inventory.
+
+ First write both inventories as numbered lists, preserving their source (`BASE` or `DIFF`).
+ Then build one matrix from their union: one row per inventory item, columns = source,
+ modality, entry point, check, pass criterion, witness to capture. Deduplicate overlapping
+ rows without dropping the stronger pass criterion. The matrix must have ≥1 row per union
+ item; the report states separate and total coverage arithmetic so a reader sees nothing was
+ silently dropped.
+
+ ### Confirm execution scope with the user
+
+ After drafting the inventories and before executing, use `AskUserQuestion` with multiple
+ choice and `multiSelect: true` to ask what the user wants included. In Codex environments,
+ use the equivalent structured user-input tool when available. Populate the choices from the
+ actual repo and diff, rather than showing a generic checklist. Offer up to four concise
+ groups such as:
+
+ - **Changed behavior + seams (Recommended)** — every DIFF case and its nearest regressions.
+ - **Full baseline catalog** — unaffected baseline features in addition to changed seams.
+ - **Stateful/destructive cases** — migrations, deletes, lifecycle, imports, billing, or
+ external writes; name the exact synthetic/isolated safeguards in the description.
+ - **Performance/soak or platform matrix** — only when the diff makes it relevant.
+
+ The stable health/auth/core-money-path baseline is always included and must be stated in the
+ question. Treat the selection as test scope, not authorization to mutate production or real
+ customer data. If the user already explicitly selected scope (for example, “smoke only” or
+ “go ham/full release QA”), do not ask a redundant question; record that choice in the plan.
+ If no structured question tool is available and scope is not explicit, ask the same concise
+ multi-choice question in plain text and wait.
+
- **Entry point is a gating column, decided here — not at execute time.** For each feature,
name the path a user takes to reach it: the button, link, route, or client API call. A
feature whose entry point you cannot trace is not tested — its disposition is
`SKIPPED (unreachable — candidate dead code)`, and it never executes. Reachability is a
planning property; deciding it now is what stops you from running a router endpoint
nothing navigates to and mistaking its breakage for a bug.
- **Disposition per row**: DEEP (run the end-to-end check), SMOKE (load the surface, assert
its key content), UNTESTED (needs a fixture you don't have), SKIPPED (unreachable, or the
branch predates the feature). A run that covers only headline features is a smoke pass —
the report says so rather than implying full coverage.
- **Modalities**: API (endpoint calls), UI (browser tooling — chrome-devtools MCP,
Playwright, or whatever the session has), CLI (the repo's own binaries). Cover every
modality the app actually has; a web app QA'd only through its API is half-tested.
- **Drive each feature by its traced entry point** — through the UI, or the API call the
client actually issues — not by a raw endpoint you found in the router.
- **Pass criteria are concrete**: status codes, visible text, row counts, terminal job
states — never "looks right".
+ - **Diff cases are additive.** `AUTOQA.md`, a feature tracker, or a prior report can never
+ suppress a test implied by the current diff. A prior PASS is context, not a witness for
+ the current run.
- Order rows: boot/health/auth first (everything else depends on them), then core money
paths, then edge/regression rows.
- Done when: the inventory list exists, the matrix has ≥1 row per inventory item, every row
- carries all five columns, and every executable row names a traced entry point.
+ Done when: baseline and diff inventories exist, the matrix has ≥1 row per union item, every
+ row carries all required columns, every executable row names a traced entry point, and the
+ user's selected execution scope is recorded.
## Phase 3 — EXECUTE
Execute only rows whose entry point was traced in PLAN. Run the matrix top to bottom
against the live instance.
- Instance not up? Bring it up exactly as discovery taught — respect the repo's own
runbook (secret-injection wrappers, port maps) over generic docker commands. If it
cannot be brought up at all (missing secrets, port conflict, boot crash), the whole run
is `BLOCKED` — report what failed to boot and stop; never force a ship/don't-ship verdict
on an app you never ran.
- Write the report and evidence side by side: report at `<scratch>/autoqa-report.md`,
evidence in `<scratch>/autoqa-evidence/`, witness paths relative to that shared parent so
they resolve. Name witness files by row: curl output with status codes, page snapshots or
screenshots, log excerpts.
- A failing row gets one diagnosis pass: is it the app, the env, or your check? Fix
env/check mistakes and rerun; app failures stay failed and get a one-line cause. (Rows
whose entry point couldn't be traced were already SKIPPED in PLAN — you never reach here
for dead code.)
- Async work (jobs, builds) is polled to a terminal state, capped at a stated timeout; on
expiry the row is UNTESTED with the elapsed time — never a pass, never an infinite poll.
- Leave the instance as healthy as you found it; if you restarted anything, re-verify
health before reporting.
Done when: every matrix row is PASS, FAIL (with cause), UNTESTED (with reason), or SKIPPED —
each with a witness that shows the asserted result — or the run is BLOCKED with the boot
failure recorded.
## Phase 4 — REPORT
Write the report from the template in
[references/report-template.md](references/report-template.md): verdict table (feature,
modality, result, witness path), failure triage (release blocker vs env quirk vs test
bug), and the one-paragraph bottom line a release owner can act on.
Done when: the report file exists next to the evidence dir, every table row's witness path
resolves, and the bottom line states ship / don't-ship / ship-with-caveats / blocked.
## Hard rules
- **No witness, no verdict.** A witness must *show* the asserted result — the 200 in the
captured status line, the expected text in the snapshot — not merely exist. A named file
that doesn't show the result is not a witness. Reruns beat inference.
- **Reachability is decided in PLAN, not blamed in EXECUTE** — a feature whose user entry
point you can't trace is SKIPPED before it runs; a break behind a path nothing navigates
to is dead code to remove, not a bug to fix.
- **Coverage is counted, not claimed** — the report shows `discovered / rows / untested` so
a dropped feature is visible arithmetic, not a silent gap.
+ - **Baseline plus diff, always** — treat `AUTOQA.md` as the reusable floor. Inspect the
+ current diff every run and add the cases it implies; never execute a stale static catalog
+ as though it covered new behavior.
+ - **Scope is an explicit user choice** — use a structured multi-select question after
+ planning unless the user already supplied an unambiguous scope. Record excluded groups
+ as out of scope; do not silently omit them.
- **The repo's runbook outranks your habits** — a project whose docs wrap startup in a
secret-injection command never gets a bare `docker compose up`.
- **Report failures as found** — a QA pass that only reports greens is a failed QA pass;
triage severity honestly instead of softening results.