autoqa · diff
git:20260915.c1df904 to git:20260915.b0f7211
53 added, 35 removed. Audit A to A.
---
name: autoqa
description: Self-QA a repo's running application end-to-end — discover how it runs, build a feature×modality test plan, execute it against a live instance, and produce a witnessed pass/fail report. Use when the user says "autoqa", "QA this repo/branch", "verify every feature works", "release readiness check", "test this like a QA engineer would", or another skill needs an automated QA pass before sign-off.
---
# autoqa — QA any repo against its own running app
You are the QA engineer for this repo. Every verdict is **witnessed** — it points at an
artifact (an HTTP response, a page snapshot, a log line, a DB row) that *shows* the result,
not merely a file that exists. The full witness contract is in Hard rules below.
## Phase 0 — RESOLVE
Establish three facts before anything else:
1. **Target repo** — path or URL the user pointed at (ask only if truly absent).
2. **Target instance** — a running deployment to test against (URL/port), or the
instruction to bring one up locally.
3. **Repo config** — look for `AUTOQA.md` at the repo root or under `docs/`. It is the
- repo's reusable **baseline**, not the complete plan: how to run, how to auth, stable
+ repo's reusable **baseline**, not the complete plan: how to run, how to auth, optional
catalog/core checks, and known env caveats. **If found, read it now and skip the generic
discovery in Phase 1, but never skip Phase 2's diff discovery.** Missing config means
full Phase 1 — and a repo you QA repeatedly earns one: write `AUTOQA.md` from what Phase
1 taught you so later runs can start from that baseline.
Done when: repo path, instance URL (or "must boot"), and config-or-none are stated.
## Phase 1 — DISCOVER
Read the repo the way a new engineer would, in this order, stopping when the three questions
below are answered. Full source-priority list and what each source answers:
[references/discovery.md](references/discovery.md).
- **How does it run?** Dockerfile / compose / Procfile / Makefile / CI workflows / README.
- **How do I authenticate?** env samples, auth middleware, dev-token conventions, CLAUDE.md / AGENTS.md.
- **What are the features?** a feature catalog or spec doc if the repo ships one (use it —
it beats inference), else routes/pages/CLI entrypoints enumerated from code.
Done when: you can write down the run command, an auth recipe, and a feature inventory —
each traced to the file that taught you it.
## Phase 2 — PLAN
- Build the plan from the union of two sources:
+ Build the plan from these sources:
- 1. **Baseline inventory** — every feature/check required by `AUTOQA.md`, or Phase 1 when no
- config exists.
- 2. **Diff inventory** — cases derived from the actual change under test. Resolve the base
+ 1. **Required smoke.** Health for every run, plus auth when authenticated code changed.
+ 2. **Diff inventory.** Cases derived from the actual change under test. Resolve the base
from the user's target or PR; otherwise use the merge-base with the repository's default
remote branch. Include committed, staged, unstaged, and relevant untracked changes. Read
the diff and the acceptance/design docs it changes or cites. Derive behavior-level cases
for changed user entry points, APIs/contracts, schemas/migrations, background work,
configuration and feature flags, compatibility/fallbacks, failure handling, security or
authorization boundaries, concurrency/idempotency, rollout/rollback, and cleanup. Do not
- mistake a large unit-test list for this inventory.
+ mistake a large unit-test list for this inventory. Include changed entry points and
+ directly affected downstream paths.
+ 3. **Optional baseline inventory.** Unaffected catalog checks from `AUTOQA.md` or Phase 1
+ only when the caller asks for full release or catalog QA.
- First write both inventories as numbered lists, preserving their source (`BASE` or `DIFF`).
- Then build one matrix from their union: one row per inventory item, columns = source,
- modality, entry point, check, pass criterion, witness to capture. Deduplicate overlapping
- rows without dropping the stronger pass criterion. The matrix must have ≥1 row per union
- item; the report states separate and total coverage arithmetic so a reader sees nothing was
- silently dropped.
+ Write the selected inventories as numbered lists, preserving their source (`BASE` or `DIFF`).
+ Then build one matrix: one row per inventory item, columns = source, modality, entry point,
+ check, pass criterion, witness to capture. Deduplicate overlapping rows without dropping the
+ stronger pass criterion. The matrix must have at least one row per item. The report states
+ separate and total coverage arithmetic.
+ Apply domain checks only when the diff affects that domain. Run UI checks when the diff
+ affects a UI entry point. Run research end-to-end checks when research execution changes.
+ Run storage or billing checks when storage or billing changes. Backend-only changes require
+ API checks. Run CLI checks only when the diff affects a CLI entry point or the CLI is needed
+ to prove the changed behavior.
+
+ State execution budgets in the plan: maximum wall time, external jobs, synthetic resources,
+ and UI sessions. Move-only work defaults to one preview and one pass over affected paths.
+ Do not launch deep research unless the diff changes research execution.
+
### Confirm execution scope with the user
Scope comes from the first of these that exists: an explicit instruction from the caller
("smoke only", "full release QA"), a scope the repo's `AUTOQA.md` pre-selects for
unattended runs, or a question to the user. When either of the first two applies, record it
in the plan and do not ask; a repo that runs this skill from a pipeline pre-selects its scope
in `AUTOQA.md` precisely so the run never blocks on a prompt.
Otherwise, after drafting the inventories and before executing, use `AskUserQuestion` with multiple
choice and `multiSelect: true` to ask what the user wants included. In Codex environments,
use the equivalent structured user-input tool when available. Populate the choices from the
actual repo and diff, rather than showing a generic checklist. Offer up to four concise
groups such as:
- **Changed behavior + seams (Recommended)** — every DIFF case and its nearest regressions.
- **Full baseline catalog** — unaffected baseline features in addition to changed seams.
- **Stateful/destructive cases** — migrations, deletes, lifecycle, imports, billing, or
external writes; name the exact synthetic/isolated safeguards in the description.
- **Performance/soak or platform matrix** — only when the diff makes it relevant.
- The stable health/auth/core-money-path baseline is always included and must be stated in the
- question. Treat the selection as test scope, not authorization to mutate production or real
- customer data. If the user already explicitly selected scope (for example, “smoke only” or
- “go ham/full release QA”), do not ask a redundant question; record that choice in the plan.
+ Health is always included. Include auth only when authenticated code changed. Treat the
+ selection as test scope, not authorization to mutate production or real customer data. If
+ the user already explicitly selected scope, do not ask a redundant question. Record that
+ choice in the plan.
If no structured question tool is available and scope is not explicit, ask the same concise
multi-choice question in plain text and wait.
- **Entry point is a gating column, decided here — not at execute time.** For each feature,
name the path a user takes to reach it: the button, link, route, or client API call. A
feature whose entry point you cannot trace is not tested — its disposition is
`SKIPPED (unreachable — candidate dead code)`, and it never executes. Reachability is a
planning property; deciding it now is what stops you from running a router endpoint
nothing navigates to and mistaking its breakage for a bug.
- **Disposition per row**: DEEP (run the end-to-end check), SMOKE (load the surface, assert
its key content), UNTESTED (needs a fixture you don't have), SKIPPED (unreachable, or the
branch predates the feature). A run that covers only headline features is a smoke pass —
the report says so rather than implying full coverage.
- - **Modalities**: API (endpoint calls), UI (browser tooling — chrome-devtools MCP,
- Playwright, or whatever the session has), CLI (the repo's own binaries). Cover every
- modality the app actually has; a web app QA'd only through its API is half-tested.
+ - **Modalities**: API (endpoint calls), UI (browser tooling), CLI (the repo's own binaries).
+ Match them to the diff. Do not add UI or CLI checks to a backend-only change unless they
+ are needed to prove the changed behavior.
- **Drive each feature by its traced entry point** — through the UI, or the API call the
client actually issues — not by a raw endpoint you found in the router.
- **Pass criteria are concrete**: status codes, visible text, row counts, terminal job
states — never "looks right".
- **Diff cases are additive.** `AUTOQA.md`, a feature tracker, or a prior report can never
suppress a test implied by the current diff. A prior PASS is context, not a witness for
the current run.
- - Order rows: boot/health/auth first (everything else depends on them), then core money
- paths, then edge/regression rows.
+ - Order rows: boot and health first, auth when required, then affected paths and regressions.
- Done when: baseline and diff inventories exist, the matrix has ≥1 row per union item, every
+ Done when: the selected inventories exist, the matrix has at least one row per item, every
row carries all required columns, every executable row names a traced entry point, and the
- user's selected execution scope is recorded.
+ user's selected execution scope and budgets are recorded.
## Phase 3 — EXECUTE
Execute only rows whose entry point was traced in PLAN. Run the matrix top to bottom
against the live instance.
- Instance not up? Bring it up exactly as discovery taught — respect the repo's own
runbook (secret-injection wrappers, port maps) over generic docker commands. If it
cannot be brought up at all (missing secrets, port conflict, boot crash), the whole run
is `BLOCKED` — report what failed to boot and stop; never force a ship/don't-ship verdict
on an app you never ran.
- Write the report and evidence side by side: report at `<scratch>/autoqa-report.md`,
evidence in `<scratch>/autoqa-evidence/`, witness paths relative to that shared parent so
they resolve. Name witness files by row: curl output with status codes, page snapshots or
screenshots, log excerpts.
- A failing row gets one diagnosis pass: is it the app, the env, or your check? Fix
env/check mistakes and rerun; app failures stay failed and get a one-line cause. (Rows
whose entry point couldn't be traced were already SKIPPED in PLAN — you never reach here
for dead code.)
- Async work (jobs, builds) is polled to a terminal state, capped at a stated timeout; on
expiry the row is UNTESTED with the elapsed time — never a pass, never an infinite poll.
- Leave the instance as healthy as you found it; if you restarted anything, re-verify
health before reporting.
+ - Record each created resource's exact ID and cleanup result in the report. Run a separate
+ cleanup audit only when cleanup fails, the diff changes cleanup behavior, or a created
+ resource remains.
+ - Report shared capacity or provider failures as `BLOCKED_INFRA` for the affected rows.
+ Keep completed scoped results intact. State review readiness separately from any merge
+ policy that treats the infrastructure failure as blocking.
- ### Before / after captures
+ ### Before / after evidence
- Every DIFF row with a UI modality already produces a screenshot of the branch as its witness.
- That screenshot is the "after". Pair it with a "before" of the same view when a running base
- instance exists, so the PR body can show the change without a second tool or a second browser
- session:
+ Compare changed behavior with the clearest observable evidence. UI changes use screenshots.
+ Non-UI and behavior-preserving changes use contract output, status and body shape, OpenAPI,
+ logs, or an equivalent observable result. Do not require screenshots when behavior must not
+ change.
+ For UI changes, pair the branch screenshot with the same view from a running base instance
+ when one exists:
+
- The base instance is one the caller names or the repo config lists (a main preview,
staging, or production), reached read-only. Never manufacture a "before" by switching
branches, stashing, or starting a second server; with no base instance, record
`before: none (no base instance)` on the row and move on.
- Capture the "before" with the same browser tooling the UI rows use, at the same route,
viewport, and element, and save it beside the "after" as `<row>-before.png` and
`<row>-after.png` under the evidence dir.
- A pair presents the change; it does not judge it. The row's PASS or FAIL still comes from
its own check, never from the two images looking different.
- Publishing the images somewhere a PR body can render them (an assets branch, a comment
attachment, the repo's own upload path) is the caller's job with the repo's tooling; the
report lists the local pair and the two instance URLs plus the "after" commit SHA.
- Done when: every matrix row is PASS, FAIL (with cause), UNTESTED (with reason), or SKIPPED —
- each with a witness that shows the asserted result — or the run is BLOCKED with the boot
- failure recorded, and every UI DIFF row names its before/after pair or the reason it has none.
+ Done when: every matrix row is PASS, FAIL (with cause), UNTESTED (with reason), SKIPPED, or
+ BLOCKED_INFRA. Each row has a witness that shows the asserted result. Every changed row names
+ its before/after evidence or why a comparison does not apply.
## Phase 4 — REPORT
Write the report from the template in
[references/report-template.md](references/report-template.md): verdict table (feature,
- modality, result, witness path), the Before / After table for UI DIFF rows, failure triage
+ modality, result, witness path), the Before / After table for changed rows, failure triage
(release blocker vs env quirk vs test bug), and the one-paragraph bottom line a release owner
can act on. The Before / After table is written so a caller can lift it into a PR body once
the images are published.
Done when: the report file exists next to the evidence dir, every table row's witness path
resolves, and the bottom line states ship / don't-ship / ship-with-caveats / blocked.
## Hard rules
- **No witness, no verdict.** A witness must *show* the asserted result — the 200 in the
captured status line, the expected text in the snapshot — not merely exist. A named file
that doesn't show the result is not a witness. Reruns beat inference.
- **Reachability is decided in PLAN, not blamed in EXECUTE** — a feature whose user entry
point you can't trace is SKIPPED before it runs; a break behind a path nothing navigates
to is dead code to remove, not a bug to fix.
- **Coverage is counted, not claimed** — the report shows `discovered / rows / untested` so
a dropped feature is visible arithmetic, not a silent gap.
- - **Baseline plus diff, always** — treat `AUTOQA.md` as the reusable floor. Inspect the
- current diff every run and add the cases it implies; never execute a stale static catalog
- as though it covered new behavior.
+ - **Scope follows the diff by default.** Always check health. Add auth and domain checks
+ only when the diff affects them. Run the full baseline catalog only when the caller asks.
- **Scope is an explicit choice, never a guess** — from the caller, from the repo's
`AUTOQA.md`, or from a structured multi-select question after planning, in that order.
Record excluded groups as out of scope; do not silently omit them.
- **The repo's runbook outranks your habits** — a project whose docs wrap startup in a
secret-injection command never gets a bare `docker compose up`.
- **Report failures as found** — a QA pass that only reports greens is a failed QA pass;
triage severity honestly instead of softening results.