adversarial-qa · diff
git:20260928.2953a1d to git:20261004.cccdee6
94 added, 94 removed. Audit A to A.
---
name: adversarial-qa
description: >
- Exploratory, adversarial QA: exercise a feature through whichever surface(s)
- it exposes — UI, API, or both — and surface issues the plan and committed
- tests did not anticipate — not a re-verification of the spec. Invoked as
- /adversarial-qa for an ad-hoc session, or applied by the adversarial-qa
- sub-agent in the /feature workflow.
+ Exploratory QA of a running feature through its UI (Playwright) and/or API
+ (curl) to find issues the plan and committed tests missed; returns findings
+ with evidence, known issues, and blockers. Use when the user asks to QA,
+ probe, or try to break a feature, after code review passes.
+ license: MIT
+ compatibility: >
+ Needs a runnable dev server (AGENTS.md Commands) and, for UI surfaces, the
+ plugin's bundled Playwright MCP server; API probing uses curl.
+ metadata:
+ author: André Cunha
---
- # /adversarial-qa — Exploratory QA
+ # /adversarial-qa
- Exercise a feature in the running app and surface anything that looks wrong,
- confusing, or likely to bite a real user. This is exploratory and adversarial,
- not a re-verification of the spec — committed end-to-end tests encode the plan's
- Requirements deterministically. Your job is to go beyond them.
+ Exercise a feature in the running app and surface anything wrong, confusing,
+ or likely to bite a real user. Go beyond the committed tests; don't
+ re-verify the spec or review code.
- If a plan was provided (inline or by path), read the Requirements section only
- to understand what the feature does — not as a checklist to tick through.
+ ## Input
- If the caller passed a list of open deferred QA findings (even an empty
- one — "none open" is a value, not an omission), use it as-is in step 5
- below. Otherwise (an ad hoc `/adversarial-qa` run) get it yourself: run
- `AGENTS.md` → *Task Tracking* → *List open deferred QA findings* if that
- bullet holds a real command or tool call (not a `[TODO:]` placeholder and
- not `none`), otherwise `gh issue list --label known-issue --state open`.
- If the bullet holds a real command but you cannot execute it (e.g. it names
- an MCP tool call you don't have), do **not** fall back to `gh` — that would
- silently query the wrong tracker. Instead, report it under *Blockers* as
- "known issues not checked: <reason>" and continue the rest of the pass.
+ - Plan text (optional): read Requirements only to understand what the feature
+ does, not as a checklist
+ - Open deferred QA findings list (optional; empty list = "none open", which
+ is a value, not an omission)
- ---
+ ## Steps
- ## What to do
+ ### 1. Get open deferred QA findings
- 1. Determine the surface(s). From the plan's Requirements (or the diff, if no
- plan was given), decide whether the feature exposes a **UI** (templates,
- views, a controller path that renders a view/fragment/client-driven
- response), an **API** (a REST or other network-callable endpoint with no
- view layer), or both. Probe every surface the feature exposes — findings
- from one do not substitute for checking another.
+ - List passed in → use it as-is
+ - Otherwise:
+ - `AGENTS.md` → *Task Tracking* → *List open deferred QA findings* holds a
+ real command/tool call → run it
+ - section/bullet absent, `[TODO:]`, or `none` →
+ `gh issue list --label known-issue --state open`
+ - holds a real command you cannot run (e.g. an MCP tool you don't have) →
+ never fall back to `gh` (wrong tracker). Report under *Blockers* "known
+ issues not checked: <reason>", skip step 5, continue the rest
- 2. Set up and drive the feature, per surface identified in step 1.
+ ### 2. Determine the surface(s)
- - **UI surface** — start the local dev server with the project's dev-server
- command and drive the feature at the documented app URL (both in
- `AGENTS.md` → *Commands*) in a browser via the Playwright MCP. When you
- are done, stop it with the documented stop command — never `kill` by PID
- or hunt processes with `lsof`. If the server will not start or Playwright
- is unavailable, STOP and report the blocker. Do not substitute `curl`,
- SQL, or any other workaround for browser exploration on a UI surface —
- those answer different questions than what a real user experiences.
+ - From the plan's Requirements (or the diff, if no plan): **UI** (templates,
+ views, a controller path rendering a view/fragment/client-driven response),
+ **API** (REST or other network-callable endpoint, no view layer), or both
+ - Probe every surface the feature exposes; findings on one don't substitute
+ for another
- For mechanical setup with a known, fixed sequence — logging in,
- navigating through boilerplate screens to reach the feature under test —
- batch the steps into one `browser_run_code_unsafe` call instead of a
- click/type/snapshot round trip per step; each round trip returns a full
- accessibility snapshot, which adds up fast. Reserve the granular tools
- (`browser_click`, `browser_snapshot`, etc.) for the actual exploration in
- step 3, where you need to see state after each action to decide the next
- one.
+ ### 3. Set up and drive, per surface
- - **API surface** — start the server the same documented way and issue
- requests against the same app URL (`AGENTS.md` → *Commands*) with `curl`
- via `Bash`. Stop the server the same documented way when done. If the
- server will not start, or a request needs credentials you don't have,
- STOP and report the blocker.
+ Start the server with the project's dev-server command; app URL from
+ `AGENTS.md` → *Commands*. Stop it with the documented stop command when done
+ (never `kill` by PID, never hunt processes with `lsof`).
- 3. Probe beyond the happy path. Try things the planner likely did not
- enumerate, per surface:
+ - **UI**: drive the feature in a browser via Playwright MCP
+ - Server won't start or Playwright unavailable → STOP, report the blocker
+ - Never substitute `curl`, SQL, or other workarounds for browser
+ exploration on a UI surface
+ - Mechanical setup with a fixed sequence (login, navigating to the
+ feature) → batch it in one `browser_run_code_unsafe` call, not a
+ click/type/snapshot round trip per step
+ - Use granular tools (`browser_click`, `browser_snapshot`, …) only for the
+ exploration in step 4
+ - To locate an element by text or regex, use `browser_find` instead of
+ capturing a full `browser_snapshot`
+ - **API**: `curl` via `Bash` against the same app URL
+ - Server won't start, or a request needs credentials you don't have →
+ STOP, report the blocker
- - **UI** — narrow viewports, keyboard-only navigation, browser back button,
- multiple tabs on the same form, paste of weird/long/XSS content,
- reloading mid-edit, error-toast timing, interactions with unrelated UI on
- the same page, stale state after a failed submit.
+ ### 4. Probe beyond the happy path
- - **API** — malformed, missing, or extra fields; wrong `Content-Type`;
- auth/authz boundaries (missing token, expired token, wrong role or
- tenant); idempotency and duplicate submission; pagination and limit edge
- cases; concurrent or racing requests; oversized payloads and
- unicode/injection strings in fields; status-code and error-envelope
- correctness; rate limiting.
+ Try what the planner likely didn't enumerate.
- 4. Surface anything that looks off — even if it is not part of this feature's
- plan. Do not act "smart" by working around issues, inferring intent, or
- deciding a bug is "probably expected". Report it and let the developer
- decide.
+ - **UI**: narrow viewports · keyboard-only navigation · browser back ·
+ multiple tabs on the same form · paste of weird/long/XSS content · reload
+ mid-edit · error-toast timing · interaction with unrelated UI on the page
+ · stale state after a failed submit · light/dark colour scheme and reduced
+ motion (`browser_emulate_media`)
+ - **API**: malformed, missing, or extra fields · wrong `Content-Type` ·
+ auth/authz boundaries (missing/expired token, wrong role or tenant) ·
+ idempotency and duplicate submission · pagination and limit edge cases ·
+ concurrent/racing requests · oversized payloads · unicode/injection strings
+ · status-code and error-envelope correctness · rate limiting
- 5. Before writing the report, compare your findings against the known
- deferred issues gathered above (skip this if you reported a *Blockers*
- entry for them instead). A finding that matches an open `known-issue`
- goes in the *Known issues* section of the report (cite its task id), NOT in
- Findings — the developer has already triaged it once and should not have
- to re-triage it on every QA pass. If the observed behaviour is worse than
- or different from what the task describes, that difference IS a finding.
+ Report anything that looks off, even outside this feature's plan. Never work
+ around issues, infer intent, or decide a bug is "probably expected"; the
+ developer decides.
- ---
+ ### 5. Compare with known issues
- ## Evidence
+ (Skip if *Blockers* already reports them as unchecked.)
- Only capture evidence once you've decided something is a finding worth
- reporting — never while just looking around.
+ - Finding matches an open `known-issue` → *Known issues* section, cite its
+ task id; not in *Findings*
+ - Observed behaviour worse than or different from the task's description →
+ that difference IS a finding
- - **UI** — `browser_take_screenshot` returns an image, which costs
- meaningfully more than the text snapshots from `browser_snapshot`, so
- screenshotting every step of the exploration adds up quickly for no
- benefit. Take one only once a finding is confirmed.
- - **API** — capture the request and response that shows the problem: method,
- URL, relevant headers, status code, and body.
+ ## Evidence
- Save each finding's evidence under `.qa-evidence/` at the repo root
- (gitignored); every finding in the report MUST cite at least one evidence
- file there, with a one-sentence description of what it shows.
+ Capture only once something is a confirmed finding, never while exploring.
- ---
+ - **UI**: `browser_take_screenshot` (costs more than `browser_snapshot`; use
+ it only for a confirmed finding)
+ - Pass `filename: .qa-evidence/<name>.png` (relative to the project
+ directory)
+ - A request the page made: `browser_network_request` (headers and bodies
+ of one request listed by `browser_network_requests`), saved with its
+ `filename` parameter under `.qa-evidence/`
+ - **API**: the request and response showing the problem: method, URL,
+ relevant headers, status code, body
+ - Run `mkdir -p .qa-evidence` first (API evidence is written with `Bash`)
+ - Save under `.qa-evidence/` at the repo root (gitignored)
+ - Every finding cites ≥1 evidence file there + a one-sentence description of
+ what it shows
- ## Output Format
+ ## Output
```
### Findings
- [Short description] — [evidence path] — [severity: bug / concern / nit]
### Known issues (already deferred — no action needed)
- [task id] [title] — [still present / not observed on this pass]
### Blockers (if any)
- [Anything that prevented you from exploring — server won't start, Playwright
+ [Anything that prevented exploring — server won't start, Playwright
unavailable, credentials needed, etc.]
```
- An empty `Findings` section is a valid output if you genuinely probed the
- feature and found nothing worth flagging. An empty output because you "ran out
- of ideas" is not.
+ - Empty *Findings* is valid only if you genuinely probed and found nothing
+ worth flagging; "ran out of ideas" is not