ccc-qa ยท diff

git:20260707.947ab2f to git:20261013.fbe187d

107 added, 183 removed. Audit A to A.

---
name: ccc-qa
- description: "QA workflow. Runs unit + integration + E2E tests, coverage delta, edge case enumeration, flaky test quarantine. Delegates to qa-engineer agent."
+ description: "QA pass before shipping: detects the framework, runs unit, integration and E2E suites, computes the coverage delta, audits flaky tests and has qa-engineer severity-rate gaps. Use when the user asks for a QA pass, to run all tests, check coverage, find flaky tests or if it is ready to ship."
allowed-tools:
- Bash
- Read
+ - Write
- Agent
+ - Skill
- AskUserQuestion
- argument-hint: "[unit | integration | e2e | all | coverage]"
+ argument-hint: "[unit | integration | e2e | all | coverage | flaky]"
+ model: sonnet
+ effort: medium
---
- # /ccc-qa โ€” Comprehensive QA Workflow
-
- Full QA pass before shipping. Detects test framework, runs all suites, reports coverage delta, flags edge cases and flaky tests. Delegates analysis to the `qa-engineer` subagent. Never marks QA passed with open ๐Ÿ”ด Critical findings.
-
- ## Triggers
+ # /ccc-qa โ€” QA pass before you ship
- - `/ccc-qa` or `/qa`
- - "QA pass", "run all tests", "test coverage check"
- - "before shipping", "pre-release checks", "is this ready to ship"
- - "check test coverage", "find missing tests"
- - "flaky test", "test suite health"
- - Coming from `/ccc-review` or `/ccc-ship`
+ **CC Commander** ยท /ccc-qa ยท every suite run, coverage delta, flaky tests and missing edge cases, severity-rated.
- ## When NOT to Use
+ ## When to use / when not to
- - Greenfield project with no tests yet โ€” use `/ccc-testing` to scaffold the test suite first
- - Single-function bug fixes where a targeted unit test is the right move (write it directly)
- - Pure documentation or config-only changes with no testable behavior
+ - Use for "QA pass", "run all tests", "check coverage", "find missing tests", "flaky test", "is this ready to ship", or when `/ccc-review` or `/ccc-ship` hands off here.
+ - Use `/ccc-testing` first when the project has no tests yet; it scaffolds a suite.
+ - Skip it for a single-function fix (write the targeted unit test directly) and for docs- or config-only changes with no testable behavior.
+ - Use `/ccc-e2e` for a fanned-out pre-release assessment across isolated worktrees.
- ## Process
+ ## What happens
- ### Step 1 โ€” Detect test framework
+ ### 1. Detect the framework
- Run the following in parallel via `Bash`:
+ Run in parallel with Bash; a failing probe means "not this framework":
```bash
- # Detect JS/TS frameworks
- cat package.json 2>/dev/null | grep -E '"(vitest|jest|playwright|mocha|cypress|jasmine)"' | head -10
-
- # Detect config files
- ls *.config.{ts,js,mjs} vitest.config.* jest.config.* playwright.config.* pytest.ini setup.cfg \
- go.mod Cargo.toml pom.xml 2>/dev/null
-
- # Detect Python
- python3 -m pytest --collect-only -q 2>/dev/null | tail -5
+ grep -E '"(vitest|jest|@playwright/test|mocha|cypress|jasmine)"' package.json 2>/dev/null | head -10
+ ls vitest.config.* jest.config.* playwright.config.* pytest.ini setup.cfg pyproject.toml go.mod Cargo.toml pom.xml 2>/dev/null
+ python3 -m pytest --collect-only -q 2>/dev/null | tail -3
+ cat .ccc/coverage-history.json 2>/dev/null
+ ```
- # Detect Go
- go test ./... -list '.*' 2>/dev/null | head -10
+ | Evidence | Framework | Run command |
+ |---|---|---|
+ | `vitest.config.*` or `vitest` dependency | Vitest | `npx vitest run --coverage --coverage.reporter=json-summary --coverage.reporter=text` |
+ | `jest.config.*` or `jest` dependency | Jest | `npx jest --coverage --coverageReporters=json-summary --coverageReporters=text` |
+ | `playwright.config.*` | Playwright | `npx playwright test --reporter=list` |
+ | `pytest.ini`, `[tool:pytest]` in `setup.cfg`, or `[tool.pytest.ini_options]` in `pyproject.toml` | pytest | `python3 -m pytest --cov=. --cov-report=term-missing -q` |
+ | `go.mod` | Go | `go test ./... -cover -count=1` |
+ | `Cargo.toml` | Rust | `cargo test` |
+ | `pom.xml` | JUnit via Maven | `mvn test` |
- # Detect Rust
- cargo test --no-run 2>/dev/null | tail -5
+ When `package.json` defines a `test` script, prefer `npm test` for the unit lane and keep the coverage flags above only if the script does not already set them. If nothing is detected, say so and offer `/ccc-testing` to scaffold a suite; stop.
- # Baseline coverage (if previously run)
- cat .ccc/coverage-history.json 2>/dev/null || cat coverage/coverage-summary.json 2>/dev/null | head -30
- ```
+ ### 2. Confirm scope
- **Detection logic:**
- - `vitest.config.*` present โ†’ **Vitest** (`npx vitest run --coverage`)
- - `jest.config.*` present or `"jest"` in scripts โ†’ **Jest** (`npx jest --coverage`)
- - `playwright.config.*` present โ†’ **Playwright** (`npx playwright test`)
- - `pytest.ini` or `setup.cfg [tool:pytest]` โ†’ **pytest** (`python3 -m pytest --cov`)
- - `go.mod` present โ†’ **Go test** (`go test ./... -cover`)
- - `Cargo.toml` present โ†’ **cargo test** (`cargo test 2>&1`)
- - `pom.xml` present โ†’ **JUnit via Maven** (`mvn test`)
- - None detected โ†’ report to user and offer to scaffold via `/ccc-testing`
+ An argument (`unit`, `integration`, `e2e`, `all`, `coverage`, `flaky`) skips this picker. Otherwise ask with AskUserQuestion (Codex: a plain-text question):
- ### Step 2 โ€” Confirm scope
+ - question: "Which test suites should I run?" ยท header: "QA scope"
+ - ๐ŸŸข All suites โ€” unit, integration and E2E; slowest and most thorough, the pre-release pass
+ - Unit and integration โ€” skips E2E; a fast mid-session check
+ - E2E only โ€” Playwright or Cypress; needs the app reachable
+ - Flaky audit โ€” runs the suite three times and flags tests whose result changes
- ```yaml
- question: "Which test suites should we run?"
- options:
- - label: "๐Ÿงช All suites (unit + integration + E2E)"
- description: "Full QA pass. Slowest, most thorough. Use before releases."
- - label: "โšก Unit + integration only"
- description: "Skip E2E. Fast โ€” good for mid-session checks."
- - label: "๐ŸŽญ E2E only (Playwright / Cypress)"
- description: "Browser tests only. Requires running dev server."
- - label: "๐Ÿ“Š Coverage report only"
- description: "No new test run โ€” parse last run's coverage output."
- - label: "๐Ÿ” Flaky test audit (run suite 3ร—)"
- description: "Detect non-deterministic tests by running the full suite three times and comparing results."
- ```
+ `coverage` parses the last run's coverage output without running tests (go to step 4).
- ### Step 3 โ€” Run tests
+ ### 3. Run the suites
- Execute based on detected framework and confirmed scope. Capture full stdout. Time the run.
+ Run each selected command from step 1 with Bash, capture the full output to a file and time it:
- **Vitest (unit + coverage):**
```bash
- npx vitest run --coverage --reporter=verbose 2>&1 | tail -80
+ mkdir -p .ccc
+ start=$(date +%s); <run command> > .ccc/qa-<lane>.log 2>&1; echo "exit=$? seconds=$(( $(date +%s) - start ))"
+ tail -80 .ccc/qa-<lane>.log
```
- **Jest:**
- ```bash
- npx jest --coverage --forceExit 2>&1 | tail -80
- ```
+ A non-zero exit with failing tests is a finding, not a skill failure. A non-zero exit with no test output (missing dependency, config error) is reported verbatim and stops that lane.
- **Playwright (E2E):**
- ```bash
- # Confirm dev server is running first
- curl -s http://localhost:3000 > /dev/null && echo "server up" || echo "server down"
- npx playwright test --reporter=list 2>&1 | tail -60
- ```
+ **E2E precondition.** If `playwright.config.*` has a `webServer` block, Playwright starts the app itself. Otherwise read `baseURL` from the config (default `http://localhost:3000`) and check it:
- **pytest:**
```bash
- python3 -m pytest --cov=. --cov-report=term-missing -q 2>&1 | tail -60
+ curl -s -o /dev/null -w '%{http_code}\n' "<baseURL>"
```
- **Go:**
- ```bash
- go test ./... -cover -count=1 2>&1 | tail -40
- ```
+ If it is unreachable, ask the user to start the dev server; never start one yourself.
- **Flaky test detection (3-run mode):**
+ **Flaky audit (three runs, Vitest or Jest):**
+
```bash
+ mkdir -p .ccc
for i in 1 2 3; do
- echo "=== Run $i ===" && npx vitest run --reporter=json 2>/dev/null | \
- jq '[.testResults[].assertionResults[] | {name:.fullName, status:.status}]'
+ npx vitest run --reporter=json --outputFile=".ccc/flaky-run-$i.raw.json" >/dev/null 2>&1
+ # Jest: npx jest --json --outputFile=".ccc/flaky-run-$i.raw.json" >/dev/null 2>&1
+ jq '[.testResults[].assertionResults[] | {name: .fullName, status}]' ".ccc/flaky-run-$i.raw.json" > ".ccc/flaky-run-$i.json"
done
+ jq -s 'add | group_by(.name) | map(select((map(.status) | unique | length) > 1)) | map(.[0].name)' .ccc/flaky-run-1.json .ccc/flaky-run-2.json .ccc/flaky-run-3.json
```
- Compare outputs across 3 runs. Any test with differing `status` across runs is flagged as flaky.
- ### Step 4 โ€” Compute coverage delta
+ Every name printed is flaky. For other frameworks, run the step 1 command three times with no watch mode (`go test -count=1`, `pytest -rA`) and compare the pass/fail lines. Never use `--watch`.
+ ### 4. Compute the coverage delta
+
+ For Vitest and Jest (Istanbul `coverage/coverage-summary.json`):
+
```bash
- # Store current coverage
mkdir -p .ccc
- CURRENT=$(cat coverage/coverage-summary.json 2>/dev/null | \
- jq '{lines: .total.lines.pct, branches: .total.branches.pct, functions: .total.functions.pct}')
- PREV=$(cat .ccc/coverage-history.json 2>/dev/null | jq '.last // {}')
- echo "$CURRENT" > .ccc/coverage-history.json
- echo "Previous: $PREV"
- echo "Current: $CURRENT"
+ CURRENT=$(jq -c '{lines: .total.lines.pct, branches: .total.branches.pct, functions: .total.functions.pct}' coverage/coverage-summary.json 2>/dev/null)
+ PREV=$(jq -c '.last // {}' .ccc/coverage-history.json 2>/dev/null)
+ [ -z "$PREV" ] && PREV='{}'
+ if [ -n "$CURRENT" ]; then
+ jq -n --argjson last "$CURRENT" --argjson previous "$PREV" '{last: $last, previous: $previous}' > .ccc/coverage-history.json
+ fi
+ echo "previous: $PREV"; echo "current: ${CURRENT:-none}"
```
- Report delta as: `lines: 74% โ†’ 82% (+8%)`, `branches: 61% โ†’ 65% (+4%)`.
- Flag any regression (delta < 0) as ๐ŸŸ  High.
+ For pytest, Go and Rust, read the total from the run log (`TOTAL` line, `coverage: N%`) and store it the same way. Report each metric as `lines: 74% โ†’ 82% (+8)`. Any negative delta is ๐ŸŸ  High. With no previous baseline, report the current numbers and say this run is the baseline.
- ### Step 5 โ€” Delegate to qa-engineer agent
+ ### 5. Delegate the analysis
- Pass raw test output + coverage delta to the `qa-engineer` subagent:
+ Dispatch the `qa-engineer` agent with the Agent tool (`effort: "medium"`). If the Agent tool does not list it, use `general-purpose` and make its first instruction "Read `${CLAUDE_PLUGIN_ROOT}/agents/qa-engineer.md` and follow it." Pass the tails of the `.ccc/qa-*.log` files, the flaky list and the delta, with this task:
```
- You are the qa-engineer persona. Analyze the test output and coverage delta below.
-
- Return a severity-rated QA report with these sections:
- 1. Pass/fail counts per suite
- 2. Coverage delta (lines, branches, functions)
- 3. Flaky tests detected (name + inconsistency description)
- 4. Top 5 missing edge cases inferred from failing or absent tests โ€” use this template:
- - empty input (null / undefined / "")
- - max boundary (max int, max string length, rate limit)
- - concurrent access (two writes at same time)
- - unicode / special chars in user-controlled fields
- - expired token / stale session at exact boundary second
- 5. Severity-rated summary:
- ๐Ÿ”ด Critical โ€” test failure blocks ship
- ๐ŸŸ  High โ€” coverage regression or flaky test in critical path
+ Analyze the test output and coverage delta below. Return a QA report with:
+ 1. Pass/fail counts per suite.
+ 2. Coverage delta (lines, branches, functions).
+ 3. Flaky tests: name and how the result changed across runs.
+ 4. The five most important missing edge cases, inferred from failing or absent tests,
+ checked against: empty input, max boundary, concurrent access, unicode or special
+ characters in user-controlled fields, expired state at the exact boundary, oversize payloads.
+ 5. Every finding severity-rated:
+ ๐Ÿ”ด Critical โ€” a test failure that blocks ship
+ ๐ŸŸ  High โ€” coverage regression, or a flaky test on a critical path
๐ŸŸก Medium โ€” missing edge case with moderate risk
- ๐ŸŸข Low โ€” informational, style, or nit
-
- Keep total output under 300 words. Do not reproduce the full test log.
-
- --- TEST OUTPUT ---
- {raw_output}
-
- --- COVERAGE DELTA ---
- {delta}
- ```
-
- ### Step 6 โ€” Present results and next action
-
- Render the agent's report. Then:
-
- ```yaml
- question: "QA complete โ€” what's next?"
- options:
- - label: "๐Ÿ”ง Fix failing tests now"
- description: "Work through ๐Ÿ”ด and ๐ŸŸ  findings before shipping."
- - label: "๐Ÿ“ Write missing edge case tests"
- description: "Scaffold the missing test cases surfaced by the qa-engineer."
- - label: "๐Ÿš€ Ship anyway (log findings)"
- description: "Only valid if there are no ๐Ÿ”ด Critical findings. Logs report to .ccc/qa-report.md."
- - label: "๐Ÿ“Œ File issues to Linear"
- description: "Create Linear issues for each ๐ŸŸ + finding."
- ```
-
- ## Edge Case Enumeration Template
-
- When the qa-engineer returns missing edge cases, use this checklist as the base:
-
- | Category | Example inputs to test |
- |---------------|------------------------|
- | Empty | `""`, `null`, `undefined`, `[]`, `{}` |
- | Boundary | `Number.MAX_SAFE_INTEGER`, string of 10,000 chars, 0, -1 |
- | Concurrent | Two simultaneous writes to the same record |
- | Unicode | `"ั‚ะตัั‚@example.com"`, emoji in username, RTL string |
- | Expired state | Token at exact TTL boundary, stale session cookie |
- | Overflow | Payload > server max body size, file > upload limit |
-
- ## Examples
-
- ### Example 1 โ€” Auth refactor QA
-
- ```
- /ccc-qa unit
+ ๐ŸŸข Low โ€” informational or style
+ Keep it under 300 words. Quote file:line for each finding. Do not reproduce the full log.
```
- Framework detected: Vitest. Output: 187/187 pass, coverage 74% โ†’ 82% (+8% on `src/auth/*`).
- qa-engineer flags: missing concurrent session creation test (๐ŸŸ  High), expired-token-at-boundary test (๐ŸŸก Medium).
- Decision chip: user picks "Write missing edge case tests."
+ The edge-case categories and worked examples are in [references/edge-cases.md](references/edge-cases.md).
- ### Example 2 โ€” Payment flow QA
+ ### 6. Present results and the next action
- ```
- /ccc-qa all
- ```
+ Render the agent's report, then ask with AskUserQuestion. Put ๐ŸŸข on "Fix failing tests" when any ๐Ÿ”ด or ๐ŸŸ  finding exists, otherwise on "Ship with findings logged":
- Framework detected: Vitest + Playwright. Unit: 204/210 (6 failing). E2E: 12/12. Coverage delta: โˆ’3% branches (๐ŸŸ  regression).
- qa-engineer flags: `POST /checkout` with empty cart returns 500 instead of 400 (๐Ÿ”ด Critical). Blocks ship.
- Decision chip: user picks "Fix failing tests now."
+ - question: "QA complete โ€” what next?" ยท header: "Next"
+ - ๐Ÿ”ง Fix failing tests โ€” work through ๐Ÿ”ด and ๐ŸŸ  findings before shipping
+ - ๐Ÿ“ Write missing edge-case tests โ€” scaffold the cases the qa-engineer surfaced
+ - ๐Ÿš€ Ship with findings logged โ€” writes `.ccc/qa-report.md` and hands off to `/ccc-ship`; offered only when there are zero ๐Ÿ”ด findings
+ - ๐Ÿ“Œ File issues to Linear โ€” opt-in Linear connector via `/ccc-linear`, one issue per ๐ŸŸ -or-worse finding; asks before creating anything
- ### Example 3 โ€” Migration QA (flaky audit)
+ ## Done when
- ```
- /ccc-qa
- # User selects: Flaky test audit (run suite 3ร—)
- ```
+ - Every pass/fail count in the reply is quoted from a run log written in this turn (`ls -l .ccc/qa-*.log`).
+ - When coverage ran, `jq . .ccc/coverage-history.json` shows the new `last` values.
+ - The verdict is "QA passed" only with zero ๐Ÿ”ด findings; otherwise it is "QA blocked" with the blocking tests named.
+ - For "Ship with findings logged", `.ccc/qa-report.md` exists and contains the report.
- Run 1: 145 pass. Run 2: 143 pass (2 fail). Run 3: 145 pass.
- qa-engineer flags `UserSessionManager.test.ts:88` as flaky (๐ŸŸ  High โ€” race condition in teardown).
- Decision chip: user picks "File issues to Linear."
+ ## Guardrails
- ## Anti-patterns
+ - Never mark QA passed with an open ๐Ÿ”ด Critical finding, and never offer the ship option while one exists.
+ - Never skip the qa-engineer analysis; raw test output alone is not a QA report.
+ - Never start a dev server or bind a port; ask the user to start it.
+ - Never run suites in watch mode; always single-run (`run`, `-count=1`).
+ - Filing tracker issues needs the user's explicit choice in step 6 and the Linear connector; it is absent by default.
- - Do not mark QA "passed" if there are ๐Ÿ”ด Critical findings
- - Do not skip the qa-engineer agent โ€” raw test output alone is not a QA report
- - Do not run E2E without confirming the dev server is running
- - Do not run the flaky audit with `--watch` mode โ€” always use `run` / `--count=1` for reproducibility
+ ## See also
- ## Attribution
+ - `/ccc-testing` โ€” testing domain router: TDD, E2E, load and visual testing
+ - `/ccc-e2e` โ€” parallel pre-release assessment in isolated worktrees
+ - `/ccc-review` โ€” code review before QA
+ - `/ccc-ship` โ€” release flow that reads the QA result
- > Adapted from gstack โ€” MIT licensed.
+ Adapted from gstack (MIT licensed).
---
> โš™๏ธ **Fable contract:** plan before build ยท verifier โ‰  worker ยท prove before alarm ยท loops need gates ยท leave durable state โ€” `rules/fable-method.md`