test-audit · diff
git:20260902.29a022a to git:20260906.015072f
5 added, 1 removed. Audit A to A.
---
name: test-audit
description: "Batch audit of test files against Q1-Q25 quality gates and AP1-AP32 anti-patterns. Detects orphan tests, phantom mocks, untested public methods. Tiered output (A/B/C/D) with critical gate enforcement and optional post-audit fix workflow. Flags: zuvo:test-audit all | [path] | [file] | --deep | --quick | --include-e2e | --details | --commit=ask|auto|off"
category: Code/Test audits
codesift_tools:
always:
- analyze_project
- index_status
- index_folder
- index_file
- plan_turn
- get_file_tree # discover *.test.* / *.spec.* / __tests__/
- get_file_outline
- search_text
- search_symbols # untested public methods (production-side)
- get_symbol
- get_symbols
- find_references # orphan-test + untested-public-method detection (no AP id — this is the
# pre-scan's ORPHAN/untested-methods signal, not an anti-pattern)
- find_dead_code # AP17 unused test data declared but never referenced
- find_clones # AP18 duplicate test names / copy-pasted test bodies
- search_patterns
- audit_scan
- scan_secrets # hardcoded credentials in fixtures (no AP id — a security finding,
# reported separately from the AP deduction)
by_stack:
typescript: [get_type_info]
javascript: []
python: [python_audit, analyze_async_correctness]
php: [php_project_audit, php_security_scan, resolve_php_namespace]
kotlin: [analyze_sealed_hierarchy, find_extension_functions, trace_flow_chain, trace_suspend_chain, trace_compose_tree, analyze_compose_recomposition, trace_hilt_graph, trace_room_schema, analyze_kmp_declarations, extract_kotlin_serialization_contract]
nestjs: [nest_audit]
nextjs: [framework_audit, nextjs_route_map]
astro: [astro_audit, astro_actions_audit, astro_hydration_audit]
hono: [analyze_hono_app, audit_hono_security]
express: []
fastify: []
react: [react_quickstart, analyze_hooks, analyze_renders]
django: [analyze_django_settings, effective_django_view_security, taint_trace]
fastapi: [trace_fastapi_depends, get_pydantic_models]
flask: [find_framework_wiring]
jest: []
yii: [resolve_php_service]
prisma: [analyze_prisma_schema]
drizzle: []
sql: [sql_audit]
postgres: [migration_lint]
---
# zuvo:test-audit — Test Quality Triage
Systematic evaluation of unit and integration test files through the Q1-Q25 binary checklist and AP anti-pattern catalog. Each test file is paired with its production source, scored against behavioral coverage standards, and assigned a tier.
**Scope:** Unit and integration tests only. E2E tests (`*/e2e/*`, `*.e2e.*`) are excluded by default. Use `--include-e2e` to include them.
**When to use:** After mass test writing, when test quality is uncertain, before releases, when test failures are hard to diagnose, periodic health check.
**Out of scope:** Single-file code review (use `zuvo:review`), writing new tests (use `zuvo:write-tests`), fixing systematic anti-patterns across many files (use `zuvo:fix-tests`).
## Argument Parsing
| Argument | Effect |
|----------|--------|
| `all` | Audit every test file in the project |
| `[path]` | Audit test files under a specific directory |
| `[file]` | Audit a single test file with full evidence (forces deep mode) |
| `[file file2 …]` | Audit exactly the listed test files (space-separated, forces deep mode) — the form the Test Quality Gate (`../../shared/includes/test-quality-gate.md`, called from build/refactor/execute/write-tests/write-e2e) uses to scope to touched files |
| `--deep` | Collect evidence and fix recommendations for every file |
| `--quick` | Binary pass/fail only, skip evidence |
| `--include-e2e` | Include E2E test files in scope |
| `--details` | Save per-file reports to `zuvo/audits/test-audit-details/` |
| `--commit=ask\|auto\|off` | Commit behavior after fix workflow (default: `ask`) |
| `--read-only` | Report only: skip Phases 4-6 (coverage.md/backlog.md writes) and Phase 7; no repo mutation beyond the `zuvo/audits/` report |
Default: `all --quick --commit=ask`
**Dispatch is already authorized — do not ask, and do not substitute.** Invoking this skill IS the
request for the gates it mandates. A session-level instruction like "do not use the Agent tool unless
the user asked" does NOT apply here: the user asked, by invoking this skill. Reading it as a
prohibition and recording a self-scored result is the substituted gate this step forbids — it
happened twice in the field (2026-08-07, 2026-08-08), the second time invented as
`WARN:substituted-inline`, a value no vocabulary defines. If the harness genuinely has no dispatch
capability (Cursor, Antigravity — NOT Codex, which dispatches mechanical workers), follow the ONE documented exception in
`test-quality-gate.md`; otherwise dispatch.
| Mode | Scope | Depth | Commit | Notes |
|------|-------|-------|--------|-------|
| `all` | Entire project | Standard | `--commit=ask` | Default |
| `[path]` | Directory | Standard | `--commit=ask` | Scoped |
| `[file]` | Single file | Deep | `--commit=ask` | Full evidence |
| `[file file2 …]` | Listed files only | Deep | `--commit=ask` | File-list scope (Test Quality Gate callers pass `--read-only --commit=off`) |
| `--deep` | Any scope | Full evidence + fixes | Per flag | Thorough |
| `--quick` | Any scope | Binary only | `--commit=off` | Fast triage |
| `--include-e2e` | + E2E files | Standard | Per flag | Expanded scope |
| `--details` | Any scope | + per-file reports | Per flag | Save individual files |
## Mandatory File Loading
- Read these files from disk before starting. Print the checklist. Do not proceed from memory.
+ First resolve `../../shared/includes/execution-policy.md` and
+ `../../shared/includes/evidence-reuse.md`. Reuse the parent policy and verified evidence for
+ a nested stage. Load only the rules needed now, with the read-once receipt protocol.
+
+ Load the applicable definitions using the read-once protocol. Defer logging/retro includes until completion.
```
CORE FILES LOADED:
1. ../../rules/testing.md -- READ/MISSING
2. ../../shared/includes/test-edge-cases.md -- READ/MISSING
3. ../../shared/includes/env-compat.md -- READ/MISSING
4. ../../shared/includes/run-logger.md -- READ/MISSING
5. ../../shared/includes/retrospective.md -- READ/MISSING
6. ../../shared/includes/no-pause-protocol.md -- READ/MISSING (HARD: no mid-batch pauses)
7. ../../shared/includes/test-quality-gate.md -- READ/MISSING (carries the dispatch-authorization rule)
8. ../../shared/includes/test-metrics.md -- READ/MISSING (frozen quality/cost/speed formulas — cite TIER_DIST/ECHO_COUNT, never restate)
```
**If any file is missing:** Stop. The quality gate definitions are required for scoring.
## Environment Compatibility
Read `../../shared/includes/env-compat.md` for agent dispatch patterns, path resolution, and progress tracking.
## MANDATORY TOOL CALLS — Test Audit Validity Gate
**INVALID if any tool below is skipped when trigger holds.** "DEFERRED", "N/A", "--quick mode" NOT valid reasons.
| Tool | Trigger | Skip allowed? |
|------|---------|---------------|
| `find_dead_code` | Always | **NO** — AP17 unused test data |
| `find_clones` | Always | **NO** — AP18 duplicate test names / copy-pasted bodies |
| `find_references` | Always | **NO** — orphan tests + untested public methods (no AP id) |
| `search_patterns` | Always | **NO** — Q-checklist anti-patterns |
| `audit_scan` | Always | **NO** — compound check |
| `scan_secrets` | Always | **NO** — hardcoded credentials in fixtures (security, not an AP) |
| Stack-specific tools | Framework/language detected | **NO** when matches |
Forbidden: `find_dead_code: skipped`, `codesift: unavailable` (when deferred), `retrospective: skipped` — all REJECTED.
POSTAMBLE: report on disk → retro appended → `~/.zuvo/append-runlog` exit 0. Every Q-fail/AP finding needs `path/to/file.ext:LINE` (verify-audit gate).
```
Mandatory-tools-acknowledgment: I will run find_dead_code + find_clones + find_references + search_patterns + audit_scan + scan_secrets + stack-specific tools for this test audit. Every finding will cite a `path/to/file.ext:LINE` resolving in the current tree.
```
---
## CodeSift Integration
**Use the deterministic preload helper FIRST.** Run `~/.zuvo/compute-preload test-audit "$PWD"` before any ToolSearch. Copy `[CodeSift matching trace]` verbatim, issue printed `ToolSearch(query="select:...")`. Math gate enforced.
Read `../../shared/includes/codesift-setup.md` for the full initialization sequence.
**Summary:** Run the CodeSift setup from `codesift-setup.md` at skill start. Use CodeSift for file discovery and production code analysis when available. If unavailable, fall back to standard tools.
### CodeSift Optimizations
| Task | CodeSift | Fallback |
|------|----------|----------|
| Find test files | `get_file_tree(repo, name_pattern="*.test.*")` | `find` command |
| Understand test structure | `get_file_outline(repo, file_path)` | `Read` each file |
| Batch-read test cases | `get_symbols(repo, symbol_ids=[...])` | Multiple `Read` calls |
| Find production file for test | `search_symbols(repo, query, kind="function")` | Path-based heuristic |
| Verify test imports | `find_references(repo, symbol_name)` | `Grep` for imports |
| Pre-scan for weak assertions | `search_text(repo, "toBeTruthy\|toBeDefined", file_pattern="*.test.*")` | `Grep` |
### Degraded Mode (CodeSift unavailable)
All steps fall back to `find`/`Read`/`Grep`/`Glob`. File discovery is slower and production file pairing relies on path conventions rather than symbol resolution.
---
## Phase 0: Discovery and Pairing
### 0.1 Locate Test Files
When CodeSift is available: `get_file_tree(repo, name_pattern="*.test.*")` with path filters excluding `node_modules`, `.next`, `e2e` (unless `--include-e2e`).
When unavailable:
```bash
find . \( -name "*.test.ts" -o -name "*.test.tsx" -o -name "*.spec.ts" -o -name "*.spec.tsx" \
-o -name "test_*.py" -o -name "*_test.py" \) \
! -path "*/node_modules/*" ! -path "*/.next/*" ! -path "*/__pycache__/*" ! -path "*/e2e/*" | sort
```
If count exceeds 50 and `--deep` was not explicitly requested, auto-switch to `--quick`. Explicit `--deep` always takes precedence.
### 0.2 Pair with Production Files
For each test, identify its production counterpart:
- `__tests__/api/projects/[id]/route.test.ts` -> `app/api/projects/[id]/route.ts`
- `tests/unit/services/bar.test.ts` -> `lib/services/bar.ts`
If production file not found: flag as ORPHAN (test without source).
When CodeSift is available, use `search_symbols` or `find_references` for more reliable pairing in non-standard project layouts.
### 0.3 Pre-Batch Grouping
Before splitting into batches, group test files by production file. If multiple test files target the same production code (`foo.test.ts` + `foo.errors.test.ts`), they MUST go into the same batch so suite-aware Q7/Q11 scoring works correctly.
### 0.4 Golden File Calibration (recommended for first audit)
If this is the first audit of a project or agent scores seem inconsistent:
1. Pick 2-3 test files with known quality (one good, one bad, one mid)
2. Run a single calibration agent on those files
3. Compare scores to expectations. If drift >2 points, adjust prompt wording
4. Proceed with full evaluation
### 0.5 Batch Output Directory
```bash
mkdir -p zuvo/audits/.test-audit-batch
```
Each batch agent writes results here. Cleaned up after the final report.
---
## Phase 1: Batch Evaluation
Split grouped files into batches of 8-10. For each batch, spawn a Task agent or process inline.
Each Task agent dispatch:
```
Agent: Test Quality Auditor (per batch)
model: "sonnet"
type: "general-purpose" # read-only: Read + CodeSift only, no Edit/Write (Explore lacks mcp__codesift__*)
instructions: evaluate test files against Q1-Q25 and AP anti-patterns (see Agent Prompt below)
input: batch file list with paired production files, CODESIFT_AVAILABLE
```
### Agent Prompt (provided to each batch agent)
```
You are a test quality auditor. Evaluate each test file below against Q1-Q25.
RED FLAG PRE-SCAN (do FIRST, before full evaluation):
- Tests with zero expect() calls (AP13) -> AUTO TIER-D. RTL exception: getByRole/getByText/getByLabelText are implicit assertions.
- Fixture:assertion ratio > 20:1 (AP16) -> AUTO TIER-D
- 50%+ of tests use toBeTruthy()/toBeDefined() as sole assertion (AP14) -> AUTO TIER-D
QUICK HEURISTICS:
- 0 CalledWith in entire file -> likely score <=4
- 10+ DI providers in test setup -> likely score <=5
- Tests calling __privateMethod() directly -> likely score <=5
POSITIVE INDICATORS:
- Factory with named overrides -> likely >=8
- Regression anchor in test name -> mature suite
- it.each with table-driven data -> Q8+Q9+Q11 likely pass
PRODUCTION CODE ANALYSIS (do BEFORE scoring):
Read the production file and extract:
1. Public API surface: all exported functions/methods
2. Branch map: all if/else, switch, ternary with line numbers
3. Enum/union values with their members
4. Error handling: try/catch, thrown errors, rejected promises
5. Complexity: THIN (<50 LOC, <=3 branches), STANDARD (50-200 LOC), COMPLEX (>200 LOC or >10 branches)
COMPLEXITY EXPECTATIONS:
| Complexity | Expected tests | Q11 depth | Edge case scope |
|------------|---------------|-----------|-----------------|
| THIN | 8-15 | cache/wiring only | null/undefined on params |
| STANDARD | 15-40 | all branches | full edge case checklist |
| COMPLEX | 40-80 (split files) | all branches + combos | full checklist + matrix |
CHECKLIST (score 1=YES, 0=NO):
<!-- GATES:BEGIN kind=q-prompt -->
Q1: Every test name describes expected behavior?
Q2: Tests grouped in logical describe blocks?
Q3: Every mock has CalledWith + not.toHaveBeenCalled?
Q4: Assertions use exact matchers (toEqual/toBe, not toBeTruthy)?
Q5: Mocks are typed (no `as any`)?
Q6: Mock state fresh per test (beforeEach, no shared mutable)?
Q7: CRITICAL -- Every error-throwing path tested with specific type+message?
Q8: Null/empty/edge inputs tested?
Q9: Repeated setup (3+ tests) extracted to helper/factory?
Q10: No magic values -- test data is self-documenting?
Q11: CRITICAL -- All code branches exercised?
Q12: Symmetric: "does X when Y" has "does NOT do X when not-Y"?
Q13: CRITICAL -- Tests import actual production function?
Q14: Behavioral assertions (not just mock-was-called)?
Q15: CRITICAL -- Content/values assertions, not just counts/shape?
Q16: Cross-cutting isolation: change to A verified not to affect B?
Q17: CRITICAL -- Assertions verify COMPUTED output, not input echo?
Q18: No flaky signals? No Date.now() without fake timers, no setTimeout for timing, no Math.random(), no real network?
Q19: Tests fully isolated? No shared mutable state between tests; each runs independently in any order?
Q20: CONDITIONAL -- Test level declared (small/medium/large) and not mixed?
Q21: CONDITIONAL -- Mutation score >= 70% on changed files, or every survivor triaged?
Q22: CONDITIONAL -- Pure/validator unit has a property test with a recorded seed?
Q23: CONDITIONAL -- Cross-service contract verified against a shared artifact, not a hand-written mock?
Q24: CONDITIONAL -- Suite green under randomized order, seed logged?
Q25: CONDITIONAL -- Patch coverage >= 90% on changed lines, enforced in CI?
<!-- GATES:END kind=q-prompt -->
ANTI-PATTERNS (each unique AP = -1 from score, max -5):
<!-- GATES:BEGIN kind=ap-list -->
AP1: try/catch in test swallowing errors
AP2: Conditional assertions (if/else in test)
AP3: Re-implementing production logic in test
AP4: Snapshot as only test for component
AP5: `as any` -> `as never` bypassing types
AP6: Testing CSS classes instead of behavior
AP7: .catch(() => {}) swallowing errors
AP8: document.querySelector bypassing Testing Library
AP9: Always-true assertion (expect(true).toBe(true))
AP10: Tautological mock (call mock -> verify mock called, no production code)
AP11: vi.mocked(vi.fn()) -- mock targeting fresh fn
AP12: waitForTimeout(N) hardcoded delays
AP13: Test with zero expect() calls -- AUTO TIER-D
AP14: toBeTruthy()/toBeDefined() as sole assertion on complex object
AP15: Testing private methods directly
AP16: Fixture:assertion ratio > 20:1 -- AUTO TIER-D
AP17: Unused test data declared but never used
AP18: Duplicate test names (copy-paste indicator)
AP19: expect.anything() hiding callback contract
AP20: Mock returns same data for ALL methods
AP21: .calls[N] magic index (fragile)
AP22: CSS selector in test
AP23: Inline mockRestore() with afterEach present (redundant)
AP24: consoleSpy typed as `any`
AP25: `expect(x.length).toBe(N)` instead of `.toHaveLength(N)` — worse failure output, masks a missing property (Q4). JS-only: pytest/Go/Rust have no equivalent, mark N/A there
AP26: Real timers in time-dependent tests (Date.now/setTimeout without useFakeTimers)
AP27: `expect(x.length).toBeGreaterThan(0)` when the fixture's exact count is known — masks off-by-one and duplicates (Q4/Q15)
AP28: Persistent `it.skip`/`describe.skip`/`@Ignore`/`#[ignore]`/`@pytest.mark.skip` with no ticket or expiry — dead code plus a silent coverage gap
AP29: Mock return value echoed in the assertion — proves the mock setup, not production logic (Q17). The most common audit failure
AP30: Mocking own code that could run with a real implementation (was AP25 until the numbering fork was resolved; overlaps `fix-tests` P-68 and Q13)
AP31: Committed focus marker — `it.only`/`describe.only`/`test.only`/`fit`/`fdescribe`/`@pytest.mark.only`-style filter left in a committed test file. Silently disables the REST of the suite while CI reports green — a whole-suite coverage collapse, worse than a skip. Deterministic detector: grep / Biome `noFocusedTests`. **AUTO TIER-D**
AP32: Flake-masking retries — per-test/per-suite `retries:`/`@Retry`/`flaky=True` annotation with no ticket or expiry (same contract as AP28's skip rule). Retry hides the nondeterminism Q18/Q24 exist to surface
<!-- GATES:END kind=ap-list -->
N/A HANDLING: N/A items excluded from both numerator and denominator. Score = passed / applicable.
Q16 N/A: score N/A when test covers single function/hook with no shared mutable state.
Q17 PASS-THROUGH: For thin controllers that are pure delegation, `expect(result).toEqual(mockReturn)` with CalledWith on service mock = Q17=1.
CRITICAL GATE: Q7, Q11, Q13, Q15, Q17 -- any = 0 -> Tier C floor (see TIER CLASSIFICATION).
SCORING MATH:
Applicable = 25 - N/A-count - out-of-scope-count # 19 gates, Q1-Q25. Was hardcoded 17 while the checklist
# claimed Q1-Q25 — Q18 (flaky) and Q19 (isolation) were
# advertised and never scored.
Score = yes-count / applicable (percentage)
AP deduction: each unique AP = -1 from yes-count (max -5)
ONE SCALE ONLY: the percentage below is the verdict. Do not also compare raw counts —
that produced two answers for one file (14/17 was simultaneously "PASS" and "Tier B")
and left score 9 belonging to no tier at all.
### Scoring Q21 — read the number, never estimate it
Q21's text lives in the generated region above; how to *answer* it does not, so it lives
here. Source of truth: the newest `$ZUVO_DIR/audits/mutation-test-*.json` written by
`zuvo:mutation-test` (§4.3b of that skill).
| Condition | Q21 value |
|---|---|
| `plan_completed: false` | `N/A (mutation run incomplete — <executed>/<planned>)` — check this FIRST |
| JSON present, `commit` == HEAD sha7 | score from **`score_triaged`** |
| JSON present, `commit` != HEAD | `N/A (mutation data STALE — <json sha7>, HEAD is <sha7>)` |
| JSON present, `tier2_ran: false` | score it, and append `(--quick: survivors never checked against the full suite)` |
| No JSON at all | `N/A (no mutation run)` — legitimate; Q21 is CONDITIONAL on a runner existing |
Use `score_triaged`, never `score_raw`. An *equivalent mutant* cannot be killed by any
test, so counting it against the suite fails work nobody can fix — and a gate people
cannot satisfy is a gate people learn to route around.
`plan_completed` is checked before anything else because a truncated run produces a score
that looks exactly like a complete one. A user hit this on 2026-08-10: the budget stopped
the run after 3 of 10 mutants and a score was printed anyway. A sample scored as the whole
plan is worse than no data — no data is honestly `N/A`.
Until 2026-08-09 `zuvo:mutation-test` wrote nothing to disk: the report went to chat and
vanished. This gate asked for a number the system never produced in readable form, so the
only available answers were a guess or `N/A`. Estimating it from test-file appearance is
not a third option — if the JSON is absent or stale, say so and score `N/A`.
FOR AUTO TIER-D FILES, use SHORT format:
### [filename]
Production file: [path or ORPHAN]
Red flags: [AP13/AP14/AP16] -> AUTO TIER-D
Phantom mocks: [list mocked modules not called by production code, or "none"]
Reason: [brief]
Top 3 gaps: [brief]
FOR ALL OTHERS, use FULL format:
### [filename]
Production file: [path or ORPHAN]
Complexity: [THIN/STANDARD/COMPLEX] ([LOC] LOC, [N] branches)
Red flags: ["none"]
Phantom mocks: [list or "none"]
Untested methods: [list of public methods with no test coverage, or "all covered"]
Score: Q1=[0/1] Q2=[0/1] ... Q25=[0/1]
Anti-patterns: [AP IDs found, or "none"]
Total: [yes]/[applicable] ([%]) - [AP count] = [adjusted%]
Critical gate: Q7=[0/1] Q11=[0/1] Q13=[0/1] Q15=[0/1] Q17=[0/1] -> [PASS/FAIL]
Tier: [A/B/C/D]
Top 3 gaps: [brief]
TIER CLASSIFICATION (derived from the percentage above — no separate count scale):
A (>= 82%, all critical gates = 1): No action needed
B (>= 53% and < 82%, all critical gates = 1): Fix gaps -- 2-5 targeted fixes
C (< 53%, OR any critical gate = 0): Major rewrite needed
D (AUTO TIER-D red flag: AP13, AP16, or AP31 (committed `.only`/focus marker — it disables the REST of the suite while CI stays green, so it is the most destructive of the three)): Delete and rewrite from scratch
A critical gate at 0 is a FLOOR (Tier C), not a ceiling. Previously it "capped at Tier B",
so a tautological suite scoring 16/17 landed in the same bucket as an honest 10/17 — and
test quality punished a critical failure LESS than code quality does (where critical = 0
is a hard FAIL). Now the two families agree.
IMPORTANT:
- Read BOTH the test file AND its production file
- Red flag pre-scan first
- COVERAGE COMPLETENESS: List all public methods in production file. For each, check if test exercises it. Flag untested methods. Exclude control flow keywords, built-ins, SQL keywords. API endpoint exception: test calling client.get("/path") IS testing the handler. Page component exception: render(<Component />) IS testing the export. Re-export exception: only test functions DEFINED in the file, not re-exports.
- PHANTOM MOCK DETECTION: List all mocked modules in test. For each: does production code actually call it? Unused mock = phantom mock.
- SUITE-AWARE MODE: Sibling test files for same production file -- evaluate Q7/Q11 at suite level.
- Q17 ECHO vs COMPUTED: mock returns X, test asserts X = echo (Q17=0). Mock returns raw data, test asserts transformation = computed (Q17=1).
- Q15 API ROUTE CALIBRATION: Status code checks, error body checks, response field checks, auth guard verification all count as Q15=1 for API routes.
- AP21 CALIBRATION: `.mock.calls[N]` = fragile (AP21). `.toHaveBeenNthCalledWith(N, ...)` = Jest API, not AP21.
Write complete output to: zuvo/audits/.test-audit-batch/batch-{N}.md
Files to audit:
[BATCH FILE LIST]
```
---
## Phase 2: Aggregate Results
Read all batch files from `zuvo/audits/.test-audit-batch/`:
1. Glob for `zuvo/audits/.test-audit-batch/batch-*.md`
2. Parse summary tables for tier counts
3. Parse per-file blocks for detailed analysis
4. If any batch file is missing (agent failure), log the gap
Build the summary report:
```markdown
# Test Quality Audit Report
Date: [date]
Project: [name]
Files audited: [N]
Total tests: [count from test runner]
## Summary by Tier
| Tier | Count | % | Action |
|------|-------|---|--------|
| A (>=82% of applicable) | [N] | [%] | No action |
| B (>= 53% and < 82% of applicable) | [N] | [%] | Fix gaps |
| C (<53% of applicable) | [N] | [%] | Major rewrite |
| D (<5 or red flag) | [N] | [%] | Delete + rewrite |
| ORPHAN | [N] | [%] | Verify or delete |
## Critical Gate Failures
| File | Score | Failed Qs | Top Gap |
|------|-------|-----------|---------|
## Red Flag Summary (Auto Tier-D)
| File | Red Flag | Details |
|------|----------|---------|
## Untested Public Methods
| File | Untested Methods | Impact |
|------|-----------------|--------|
## Top Failed Questions (across all files)
| Question | Fail count | % of files | Pattern |
|----------|-----------|------------|---------|
## Anti-pattern Hot Spots
| Anti-pattern | Files affected | Instances |
|-------------|---------------|-----------|
## Tier D -- Rewrite Queue
## Tier C -- Major Fix Queue
## Tier B -- Targeted Fix Queue
## Tier A -- No Action
```
Save to: `zuvo/audits/test-quality-audit-[date].md` — at the **project root** (`zuvo/` resolves via `git rev-parse --show-toplevel`; override `$ZUVO_OUTPUT_DIR`. See `../../shared/includes/report-output-location.md`).
If `--details` flag: also save per-file reports to `zuvo/audits/test-audit-details/`
## Phase 3: Cleanup Batch Files
```bash
rm -rf zuvo/audits/.test-audit-batch
```
## Phase 3b: Adversarial Review on Audit Report (MANDATORY — do NOT skip)
After the audit report is generated, run cross-model validation to catch Q-score inflation and coverage theater. Runs on ALL audits (not just --deep).
```bash
~/.zuvo/adversarial-review --mode tests --files "zuvo/audits/test-quality-audit-[date].md"
```
If `adversarial-review` is not in PATH: `~/.zuvo/adversarial-review` (stable; the versioned cache path breaks after any release)
Wait for complete output. Then:
- **CRITICAL** (passing Q-score contradicted by evidence) → fix in report before delivery
- **WARNING** (coverage theater not flagged) → append to Known Gaps section
- **INFO** → ignore
## Phase 4: Coverage Registry Update
Under `--read-only`: skip Phases 4-6 entirely and present the report (Phase 7 fix workflow is also off).
Read `memory/coverage.md`. If it does not exist, create it now.
For each audited test file, find its production file row in coverage.md:
| Audit Tier | Coverage Status | Rationale |
|-----------|----------------|-----------|
| A (>=16, gate PASS) | COVERED | Tests are solid |
| B (10-15 or gate FAIL >=10) | PARTIAL-QUALITY | Has tests but quality issues |
| C (<53% of applicable) | PARTIAL-QUALITY | Major quality gaps |
| D (<5 or red flag) | PARTIAL | Effectively untested |
Only downgrade coverage status, never upgrade. If production file is not yet in coverage.md, add it.
Output: `COVERAGE UPDATE: [N] rows updated ([N] downgraded, [N] confirmed, [N] new)`
## Phase 5: Backlog Persistence
Persist findings to `memory/backlog.md`:
1. Read `memory/backlog.md`. If missing, create with template.
2. Fingerprint each finding: `file|Q/AP-id|signature`. Dedup: existing = increment `Seen`.
3. Delete resolved items.
Full protocol: `../../shared/includes/backlog-protocol.md`.
**What to persist:**
- **Tier C/D files:** all findings. Source: `test-audit/{date}`. Category: Test.
- **Tier B critical gate failures** (Q7/Q11/Q13/Q15/Q17=0): separate item per gate
- **Auto Tier-D red flags** (AP13/AP14/AP16): always persist as HIGH
## Phase 6: Persistence Verification
Before presenting the report, verify all writes completed:
```
PERSISTENCE VERIFICATION
coverage.md updated: [N] rows ([N] downgraded, [N] confirmed, [N] new)
backlog.md updated: [N] entries ([N] new, [N] deduped)
batch files cleaned: [yes/no]
```
If any step is incomplete, go back and finish it before continuing.
## Phase 7: Post-Audit Fix Workflow
After presenting the report, the user may request fixes:
1. **Fix** -- rewrite test files following the quality rules
2. **Test** -- run the test suite to confirm all tests pass
3. **Verify** -- for each fixed file:
- Only test files modified (no production code changes)
- Full test suite green
- All modified test files <= 400 lines
- Q1-Q25 self-eval on each fixed file
- Tier improvement confirmed (D->C+, C->B+, B->A)
4. **Commit** -- behavior per `--commit` flag (ask/auto/off)
5. **Re-audit** -- optionally re-run on fixed files to verify improvement
## Next-Action Routing
| Finding | Action | Command |
|---------|--------|---------|
| Tier D files (any AUTO TIER-D red flag, or a critical Q gate = 0) | Rewrite tests | `zuvo:write-tests [path]` |
| Same AP across 10+ files | Batch fix | `zuvo:fix-tests --pattern [AP-ID]` |
| Tier B-C with Q7=0 | Add error tests | `zuvo:write-tests [path]` |
| Coverage gaps (methods untested) | Write missing tests | `zuvo:write-tests [path]` |
| Test infra issues (runner config) | Optimize runner | `zuvo:tests-performance` |
## Completion Gate Check
Before printing the final output block, verify every item. Unfinished items = pipeline incomplete.
```
COMPLETION GATE CHECK
[ ] Red flag pre-scan ran on every batch
[ ] Phantom mock detection ran: unused mocks listed
[ ] Untested public methods listed per file
[ ] Adversarial review ran on audit report
[ ] Coverage registry updated: memory/coverage.md rows written
[ ] Backlog updated for critical gate failures
[ ] Report saved to zuvo/audits/
[ ] Run: line printed and appended to log
```
## TEST AUDIT COMPLETE
### Validity Gate (REQUIRED — print BEFORE Run line, AFTER retro append + append-runlog)
```
VALIDITY GATE
triggers_held: language=<X> framework=<X> test_runner=<X>
required_tool_calls:
find_dead_code: [<N> orphan helpers | NOT_CALLED — VIOLATES_TRIGGER]
find_clones: [<N> dup tests | NOT_CALLED — VIOLATES_TRIGGER]
find_references: [<N> ref-checks | NOT_CALLED — VIOLATES_TRIGGER]
search_patterns: [<N> hits | NOT_CALLED — VIOLATES_TRIGGER]
audit_scan: [<N> findings | NOT_CALLED — VIOLATES_TRIGGER]
scan_secrets: [<N> hits | NOT_CALLED — VIOLATES_TRIGGER]
stack_specific: [<result> | not_required | NOT_CALLED — VIOLATES_TRIGGER]
postamble:
retros_log_appended: [yes(bytes_added=N) | NOT_APPENDED]
retros_md_appended: [yes(entry_count=N) | NOT_APPENDED]
verify_audit_pass: [yes(<verified>/<total>) | NOT_RUN | REJECTED]
gate_status: [PASS | FAIL — <which gates missing>]
```
If `gate_status = FAIL` → VERDICT = INCOMPLETE.
Append the Run line via the retro-gated wrapper (NOT direct `>> runs.log`):
```bash
printf '%b\n' "$RUN_LINE" | ~/.zuvo/append-runlog
```
Run: <ISO-8601-Z>\ttest-audit\t<project>\t<N-critical>\t<N-total>\t<VERDICT>\t-\t<N>-dimensions\t<NOTES>\t<BRANCH>\t<SHA7>\t<INCLUDES>\t<TIER>
### Retrospective (REQUIRED)
Follow the retrospective protocol from `retrospective.md`.
Gate check → structured questions → TSV emit → markdown append.
If gate check skips: print "RETRO: skipped (trivial session)" and proceed.
After printing this block, append the `Run:` line value (without the `Run: ` prefix) to the log file path resolved per `run-logger.md`.
VERDICT: PASS (0 critical findings), WARN (1-3 critical), FAIL (4+ critical).
---
## Execution Notes
- Use **Sonnet** for batch agents in both QUICK and DEEP modes
- Process batches sequentially in Cursor/Codex. Claude Code may parallelize with up to 6 Task agents.
- Run the project's test suite first to confirm baseline passes. Auto-detect runner from config files.
- Estimated durations: QUICK ~2 min for 50 files, DEEP ~10 min for 50 files