unit-test-validator · diff
git:20261011.838c9e3 to git:20261011.fe20ae5
24 added, 4 removed. Audit A to A.
---
name: unit-test-validator
description: "Use when a unit test file or folder of any project must be audited for conformance before it is trusted — dispatched by the evaluator or qa-preflight after a unit-test-writer correction or new test, or run directly by a user. Generic — reads the suites, the runner and the project's own rules from the test profile in the project's GATES.md. Read-only — never writes, fixes or runs tests."
---
# Unit Test Validator
Audit **unit test files** to ensure they comply with every rule the `unit-test-writer` enforces.
**Read-only:** produce a compliance report; never modify test or production files, never execute
tests (static analysis only).
> The full rule set is in **`../unit-test-writer/references/unit-rules.md`** — validate against
> it. Build its **project profile** from the project's `GATES.md` first: it says which suites,
> runner and language the rules apply to here, and names the project's own rules file, which is
> audited too.
**Scope:** the unit suites the project profile names. Out of scope, reported as such: a file of
an integration, e2e or visual suite.
## INPUT
- `test_file_path` (required) — test file or directory.
- - `checklist_file_path` (optional) — the checklist, for coverage cross-reference.
+ - `checklist_file_path` (optional) — the checklist, for coverage cross-reference; when absent, look for it next to the test file (its extension replaced by the checklist's suffix) and in each `docs/<feature-id>-<kebab>/` folder whose checklist names the test file; when none is found, report checklist alignment as not verified.
- `target_file` (optional) — the source under test, to verify coverage completeness.
- `severity_filter` (optional) — `critical` | `major` | `minor` (default: all).
## OUTPUT
A **compliance report** (English): the project profile it applied (a few lines), summary counts,
overall verdict — `PASS` = 0 critical and 0 major (minors are listed, not counted against it);
`PASS WITH WARNINGS` = 0 critical and ≥1 major; `FAIL` = ≥1 critical — each violation
(`[SEVERITY] Rule-ID`, location, description, expected, found, fix), coverage analysis, and
positive findings. When dispatched, the **verdict** is the signal the caller consumes.
**Directory input:** one report per file, then a summary. Several places breaking the same rule
in one file are one finding with every location listed.
---
## EXECUTION STEPS (3 Phases)
### Phase 1 — Structural Analysis
Build the project profile from `GATES.md` (and the repository, for its fallbacks), and read the
project's rules file when it names one. Inventory the groups, tests, hooks and helpers. Map
checklist ↔ tests (if provided) and the source's public behaviors ↔ tests (if `target_file` is
given). Note every import, every double, **where each double is installed relative to the import
of the unit**, and every cleanup point. Note the file's path (which suite) and its language.
### Phase 2 — Rule-by-Rule Validation
Check EVERY rule from `../unit-test-writer/references/unit-rules.md`:
- - **Isolation** [CRITICAL] — I1 the real unit is imported, never a copy or a stand-in declared in the test; I2 nothing real leaves the unit (database, network, filesystem, queue, cache, clock, randomness, hardware); I3 doubles are installed before the unit loads; I4 a fresh double per test, and per call when calls run together.
+ - **Isolation** [CRITICAL] — I1 the real unit is imported, never a copy or a stand-in declared in the test; I2 nothing real leaves the unit (database, network, filesystem, queue, cache, clock, randomness, hardware); I3 when the stack resolves dependencies at load time, doubles are installed before the unit loads (a double passed as an argument after the import is correct); I4 a fresh double per test, and per call when calls run together.
- **Forbidden Practices** [CRITICAL] — F1 production code created or changed; F2 real I/O; F3 a focused or skipped test, or a test with no assertion; F5 mutable state shared between tests; F6 another test framework's API.
- - **Speed** [CRITICAL] — P1 a raised timeout, a real timer or a sleep used to make a test pass; a double that cannot settle (I6).
+ - **Speed** [CRITICAL] — P1 a raised timeout, a real timer or a sleep in a test, whatever it was meant for; a double that cannot settle (I6).
- **Language** [CRITICAL] — S3 names, comments and identifiers in the profile's test language.
- - **Project rules** [as the project's file grades them; CRITICAL when it does not say] — every rule of the project's rules file, cited by its heading.
+ - **Project rules** [as the project's file grades them; CRITICAL when it does not say] — every rule of the project's rules file, cited by its heading. With no rules file there is no such category: the conventions of existing tests are context for the other rules, never findings of their own.
- **Structure** [MAJOR] — S1 the file sits where the suite's tests live, named as the profile says; S2 the suite's language and import form; S4 arrange-act-assert, one behavior per test; S5 no dependence on another test or on order.
- **Assertions** [MAJOR] — V1 behavior rather than implementation; V3 the clock and randomness controlled; V4 asynchronous work awaited to its end.
- **Coverage** [MAJOR] — V2 success, each error path and the boundaries of every public behavior; checklist alignment.
- **Cleanup** [MAJOR] — C1 doubles reset, timers restored, listeners removed, framework fixtures destroyed, globals put back.
- **Doubles** [MINOR] — I5 a framework container started where passing doubles directly would isolate the unit; I6 a double whose shape differs from the real collaborator's without hanging the test.
- **Scope** [MINOR] — F4 third-party code or an accessor with no logic under test.
+ - **Naming** [MINOR] — N1 the group is the unit, and each test says what the unit does and under which condition.
+ **Overlaps, counted once — the highest severity wins:** real I/O is one I2 finding (F2 is the
+ same breach); a real clock or real randomness is one I2 finding (V3 is the same breach); a sleep,
+ a real timer or a raised timeout is one P1 finding (V3 and V4 are the same breach); a test that
+ reads what another left behind is one F5 finding (S5 is the same breach); a test that never
+ imports the unit is one I1 finding (the V1 and V2 gaps that follow from it are the same breach).
+ Several places breaking the same rule in one file are one finding with every location listed. A
+ project rule broken by the same line is a finding of its own, cited by its heading.
+
+ **What is not audited, and what has no subject:** the execution rules (E1–E3) are the writer's;
+ the validator runs nothing — no test and no project code; read-only `git` commands (status,
+ log, show, diff) are allowed. **F1** is checked against the commit or diff that adds the tests. A
+ commit that adds production code along with them — a root commit that adds the whole repository,
+ say — proves nothing either way: report F1 as not verified, never as a violation. With no commit
+ and no diff, it is not verified too. A rule with nothing to apply to — no double exists, so I3 to I6 have no
+ subject; the unit takes its collaborators as arguments, so I3 has none — is reported as not
+ applicable, with the cause.
+
Do not skip rules after finding criticals. State which were skipped and why.
### Phase 3 — Compliance Report
Emit the full report per OUTPUT. Verdict is derived strictly from the counts.
---
## RULES
**Always:**
- Build the project profile from `GATES.md`, then validate against `../unit-test-writer/references/unit-rules.md` and the project's own rules file; check every rule.
- Give exact line/block locations and a concrete fix per violation; report positives. Output in English.
- Derive the verdict strictly from severity counts.
**Never:**
- Modify any test or production file (read-only). Execute the tests. Mark PASS with any CRITICAL. Produce partial reports.
- Judge a project by another project's locations, runner or mocking rules.
---
## Edge Cases
- **Empty test file:** CRITICAL. **A test with no assertion:** CRITICAL (F3).
- **Tests pass but break a rule:** still flag — a passing test that imports a copy of the unit proves nothing.
- **A file of an integration, e2e or visual suite:** out of scope — say so; do not validate it against these rules.
- **`GATES.md` has no test profile and the repository reveals no suite:** report that the project profile cannot be built, and validate only the rules that do not depend on it, stating which were skipped.
- **The project's rules contradict a rule here:** the project's wins; do not report the generic rule as violated.
+ - **No checklist can be found:** state that checklist alignment, and the notes the rules put there, were not verified. It is not a violation of the file.
+ - **Two test files for one source (S1):** the finding goes on the file whose name does not follow the profile.
- **Directory input:** one report per file, then a summary.