integration-test-validator · diff
git:20261011.fe20ae5 to git:20261012.b0ec4ea
28 added, 15 removed. Audit A to A.
---
name: integration-test-validator
description: "Use when an integration test file or folder of any project must be audited for conformance before it is trusted — dispatched by the evaluator or qa-preflight after an integration-test-writer correction or new test, or run directly by a user. Generic — reads the environment, the helpers, the data policy and the project's own rules from the test profile in the project's GATES.md. Read-only — never writes, fixes or runs tests."
---
# Integration Test Validator
Audit **integration test files** to ensure they comply with every rule the
`integration-test-writer` enforces. **Read-only:** produce a compliance report; never modify
test or production files, never execute tests (static analysis only).
> The full rule set is in **`../integration-test-writer/references/integration-rules.md`** —
> validate against it. Build its **project profile** from the project's `GATES.md` first: it
> says which suites, helpers, removal order and query counter the rules apply to here, and names
> the project's own rules file, which is audited too.
+ > **The suite's language file,** `../integration-test-writer/references/languages/<language>.md`,
+ > is part of the rules: resolve the language as `integration-rules.md` ("Language") says, and
+ > read that one file.
**Scope:** the integration suites the project profile names. Out of scope, reported as such: a
file of a unit, e2e or visual suite.
## INPUT
- `test_file_path` (required) — test file or directory.
- - `checklist_file_path` (optional) — the checklist, for coverage and query-growth cross-reference; when absent, look for it next to the test file (its extension replaced by the checklist's suffix) and in each `docs/<feature-id>-<kebab>/` folder whose checklist names the test file; when none is found, report checklist alignment as not verified.
+ - `checklist_file_path` (optional) — the checklist, for coverage and query-growth cross-reference; when absent, look for it next to the test file (its extension replaced by `.integration-test.md`) and in each `docs/<feature-id>-<kebab>/` folder whose checklist names the test file; when none is found, report checklist alignment as not verified.
- `target_file_or_endpoint` (optional) — the handler or endpoint under test.
+ - `language` (optional) — the suite's language; when given, it wins over what the profile and the repository say.
- `severity_filter` (optional) — `critical` | `major` | `minor` (default: all).
## OUTPUT
A **compliance report** (English): the project profile it applied (a few lines), summary counts,
overall verdict — `PASS` = 0 critical and 0 major (minors are listed, not counted against it);
`PASS WITH WARNINGS` = 0 critical and ≥1 major; `FAIL` = ≥1 critical — each violation
- (`[SEVERITY] Rule-ID`, location, description, expected, found, fix), coverage analysis, and
- positive findings. When dispatched, the **verdict** is the signal the caller consumes.
+ (`[SEVERITY] Rule-ID`, location, description, expected, found, fix), the rules not verified or not applicable with the cause
+ of each, coverage analysis, and positive findings. When dispatched, the **verdict** is the signal the caller consumes.
- **Directory input:** one report per file, then a summary. Several places breaking the same rule
+ **Directory input:** one report per test file — a checklist or any other file in the folder is
+ not audited — then a summary: a row per file with its counts and verdict, and the folder's
+ verdict, derived from the summed counts. Several places breaking the same rule
in one file are one finding with every location listed.
---
## EXECUTION STEPS (3 Phases)
### Phase 1 — Structural Analysis
Build the project profile from `GATES.md` (and the repository, for its fallbacks), and read the
- project's rules file when it names one. Inventory the groups, tests, hooks and helpers. Map
+ project's rules file when it names one. Resolve the suite's language and read its language file;
+ when the language has none, say so and go on with the generic rules. Inventory the groups,
+ tests, hooks and helpers. Map
checklist ↔ tests and endpoint ↔ tests (if provided). Identify **every data-creation point and
the cleanup that answers it** — the guaranteed block, the suite-level hook, the removal order.
Note the entry point each request uses, the credential it carries, and the file's imports.
### Phase 2 — Rule-by-Rule Validation
Check EVERY rule from `../integration-test-writer/references/integration-rules.md`:
- - **Data** [CRITICAL] — D1 nothing the test did not create is changed or removed; D2 the test creates its own records; D3 every data-creating test removes its data in a block the runner guarantees, never in a suite-level hook alone; D4 no removal fails silently — an empty `catch` is a violation; D5 children before parents, in the profile's order; D6 by stored id only, no broad delete; D7 no manual cleanup stacked on a suite that isolates another way.
+ - **Data** [CRITICAL] — D1 nothing the test did not create is changed or removed; D2 the test creates its own records; D3 every data-creating test removes its data in a block that runs even when the test fails, never in a suite-level hook alone; D4 no removal fails silently — a swallowed failure is a violation; D5 children before parents, in the profile's order; D6 by stored id only, no broad delete; D7 no manual cleanup stacked on a suite that isolates another way.
- **Forbidden Practices** [CRITICAL] — F1 production code created or changed; F2 data left behind; F3 cleanup of a test's own data only in a suite-level hook; F4 an external data-population script; F5 a focused or skipped test, or a test with no assertion; F6 the handler, router or data-access layer under test replaced by a double.
- **Query growth** [CRITICAL] — only when the profile declares a query counter. Cite the id of each violated Q rule instead of paraphrasing it. Check:
- each endpoint in the section's scope has exactly one recorded outcome in the checklist;
- every `not measurable` or `pre-existing N+1` outcome holds against the code, and no other reason is used;
- every growth test meets every Q rule.
**Check only what the dispatch targets.** In a suite written or expanded from a checklist, that means the checklist's endpoints. When confirming a correction, it means only the tests the writer wrote or changed. Run directly on a file with no checklist, it means the growth tests that file contains: audit them against the Q rules, and state that recorded outcomes were not verified. A growth test missing for an endpoint outside that target is not a violation: legacy suites have none.
- - **Language** [CRITICAL] — S3 names, comments and identifiers in the profile's test language.
+ - **Test language** [CRITICAL] — S3 names, comments and identifiers in the profile's test language.
- **Project rules** [as the project's file grades them; CRITICAL when it does not say] — every rule of the project's rules file, cited by its heading. With no rules file there is no such category: the conventions of existing tests are context for the other rules, never findings of their own.
- **Structure** [MAJOR] — S1 location and name; S2 the suite's language and import form; S4 arrange-act-assert; S5 independence; S6 the environment opened in the suite's setup and closed in its teardown, through the profile's helpers.
- **Authentication** [MAJOR] — A1 the credential comes from the profile's helper, never hard-coded; A2 authenticated requests carry it.
- - **Coverage** [MAJOR] — V2 success, authentication and authorization, validation, not-found, boundaries, relationships, hostile input; checklist alignment. The scope is the endpoint the dispatch names — with none, each endpoint the file exercises. A category the checklist records as not applicable is not missing.
+ - **Tautology** [MAJOR] — V5 no expected value rebuilt with the product's own code — its serializer, its mapper, its query.
+ - **Coverage** [MAJOR] — V2 success, authentication and authorization, validation, not-found, boundaries, relationships, hostile input; checklist alignment. The scope is the endpoint the dispatch names — with none, each endpoint the file exercises. A category the checklist records as not applicable is not missing; with no checklist, neither is one the code shows the endpoint cannot exhibit.
- **Assertions** [MINOR] — V1 status asserted on every request, body on success, error shape on failure; V3 a write confirmed by reading it back; V4 no comparison against a literal count of shared data.
- **Naming** [MINOR] — N1 the group names the endpoint, and each test says what it does and under which condition.
+ - **Language anti-patterns** [as the language file grades each] — every anti-pattern of the suite's language file, cited by its id. With no language file there is no such category.
**Overlaps, counted once — the highest severity wins:** data left behind because a test has no
guaranteed cleanup is one D3 finding (F2 and F3 are the same breach); a seed record changed and
never restored is one D1 finding; a test that relies on data another created is one D2 finding
(S5 is the same breach); a growth test with no guaranteed cleanup is one D3 finding (Q4 is the
same breach). Every other violated Q rule is a finding of its own, even when one test breaks
- several. Several places breaking the same rule in one file are one finding with every location
- listed.
+ several. A language anti-pattern that is the form a generic rule's breach takes is one finding,
+ citing both ids. Several places breaking the same rule in one file are one finding with every
+ location listed.
- **What is not audited, and what has no subject:** the execution rules (E1–E4) are the writer's;
+ **What is not audited, and what has no subject:** the execution rules (E1–E5) are the writer's;
the validator runs nothing — no test and no project code; read-only `git` commands (status,
log, show, diff) are allowed. **F1** is checked against the commit or diff that adds the tests. A
commit that adds production code along with them — a root commit that adds the whole repository,
say — proves nothing either way: report F1 as not verified, never as a violation. With no commit
- and no diff, it is not verified too. A rule with nothing to apply to is reported as not applicable, with the
- cause.
+ and no diff, it is not verified too. Where the language file says tests live inside the source
+ file, a change confined to its test module is test code, and a changed line outside it is an F1
+ violation. A rule with nothing to apply to is reported as not applicable, with the cause.
Do not skip rules after finding criticals. State which were skipped and why.
### Phase 3 — Compliance Report
Emit the full report per OUTPUT. Verdict is derived strictly from the counts.
---
## RULES
**Always:**
- - Build the project profile from `GATES.md`, then validate against `../integration-test-writer/references/integration-rules.md` and the project's own rules file; check every rule.
+ - Build the project profile from `GATES.md`, then validate against `../integration-test-writer/references/integration-rules.md`, the suite's language file and the project's own rules file; check every rule.
- Give exact line/block locations and a concrete fix per violation; report positives. Output in English.
- Derive the verdict strictly from severity counts.
**Never:**
- Modify any test or production file (read-only). Execute the tests. Mark PASS with any CRITICAL. Produce partial reports.
- Judge a project by another project's helpers, removal order or data rules.
---
## Edge Cases
- **Empty test file:** CRITICAL. **A test with no assertion:** CRITICAL (F5).
- **Data created with no guaranteed cleanup:** CRITICAL (D3). **Removal in the wrong order:** CRITICAL (D5). **A swallowed removal failure:** CRITICAL (D4).
- **The profile says the suite's isolation replaces cleanup** (a rollback per test, a database per worker): D3 to D6 are met by that mechanism — do not report their absence; report manual cleanup stacked on top (D7). A private database whose data policy still names removal helpers is not that case: D3 to D6 apply as written.
- **The profile declares no query counter:** skip the query-growth category and say so.
+ - **The suite's language has no language file:** validate against the generic rules alone, read in the language's own idiom, and say so in the profile lines of the report.
- **A file of a unit, e2e or visual suite:** out of scope — say so; do not validate it against these rules.
- **`GATES.md` has no test profile and the repository reveals no suite:** report that the project profile cannot be built, and validate only the rules that do not depend on it, stating which were skipped.
- **No checklist can be found:** state that checklist alignment, not-applicable categories and recorded query-growth outcomes were not verified. It is not a violation of the file.
- - **Directory input:** one report per file, then a summary.
+ - **Directory input:** one report per test file, then the summary with the folder's verdict, as OUTPUT describes.