integration-test-validator · diff
git:20260930.17fecc4 to git:20261011.838c9e3
42 added, 27 removed. Audit A to A.
---
name: integration-test-validator
- description: Audits PABX backend integration tests (apps/backend/__tests__/integration/) against the integration-test-writer rules, producing a compliance report with a PASS/FAIL/PASS WITH WARNINGS verdict, per-violation severity, and fix suggestions. Read-only — never writes or fixes tests. Dispatched by the evaluator to confirm a corrected integration test conforms before resuming evaluation, or run directly by a user.
+ description: "Use when an integration test file or folder of any project must be audited for conformance before it is trusted — dispatched by the evaluator or qa-preflight after an integration-test-writer correction or new test, or run directly by a user. Generic — reads the environment, the helpers, the data policy and the project's own rules from the test profile in the project's GATES.md. Read-only — never writes, fixes or runs tests."
---
- # Integration Test Validator (PABX)
+ # Integration Test Validator
Audit **integration test files** to ensure they comply with every rule the
`integration-test-writer` enforces. **Read-only:** produce a compliance report; never modify
- test/production files, never execute tests (static analysis only).
+ test or production files, never execute tests (static analysis only).
- > **Project scope:** PABX backend. The full rule set is in
- > **`../integration-test-writer/references/pabx-rules.md`** — validate against it.
+ > The full rule set is in **`../integration-test-writer/references/integration-rules.md`** —
+ > validate against it. Build its **project profile** from the project's `GATES.md` first: it
+ > says which suites, helpers, removal order and query counter the rules apply to here, and names
+ > the project's own rules file, which is audited too.
- **Scope:** `apps/backend/__tests__/integration/`.
+ **Scope:** the integration suites the project profile names. Out of scope, reported as such: a
+ file of a unit, e2e or visual suite.
## INPUT
- `test_file_path` (required) — test file or directory.
- - `checklist_file_path` (optional) — the `.test.md` for coverage cross-reference.
- - `target_file_or_endpoint` (optional) — the controller/endpoint under test.
+ - `checklist_file_path` (optional) — the checklist, for coverage and query-growth cross-reference.
+ - `target_file_or_endpoint` (optional) — the handler or endpoint under test.
- `severity_filter` (optional) — `critical` | `major` | `minor` (default: all).
## OUTPUT
- A **compliance report** (English): summary counts, overall verdict (`PASS` = 0 critical + 0
- major; `PASS WITH WARNINGS` = 0 critical, some major/minor; `FAIL` = ≥1 critical), each violation
+ A **compliance report** (English): the project profile it applied (a few lines), summary counts,
+ overall verdict — `PASS` = 0 critical and 0 major (minors are listed, not counted against it);
+ `PASS WITH WARNINGS` = 0 critical and ≥1 major; `FAIL` = ≥1 critical — each violation
(`[SEVERITY] Rule-ID`, location, description, expected, found, fix), coverage analysis, and
- positive findings. When dispatched by the evaluator, the **verdict** is the signal it consumes.
+ positive findings. When dispatched, the **verdict** is the signal the caller consumes.
+ **Directory input:** one report per file, then a summary. Several places breaking the same rule
+ in one file are one finding with every location listed.
+
---
## EXECUTION STEPS (3 Phases)
### Phase 1 — Structural Analysis
- Inventory `describe`/`it`/hooks/helpers. Map checklist ↔ tests and endpoint ↔ tests (if provided).
- Identify every data-creation point and every cleanup point (`try-finally`, `afterEach`,
- `afterAll`, direct deletes). Note imports.
+ Build the project profile from `GATES.md` (and the repository, for its fallbacks), and read the
+ project's rules file when it names one. Inventory the groups, tests, hooks and helpers. Map
+ checklist ↔ tests and endpoint ↔ tests (if provided). Identify **every data-creation point and
+ the cleanup that answers it** — the guaranteed block, the suite-level hook, the removal order.
+ Note the entry point each request uses, the credential it carries, and the file's imports.
### Phase 2 — Rule-by-Rule Validation
- Check EVERY rule from `../integration-test-writer/references/pabx-rules.md`:
- - **Language & Naming** [CRITICAL] — English only; descriptive names; no double blank lines.
- - **Test Structure** [MAJOR] — AAA; independence; deterministic; `setupTestDatabase`/`cleanupTestDatabase`/`supertest`/`app` file pattern; no prod modification.
- - **Data Cleanup** [CRITICAL] — `try-finally` per data-creating test; `try-catch` inside `finally`; FK order (`audit_logs` → `users_permissions`/`user_permissions_group` → `group_permissions` → `permissions` → `users` → `dr_agent` → `dr_domain`); only test-created data; no data left behind; ≥2 cleanup levels.
- - **Execution Setup** [MAJOR] — E1–E3 of `pabx-rules.md`: `setupTestDatabase()` in `beforeAll` and the token from `generateToken`; `cleanupTestDatabase()` in `afterAll`; correct imports.
- - **Coverage** [MAJOR] — CRUD, auth/authorization, validation errors, not-found, edge cases, security (SQLi/XSS), checklist alignment.
- - **Query growth** [CRITICAL] — "Query growth" in `pabx-rules.md`, by its rule ids. Cite the id of each violated Q rule instead of paraphrasing it. Check:
+ Check EVERY rule from `../integration-test-writer/references/integration-rules.md`:
+ - **Data** [CRITICAL] — D1 nothing the test did not create is changed or removed; D2 the test creates its own records; D3 every data-creating test removes its data in a block the runner guarantees, never in a suite-level hook alone; D4 no removal fails silently — an empty `catch` is a violation; D5 children before parents, in the profile's order; D6 by stored id only, no broad delete; D7 no manual cleanup stacked on a suite that isolates another way.
+ - **Forbidden Practices** [CRITICAL] — F1 production code created or changed; F2 data left behind; F4 an external data-population script; F5 a focused or skipped test, or a test with no assertion; F6 the handler, router or data-access layer under test replaced by a double.
+ - **Query growth** [CRITICAL] — only when the profile declares a query counter. Cite the id of each violated Q rule instead of paraphrasing it. Check:
- each endpoint in the section's scope has exactly one recorded outcome in the checklist;
- every `not measurable` or `pre-existing N+1` outcome holds against the code, and no other reason is used;
- every growth test meets every Q rule.
**Check only what the dispatch targets.** In a suite written or expanded from a checklist, that means the checklist's endpoints. When confirming a correction, it means only the tests the writer wrote or changed. A growth test missing for an endpoint outside that target is not a violation: legacy suites have none.
- - **Forbidden Practices** [CRITICAL] — no prod modification; no external data scripts; no pre-existing-data removal; no afterEach-only cleanup for created data; no Portuguese.
- - **API Behavior** [MINOR] — status assertions; response-body assertions; correct HTTP methods; `Authorization` header on authenticated requests.
+ - **Language** [CRITICAL] — S3 names, comments and identifiers in the profile's test language.
+ - **Project rules** [as the project's file grades them; CRITICAL when it does not say] — every rule of the project's rules file, cited by its heading.
+ - **Structure** [MAJOR] — S1 location and name; S2 the suite's language and import form; S4 arrange-act-assert; S5 independence; S6 the environment opened in the suite's setup and closed in its teardown, through the profile's helpers.
+ - **Authentication** [MAJOR] — A1 the credential comes from the profile's helper, never hard-coded; A2 authenticated requests carry it.
+ - **Coverage** [MAJOR] — V2 success, authentication and authorization, validation, not-found, boundaries, relationships, hostile input; checklist alignment.
+ - **Assertions** [MINOR] — V1 status asserted on every request, body on success, error shape on failure; V3 a write confirmed by reading it back; V4 no comparison against a literal count of shared data.
Do not skip rules after finding criticals. State which were skipped and why.
### Phase 3 — Compliance Report
Emit the full report per OUTPUT. Verdict is derived strictly from the counts.
---
## RULES
**Always:**
- - Validate against `../integration-test-writer/references/pabx-rules.md`; check every rule.
+ - Build the project profile from `GATES.md`, then validate against `../integration-test-writer/references/integration-rules.md` and the project's own rules file; check every rule.
- Give exact line/block locations and a concrete fix per violation; report positives. Output in English.
- Derive the verdict strictly from severity counts.
**Never:**
- Modify any test or production file (read-only). Execute the tests. Mark PASS with any CRITICAL. Produce partial reports.
+ - Judge a project by another project's helpers, removal order or data rules.
---
## Edge Cases
- - **Empty test file:** CRITICAL. **Missing `setupTestDatabase`/imports:** report each individually.
- - **Data created without `try-finally`:** CRITICAL (F4). **Wrong FK cleanup order:** CRITICAL (C3).
- - **Directory input:** one report per file + a summary.
+ - **Empty test file:** CRITICAL. **A test with no assertion:** CRITICAL (F5).
+ - **Data created with no guaranteed cleanup:** CRITICAL (D3). **Removal in the wrong order:** CRITICAL (D5). **A swallowed removal failure:** CRITICAL (D4).
+ - **The suite isolates by rollback or by a database per worker:** D3 to D6 are met by that mechanism — do not report their absence; report manual cleanup stacked on top (D7).
+ - **The profile declares no query counter:** skip the query-growth category and say so.
+ - **A file of a unit, e2e or visual suite:** out of scope — say so; do not validate it against these rules.
+ - **`GATES.md` has no test profile and the repository reveals no suite:** report that the project profile cannot be built, and validate only the rules that do not depend on it, stating which were skipped.
+ - **Directory input:** one report per file, then a summary.