test-strategy-hardening · diff
git:20260522.ba97b05 to git:20260801.b8d38b2
122 added, 32 removed. Audit A to A.
---
name: test-strategy-hardening
- description: Audit and harden a repository test suite so tests carry their weight. Use when the user asks whether tests are useful, flaky, duplicated, slow, under-covering critical behavior, missing contract/regression coverage, or when a codebase needs a test strategy before major refactors, releases, security work, or production hardening.
+ description: Write, review, refactor, consolidate, select, and debug software tests and test infrastructure so tests carry their weight. Use whenever Codex adds or changes tests, fixes flaky or nondeterministic tests, evaluates coverage, creates fixtures or mocks, removes duplicated tests, improves test runtime or CI selection, or audits a repository before refactors, releases, security work, or production hardening. Correct common LLM testing failures such as assertion inflation, copy-pasted cases, over-mocking, broad snapshots, wall-clock dependence, random fixtures, arbitrary sleeps, and running every test for every edit.
---
# Test Strategy Hardening
- Use this skill to turn a decorative or fragile test suite into a trustworthy engineering control. Optimize for confidence per minute, not raw coverage count.
+ Make each test prove a meaningful behavioral contract with the smallest faithful setup. Optimize for confidence per minute, diagnostic clarity, and determinism—not test count or coverage theater.
- ## Workflow
+ ## Establish the contract
- 1. **Inventory the test system**
- - Discover test commands, frameworks, e2e harnesses, fixtures, mocks, snapshots, CI config, coverage tooling, and runtime.
- - Classify tests by layer: pure unit, contract/schema, integration, e2e/browser, visual, smoke, load/perf, migration/data.
- - Record current pass/fail/flaky/skip status and approximate runtime.
+ Before writing or changing a test:
- 2. **Judge whether tests carry their weight**
- - Identify tests that pin real product contracts, past regressions, critical user journeys, external/provider parsing, migrations, permissions, and failure modes.
- - Flag broad snapshots, duplicate happy paths, brittle implementation assertions, no-op smoke tests, overmocked tests, and tests whose names exceed their proof.
- - Keep valuable slow tests, but move them to explicit suites if they block normal development.
+ 1. Read repository instructions, test commands, nearby tests, and existing fixtures.
+ 2. State the behavior that could regress and who observes it.
+ 3. Search for coverage of that behavior.
+ 4. Choose the lowest boundary that can prove it faithfully.
+ 5. Identify clocks, randomness, concurrency, storage, networks, globals, and cleanup.
- 3. **Write a decision-complete test plan**
- - Define the critical behaviors that must not regress.
- - Pick the minimum suite shape: fast local default, focused changed-surface suites, e2e golden paths, release smoke, and optional nightly/heavy checks.
- - Specify deletions, rewrites, new characterization tests, and fixtures before touching behavior.
+ Do not add a test merely because a source line changed. A useful test must fail for a plausible defect and pass for the intended behavior.
- 4. **Harden in green slices**
- - Add characterization tests before refactors.
- - Replace low-value tests with narrower tests that fail for the right reason.
- - Build contract tests around external/API/provider boundaries.
- - Add e2e/visual coverage only for user-critical journeys; visually inspect screenshots when UI matters.
- - Keep fixtures small, named, deterministic, and close to the behavior they prove.
+ ## Choose the boundary deliberately
- 5. **Report the result**
- - Summarize tests added, removed, merged, skipped, or quarantined.
- - Report runtime before/after and what confidence improved.
- - Leave explicit gaps and next target suites.
+ - Use pure tests for transformations, validation, reducers, parsers, policies, and invariants.
+ - Use module or component tests for real framework behavior.
+ - Use integration tests for storage, protocols, processes, concurrency, or multi-module contracts.
+ - Use end-to-end tests for a small number of critical user journeys and owning-runtime proofs.
+ - Use property tests when many inputs share a crisp invariant.
- ## Quality Bar
+ Prefer the lowest faithful boundary, not the lowest possible boundary. Do not mock away the database, browser, protocol, or race that defines the behavior under test.
- - A normal developer can run the fast suite frequently.
- - Critical product journeys have at least one high-signal regression path.
- - External contracts fail loudly before bad data reaches UI or business logic.
- - Tests have clear names, minimal mocks, deterministic fixtures, and useful failure messages.
- - The final report answers: "What bugs would this suite catch that it used to miss?"
+ ## Make assertions carry weight
- For the audit checklist, read [checklist.md](references/checklist.md).
+ Assert stable, externally meaningful outcomes: domain values, persisted state, ownership, error codes, protocol messages, user-visible behavior, accessibility, retries, rollback, authorization, and cleanup.
+
+ Reject tests that:
+
+ - return exactly what was programmed into a mock;
+ - copy production logic into the expected value;
+ - assert incidental ordering, generated IDs, timestamps, or private call sequences;
+ - replace semantic assertions with broad snapshots;
+ - update expected output without confirming intended behavior;
+ - would still pass if the feature returned a constant.
+
+ Use snapshots only when the representation itself is the contract and its diff remains reviewable. Use mutation reasoning: name one plausible incorrect implementation the test must catch.
+
+ ## Keep tests deterministic
+
+ - Inject or freeze time. Never depend on today remaining before a hard-coded expiry.
+ - Seed or inject randomness and ID generation.
+ - Isolate environment variables, globals, storage, providers, ports, and caches.
+ - Await background work and close resources in teardown.
+ - Synchronize concurrency through observable events or barriers.
+ - Replace sleeps with state, events, or bounded polling against a real condition.
+ - Keep real-network tests behind explicit integration commands.
+
+ Do not treat larger timeouts, retries, global serialization, skipped tests, or weaker assertions as flake fixes.
+
+ ## Consolidate without hiding behavior
+
+ Compare setup, action, and invariant before adding another case.
+
+ - Use table-driven cases when one contract varies only by input and expected outcome.
+ - Extract repeated realistic setup into feature-owned fixtures.
+ - Keep scenario-specific decisions visible at the call site.
+ - Avoid test-framework abstractions harder to understand than the duplication.
+ - Split large files by behavior or boundary.
+ - Retain focused tests when they localize failures that broad tests cannot.
+
+ Delete or merge a test only after naming its replacement coverage. “The integration test touches this code” is not replacement evidence unless it asserts the same invariant.
+
+ ## Diagnose flakes instead of masking them
+
+ 1. Reproduce without editing assertions.
+ 2. Run the smallest failure repeatedly and capture seed, order, timing, and environment.
+ 3. Classify the cause as test state, fixture design, infrastructure, or product behavior.
+ 4. Fix the source of nondeterminism.
+ 5. Add or tighten the regression assertion.
+ 6. Repeat the repaired test enough to exercise the former failure mode.
+ 7. Run neighboring and affected suites.
+
+ Treat exposed races, rollback failures, authorization gaps, and stale-state bugs as product defects. Quarantine only through an explicit repository mechanism with an owner, evidence, and removal condition. Never silently skip or delete a flaky test.
+
+ ## Control fixture and suite cost
+
+ - Measure setup, imports, execution, and teardown before optimizing.
+ - Share immutable baseline construction while preserving isolated mutable state.
+ - Key generated fixtures by every relevant input and fail closed on mismatch.
+ - Separate migration/setup verification from product tests that only need a current baseline.
+ - Keep one canonical path that rebuilds state from source.
+ - Prefer small representative fixtures unless scale is the behavior under test.
+
+ Do not cache test results by default. Cache dependencies, transforms, and immutable fixture inputs only with complete invalidation and a correct miss path.
+
+ Run tests proportionally during implementation:
+
+ 1. changed or nearest focused test;
+ 2. tests related to changed modules;
+ 3. affected package tests and typecheck;
+ 4. cross-package contracts when shared boundaries change;
+ 5. the full release suite when risk or repository policy requires it.
+
+ For CI, prefer explicit tiers:
+
+ - **affected:** change-aware package tests for normal changes;
+ - **contracts:** migrations, persistence, protocols, authorization, provider, and cross-package behavior;
+ - **full:** all tests, builds, audits, and end-to-end release gates.
+
+ Keep routing repository-owned and tested. Treat lockfiles, schemas, shared contracts, test configuration, generated code, and routing rules as broad invalidation triggers. Fall back wider when dependency analysis is ambiguous or dynamic. Never report a focused suite as proof that the full suite passed.
+
+ ## Audit an existing suite
+
+ For a suite-wide hardening pass:
+
+ 1. Inventory commands, layers, fixtures, mocks, snapshots, skips, CI, runtime, and current failures.
+ 2. Map critical behaviors and missing failure paths.
+ 3. Produce a keep, rewrite, merge, delete, quarantine, and add plan before broad edits.
+ 4. Harden in green, behavior-preserving slices.
+ 5. Measure confidence and runtime before and after.
+
+ Read [checklist.md](references/checklist.md) for the audit checklist.
+
+ ## Preserve behavior and finish with evidence
+
+ Do not reshape production code solely for shallow tests. Introduce clocks, ID sources, provider interfaces, or pure seams when they also clarify the production design. Keep behavioral fixes, test consolidation, and CI routing in separate commits when practical.
+
+ Report:
+
+ - contracts added or strengthened;
+ - tests merged or removed and their replacement coverage;
+ - flaky-test causes and deterministic fixes;
+ - focused, affected, contract, and full suites actually run;
+ - runtime or fixture-cost changes;
+ - suites not run and why;
+ - remaining gaps or quarantine.
+
+ The final result must answer: “What bugs would this suite catch that it used to miss?”