flaky-test-investigation · diff
git:20260502.145b611 to git:20260606.4d49a49
51 added, 12 removed. Audit A to A.
---
name: flaky-test-investigation
- description: "Use to diagnose tests that pass and fail intermittently without code changes."
+ description: Use when tests pass and fail intermittently without code changes, or a test passes alone but fails in the full suite.
---
- # Flaky Test Investigation
+ ## Purpose
- Flaky tests erode trust in CI. Do not just re-run them and hope for the best.
+ Flaky tests erode trust in CI. Do not just re-run them and hope for the best — isolate the flake vector, fix it, and prove the fix with a stress loop.
- ## Investigation Protocol
+ ## When to Use
- 1. **Isolate the Test**: Run the specific failing test by itself. If it passes, the flake is likely an **order dependency** or **state leakage** from a previous test.
- 2. **Stress Test**: Run the test in a tight loop (e.g., `for i in {1..100}; do npm test -- -t "My Test"; done`).
- 3. **Check for Common Vectors**:
- - **Time**: Does the test rely on `Date.now()` or `setTimeout`? Mock the clock.
- - **Async/Promises**: Are we asserting before a background task finishes? Ensure proper `await` or `waitFor` usage.
- - **Shared State**: Are we reusing database records, global singletons, or mutated variables between runs? Ensure clean teardowns in `afterEach`.
- - **Randomness**: Does the test rely on random IDs or sorts? Force deterministic seeds or sort orders.
- 4. **Prove the Fix**: Do not just guess. The fix must be verified by running the stress test loop again and achieving a 100% pass rate.
+ - A test fails intermittently in CI but passes locally (or vice versa)
+ - A test passes alone but fails in the full suite
+ - A re-run "fixed" a failure and nobody knows why
+
+ ## Inputs
+
+ - The flaky test's name/path and the runner command for it
+ - Recent failing runs, if available, to estimate the failure rate
+
+ ## Workflow
+
+ 1. **Isolate the test.** Run the specific failing test by itself. If it passes alone, the flake is likely an **order dependency** or **state leakage** from a previous test — run the suite up to and including it to confirm.
+
+ 2. **Stress test.** Run the test in a tight loop to establish the failure rate before changing anything:
+
+ ```bash
+ for i in {1..100}; do npm test -- -t "My Test" || echo "FAIL on run $i"; done
+ ```
+
+ (Adapt the inner command to the project's runner; some runners have repeat flags built in.)
+
+ 3. **Check the common vectors:**
+ - **Time** — does the test rely on `Date.now()` or `setTimeout`? Mock the clock.
+ - **Async/Promises** — asserting before a background task finishes? Ensure proper `await` or `waitFor` usage.
+ - **Shared state** — reusing database records, global singletons, or mutated variables between runs? Ensure clean teardowns in `afterEach`.
+ - **Randomness** — random IDs or sort orders? Force deterministic seeds or sort orders.
+
+ 4. **Prove the fix.** Do not just guess. The fix must be verified by running the stress test loop again and achieving a 100% pass rate.
+
+ ## Output
+
+ - The identified flake vector (order/state, time, async, randomness)
+ - The fix, plus stress-loop evidence (pre-fix failure rate vs post-fix 100% pass)
+
+ ## Verification
+
+ - [ ] Test run in isolation to separate order-dependency from intrinsic flake
+ - [ ] Stress loop run before the fix to establish a baseline failure rate
+ - [ ] Flake vector named explicitly
+ - [ ] Stress loop re-run after the fix with a 100% pass rate
+
+ ## Failure Modes
+
+ - **Re-run and hope** — a green re-run proves nothing; the flake is still there.
+ - **Fixing without a baseline** — without a pre-fix failure rate, a "fix" cannot be distinguished from luck.
+ - **Quarantining forever** — skipping the test removes the signal but keeps the bug.
+