test-mutation git:20260818.6723100A

Immutable. This exact content never changes and is served at /api/v1/blob/a6e3c25fa765ce9f.

---
name: test-mutation
description: >
  Set up and run mutation testing (StrykerJS / mutmut) to measure whether tests
  assert behavior, not just execute lines. Use when "add mutation testing", "are
  our tests real", "can our test suite be gamed", or after an agent
  bulk-generated tests. Coverage plan → plan-test-coverage. Writing tests →
  test-unit.
license: MIT
---

# test-mutation — Coverage proves the test ran; mutants prove it would notice

Install and run the gate coverage cannot close. **Coverage proves the test
*ran* the code; mutation testing proves the test would *notice* if the
code were wrong.** The tool makes hundreds of small deliberate bugs
(mutants) — flip `>` to `>=`, delete a statement, replace a return — and
reruns the tests. Survivors mark code the tests execute but verify
nothing.

This is the specific counter to agent-generated suites: an agent can
trivially produce 90% coverage with no meaningful assertions, and every
coverage ratchet will wave them through. Mutants do not care about
coverage theater.

## This skill vs neighbors

| Skill | Owns |
|---|---|
| **test-mutation** (this) | Install / run / triage mutation testing; score ratchet |
| `plan-test-coverage` | What *should* be tested — plan only, no harness |
| `test-unit` | Write the tests |
| `audit-gate-logic` / `housekeep-gates` | Ratchet policy this score plugs into |
| `enhance-agent-guardrails` | Wording of the agent-rule hook |

---

## Phase 0 — Detect stack and scope deliberately

Identify the runner and pick the tool: **StrykerJS** (Jest / Vitest /
Mocha, TS-aware), **mutmut** / cosmic-ray (Python), or the ecosystem
built-in. Then scope — this step decides whether mutation testing
*survives* in the repo:

**Never start with the whole repo.** Each mutant is a test run. A 6-hour
first run gets the tool deleted. Scope the first pass to where a silent
bug costs most: money movement, auth, entitlements, data mutation, core
business logic. Explicitly exclude: generated code, config, type-only
files, UI glue better covered by `test-visual-regression`, and third-party
wrappers.

Record the suite's baseline runtime. Mutation runtime ≈ (mutants ×
affected-test runtime). Turn on per-test coverage analysis from day one
so only tests covering the mutated line rerun.

---

## Phase 1 — Configure for signal, not noise

- **Mutate patterns:** scoped critical paths only, via `mutate:` globs.
- **Incremental mode on** (Stryker `--incremental`): later runs only
  re-test mutants in changed code — this is what makes per-PR runs
  feasible.
- **Thresholds:** `high` / `low` for reporting, and `break` (CI-failing
  floor) at or slightly below the *measured* first score. Never invent an
  aspirational floor; measure, then ratchet. Same
  auto-tighten-with-reviewed-resets policy as `housekeep-gates`.
- **Timeouts:** mutants that loop are killed by timeout — keep the factor
  sane or runs balloon.
- **Reporters:** HTML for humans (each survivor in context), JSON for the
  ratchet.

---

## Phase 2 — First run and triage the survivors

Run, then classify every surviving mutant — this triage **is** the
deliverable:

1. **Missing assertion** — the test executes the line but asserts nothing
   about its effect. Fix: strengthen the test. Each of these is a hole a
   bug could walk through today.
2. **Untested branch** — no test reaches the mutated logic despite
   file-level coverage. Fix: add the case (hand to `plan-test-coverage`).
3. **Dead / vestigial code** — the mutant survives because the code has
   no observable effect. Fix: delete the code, not add a test
   (`workflow-housekeep`).
4. **Equivalent mutant** — the mutation does not change behavior (e.g.
   `<` vs `<=` on a boundary that cannot occur). No test can kill it;
   mark ignored with a comment. Expect a small percentage; a large one
   means the mutator is mutating the wrong things.

Prioritize by blast radius: a survivor in `entitlements.ts` outranks
fifty in a formatting helper.

---

## Phase 3 — Wire into CI (sustainably)

- **Per-PR:** incremental mutation on changed files only, as a job wired
  into the aggregator gate (`housekeep-gates`).
- **Scheduled full run** (nightly / weekly) over the mutated scope;
  publishes the score, updates the ratchet floor upward per policy, and
  files the new-survivor list.
- **Score ratchet:** `break` rises as the score does (flake tolerance);
  lowering it is a separate reviewed PR, never bundled.
- **Agent-rule hook:** bulk-generated tests must pass a mutation run over
  the touched files before "done" counts. `enhance-agent-guardrails`
  installs the wording.

---

## Definition of Done

- [ ] Tool chosen for the stack; per-test coverage analysis enabled
- [ ] Scope limited to critical paths with exclusions recorded; whole-repo explicitly deferred
- [ ] Incremental mode on; timeouts and reporters configured
- [ ] First run complete; baseline mutation score recorded
- [ ] Every survivor triaged into missing-assertion / untested-branch / dead-code / equivalent, with fixes filed by class
- [ ] `break` threshold set from the measured score and wired to auto-tighten; resets require a separate reviewed PR
- [ ] Per-PR incremental job in the aggregator gate; scheduled full run publishing the score
- [ ] Agent-rule hook added: generated tests validated by mutation before done
- [ ] Total added CI time measured and reported

## Output format

1. **Scope & config** — mutated paths | exclusions | tool + modes
2. **Score report** — baseline mutation score | per-module breakdown | coverage-vs-mutation gap (high coverage + low mutation score = assertion theater)
3. **Survivor triage** — mutant | file:line | class | fix filed
4. **CI wiring** — PR job, scheduled run, ratchet policy

Implement the harness and triage. Strengthen tests only on approval —
each fix is a reviewable change.