test-fix-loop git:20260811.b9a0fffA

Immutable. This exact content is served forever at /api/v1/blob/a59479aa4a5ded31.

---
name: test-fix-loop
description: "This skill should be used when autonomously iterating on test failures: runs the suite, diagnoses, applies minimal fixes, re-runs with checkpoint commit isolation until all tests pass."
---

# Test-Fix Loop

Autonomous test-fix iteration loop. Run the test suite, diagnose failures, apply fixes to implementation code, and re-run until all tests pass or a termination condition is met. This is a recovery mechanism for unexpected failures -- not a replacement for RED/GREEN/REFACTOR (use `atdd-developer` for TDD discipline).

## When to Use

- After implementation produces unexpected test failures in unrelated modules
- When `soleur:work` GREEN phase fails and manual diagnosis is tedious
- To batch-fix multiple test failures across a codebase
- NOT for writing new tests (use `atdd-developer`)
- NOT for linting, type-checking, or non-test failures

## Phase 0: Detect and Confirm

### Detect Test Runner

Auto-detect the test command from project files in priority order:

1. `CLAUDE.md` -- explicit test command (highest priority)
2. `package.json` -- `scripts.test` field
3. `Cargo.toml` -- `cargo test`
4. `Makefile` / `Justfile` -- `test` target
5. `Gemfile` / `Rakefile` -- `bundle exec rake test` or `bin/rails test`
6. `pyproject.toml` -- `pytest`
7. `go.mod` -- `go test ./...`

If `$ARGUMENTS` contains a custom test command, use it instead of auto-detection.
If `$ARGUMENTS` contains a number, use it as max iterations (default: 5).
If no runner is detected, ask the user for the test command.

### Require Clean Working Tree

Run `git status --porcelain`. If output is non-empty, STOP and tell the user to commit their changes first.

### Pre-flight Confirmation

<decision_gate>
**API budget.** Each iteration of this loop consumes one main-model turn (parse failures → cluster → fix → re-run) against the Anthropic API key in your Claude Code session. The `max iterations` cap (default 5, configurable via `$ARGUMENTS`) is the only cost ceiling — a runaway against a perpetually-flaky test command or an infinite-regression chain runs up to the cap before terminating. Soleur does not bill or proxy these calls — Anthropic does, against the key in your session. The Soleur LICENSE (BSL 1.1) disclaims warranty for runtime cost; you operate this loop against your own budget.

Show the user: detected test command, max iterations, current branch.
Get one confirmation before starting the loop. This is the only approval gate --
no per-iteration approval.
</decision_gate>

## Phase 1: Test-Fix Loop

Record the current commit SHA as `<initial-sha>` before entering the loop. This is the rollback target if the loop terminates on failure after multiple iterations.

Run the initial test suite and **capture its exit code** (`rc`) — the verdict is `rc`, not the parsed output. Exit with "All tests already pass. Nothing to fix." only when `rc` is 0. A run that exits non-zero while parsing to zero failures is not a pass: this repo's `test-all.sh` exits 3 when zero suites failed and at least one was terminated with a signal-shaped status, rendered `[KILLED]` rather than `[FAIL]` (see its `EXIT CONTRACT` block). Handle that through the *Suite terminated* row in §2 — never by entering the loop and never by reporting success.

For each iteration (up to max iterations):

### 1. Parse Failures

Extract failure summaries from test output: test name and error message only (one line each). Discard full stack traces and passing test output to minimize context consumption.

Distinguish build/compilation errors from test failures. If the suite fails to compile, treat the entire build error as a single cluster and fix the compilation issue first.

Count `^[KILLED]` lines separately from failures, and record both alongside `rc`. A terminated suite emits no assertion output, so it parses to zero failures and produces zero clusters: "non-zero `rc`, nothing parseable" is a real shape, and it is *unresolved coverage*, not an empty failure set. Never let it fall through to "all tests pass".

### 2. Check Termination Conditions

Before attempting fixes, check whether to stop. Rows are checked top to bottom; the first match wins.

**The count driving every delta row below is `failures + killed`, and a suite that is KILLED in either of the two iterations being compared is excluded from the delta rather than counted as fixed.** A failures-only count is arithmetically unsafe here: a suite that FAILED in iteration N and is KILLED in N+1 lowers it, which reads as a fabricated improvement; when the same suite completes again in N+2 the count jumps back, which reads as **Regression** and would `git reset --hard HEAD` over real fixes on the strength of a signal artifact.

| Condition | Detection | Action |
|-----------|-----------|--------|
| Suite terminated (unresolved) | Any `^[KILLED]` line in the output, or runner `rc` 3 | Do NOT stage, do NOT report success, do NOT reset. Re-run that suite alone; if it is KILLED again, STOP and report UNRESOLVED naming the suite |
| All tests pass | `rc` 0 (zero failures **and** zero killed) | Stage fixes with `git add -A`, report success |
| Max iterations | iteration == limit | `git reset --hard <initial-sha>` (revert all iterations), report |
| Regression | Failure count increased vs previous iteration | `git reset --hard HEAD` (discard uncommitted fixes), report |
| Circular fix | Failure name set matches any prior iteration | `git reset --hard <initial-sha>` (revert all iterations), report |
| Non-convergence | Failure count unchanged for 2 consecutive iterations | `git reset --hard <initial-sha>` (revert all iterations), report |
| Build error persists | Same compilation error after fix attempt | `git reset --hard <initial-sha>` (revert all iterations), report |

If a termination condition triggers, skip to the Diagnostic Report.

### 3. Cluster and Diagnose

Cluster failures by file or module (max 5 groups, sorted by failure count descending). If more than 5 modules fail, take the top 5 and note the skipped modules.

For each cluster, apply the diagnostic-first rule:

- Read the failing test to understand expected behavior
- Read the implementation code referenced by the error
- Identify the root cause before proposing a fix

### 4. Checkpoint and Fix

<critical_sequence>
Commit the current working tree as a rollback checkpoint before applying fixes. Skip on iteration 1 if the tree is clean -- `<initial-sha>` already serves as the rollback point.

    git add -A && git commit -m "test-fix-loop: checkpoint iteration N"

Apply fixes to implementation code only. NEVER modify test files, add skip annotations, delete tests, or weaken assertions.

Re-run the full test suite after applying fixes.

Evaluate the result (same `failures + killed` count and the same row order as §2):
- Any `^[KILLED]` line or `rc` 3: do NOT stage, do NOT report success, do NOT reset — take the *Suite terminated* row
- All pass (`rc` 0): stage all fixes with `git add -A`, report success, STOP
- Failures decreased: continue to next iteration (fixes stay in working tree; the next iteration's checkpoint commits them)
- Regression: `git reset --hard HEAD` (discard uncommitted fixes, return to checkpoint), STOP
- Circular or non-convergence: `git reset --hard <initial-sha>` (revert ALL iterations), STOP
- Max iterations reached: `git reset --hard <initial-sha>` (revert ALL iterations), STOP
</critical_sequence>

## Diagnostic Report

On termination (success or failure), write a report to stdout:

- **Result**: SUCCESS, REGRESSION, CIRCULAR, MAX_ITERATIONS, NON_CONVERGENCE, or UNRESOLVED
- **Iterations completed**: N out of max
- **Termination reason**: one-line explanation
- **Iteration history**: failure count per iteration with delta
- **Remaining failures**: test name and error message for each (if not success)
- **Fixes applied**: files modified and what changed (last iteration)
- **Recommendation**: what the user should investigate next (if not success)

On success, fixes are staged but NOT committed. The user reviews and commits via `/ship` or manually.

## Key Principles

- Diagnose before fixing -- never guess at the root cause
- Fix implementation code only -- tests define the contract
- Truncate aggressively -- failure summaries only, no full stack traces
- Fail safe -- checkpoint commit before every fix attempt, revert on regression
- Exit early -- stop as soon as the trajectory indicates non-convergence
- Stage, do not commit -- respect the Workflow Completion Protocol