improving-tests · diff

git:20260716.29b4d22 to git:20260928.d0a85d1

47 added, 148 removed. Audit A to A.

---

description: Improve test design, speed, and coverage with behavior-focused tests,

useful seams, characterization tests, TDD, and test refactoring. Use when improving

tests, optimizing slow suites, adding coverage, refactoring brittle tests, removing

test waste, or working test-first. NOT for fixing production bugs (use fixing-code),

production-code refactors (use refactoring-code), or reviewing non-test code quality

(use reviewing-code).

name: improving-tests

---

# Test Improvement

- Improve tests so they catch real behavior regressions without blocking safe code

- changes. Suite latency is a quality attribute. Coverage is a signal, not the goal.

- ## Role-gated action

- Detect capability from tools:

- - Write-capable role: inspect tests, apply changes, and run verification.

- - Read-only role: inspect supplied files/output and emit changes in the Proposed Changes contract. Apply nothing; run nothing.

- - Missing key tool or permission: stop with Blocked and ask for the exact artifact, access, or approval needed.

- Use an interactive question tool when available for mode selection, missing scope, missing framework approval, or unsafe test-stack choices.

- ## Route elsewhere

- Do not use this for:

- - production bug fixes → `fixing-code`

- - production-code refactors → `refactoring-code`

- - non-test code review → `reviewing-code`

- - new feature implementation unless the user asked for TDD

- - browser-only UI investigation without a test-improvement goal → `browser-automation`

- ## References

+ Make tests catch real behavior regressions without blocking safe changes. Suite

+ latency is a quality attribute; coverage is a signal, not the goal.

- Detect languages from files in scope and read only the matching reference:

+ Without write access, return proposed changes (file, change, reason) instead of

+ applying them.

- - C# → `references/csharp.md`

- - Go → `references/go.md`; for slow-suite or feedback-loop work, also read `references/go-performance.md`

- - Java/Kotlin → `references/java-kotlin.md`

- - Python → `references/python.md`; for slow-suite or feedback-loop work, also read `references/python-performance.md`

- - TypeScript → `references/typescript.md`; for slow-suite or feedback-loop work, also read `references/typescript-performance.md`

- - Rust → `references/rust.md`

- - Web → `references/web.md`

+ Read the reference for each language in scope; it covers test patterns and

+ suite speed. Use the matching `writing-<lang>` skill for toolchain commands.

- Use generic rules only for unsupported languages.

+ - C#: `references/csharp.md`

+ - Go: `references/go.md`

+ - Java/Kotlin: `references/java-kotlin.md`

+ - Python: `references/python.md`

+ - Rust: `references/rust.md`

+ - TypeScript/JavaScript: `references/typescript.md`

+ - Browser, Playwright, HTMX: `references/web.md`

## Modes

- - `review`: find weak, duplicate, brittle, missing, slow, or flaky tests

- - `refactor`: simplify tests without changing covered behavior

- - `coverage`: add tests for uncovered business behavior or error paths

- - `tdd`: one red-green-refactor slice at a time

- - `performance`: measure test latency and remove speed waste without weakening behavior

- - `full`: review, refactor, performance, and add coverage

- If mode is missing, ask one question with these options.

- ## Choose the seam

- Test through the contract that users or adjacent modules rely on:

- - Public module, package, API, CLI, component, or service boundary.

- - Integration seam when behavior depends on real wiring: database, filesystem, HTTP, queue, cache, framework routing, serialization, or config.

- - Unit seam when behavior is pure, local, deterministic, and cheap to exercise.

- Use graph tools only when available and when they help choose the seam or risk:

- - GitNexus: use query/context to find flows around a behavior; use impact or detect-changes to choose regression tests for changed symbols and affected processes.

- - codegraph: check freshness first; if fresh, use affected/context to find callers, high fan-in surfaces, and modules that need regression coverage.

- - Stale graph indexes are not evidence. Refresh if allowed; otherwise report the gap and use search, coverage output, and source reads.

- ## Test rules

- - Test behavior, not private helpers, call counts, or layout.

- - Treat slow feedback as test waste. Fast tests make agents verify more often.

- - Remove waits, real I/O, redundant setup, expensive collection, unbalanced parallelism, and broad default commands before reducing checks.

- - Keep coverage, race, mutation, live, browser, and end-to-end modes off the hot path unless the task is about that mode.

- - Mock only system boundaries: network, clock, randomness, filesystem, subprocesses, external services.

- - Prefer real collaborators or in-memory fakes for internal domain code.

- - Cover success, failure, edge, boundary, and regression cases that matter.

- - Use coverage to find gaps; do not write low-value assertions just to raise a number.

- - Delete shallow or duplicate tests once stronger public-boundary tests cover the behavior.

- - Extract helpers only after repeated setup or assertions make tests harder to read.

- - Follow project conventions before introducing new frameworks, helpers, or generators.

- ## Feedback-loop performance

- For performance mode or any slow-suite work:

- 1. Measure the baseline command and wall time.

- 2. Classify the bottleneck: discovery, import/compile/transform, setup, test body, external boundary, runner config, or parallel balance.

- 3. Make one cluster of changes that removes waste without weakening behavior.

- 4. Rerun the same command and record before/after time.

- 5. Add or recommend a guard: durations output, per-test ceiling, slow marker, cache, or focused command.

- Prefer focused deterministic checks during edits and the broader relevant suite before final output. Do not hide failures, delete edge cases, or skip important fast tests to make a number look better.

- ## TDD and characterization

- TDD:

- 1. Name one behavior at the public seam.

- 2. Write one failing test that fails for the expected reason.

- 3. Implement the smallest passing code.

- 4. Refactor only while green.

- 5. Repeat one behavior at a time.

- Characterization tests:

- - Use before risky changes to legacy or under-specified code.

- - Capture current externally visible behavior, including quirks.

- - Place tests at the public boundary first; add narrower tests only when they add diagnostic value.

- ## Review checks

- Look for:

- - tests coupled to private helpers, internals, or incidental call order

- - mocks hiding real behavior or contracts

- - duplicate scenario matrices that should be parameterized when readability stays high

- - missing business, error, edge, concurrency, or permission cases

- - flaky tests from time, randomness, ordering, shared state, or real external services

- - slow tests that could move down a seam without losing confidence

- - real sleeps, real external I/O, coverage-on-default, expensive imports, broad discovery, repeated setup, or slow transforms in the fast path

- - order dependencies, leaked globals, or shared mutable resources that block safe parallelism

- - dead tests that cover deleted behavior or generated glue

- Prefer each language's table-driven or parameterized pattern from `references/<lang>.md` when cases share setup and assertions. Do not force consolidation when separate tests make distinct behavior clearer.

- ## Verification

- Run the relevant project command after changes, for example `pytest -q --maxfail=1 --tb=short`. Exact commands per language live in `references/<lang>.md`.

+ If the mode is missing, ask which one:

- Use coverage commands only when coverage mode or review needs them. Report skipped checks with exact reasons.

+ - `review`: find weak, duplicate, brittle, missing, slow, or flaky tests.

+ - `refactor`: simplify tests without changing covered behavior.

+ - `coverage`: add tests for uncovered business behavior and error paths.

+ - `tdd`: one red-green-refactor slice at a time.

+ - `performance`: cut suite latency without weakening behavior.

+ - `full`: all of the above.

- ## Output

+ ## Rules

- Engineer:

+ - Ask before adding a test framework, runner plugin, or tool. Learn the framework, helpers, and conventions from nearby tests and follow them first.

+ - Change production code only inside an approved TDD slice. If no safe behavior seam exists, stop and report what production change would create one.

+ - Test through the contract users or adjacent modules rely on: public module, API, CLI, component, or service. Use an integration seam when behavior depends on real wiring (database, filesystem, HTTP, serialization, config); a unit seam when behavior is pure and cheap.

+ - Mock only system boundaries (network, clock, randomness, filesystem, subprocesses, external services). Prefer real collaborators or in-memory fakes for domain code.

+ - Assert behavior, not private helpers, call counts, or layout. Business-critical arguments get exact matches.

+ - Parameterize cases that share setup and assertions; keep separate tests when that reads clearer.

+ - Delete shallow or duplicate tests once stronger boundary tests cover the behavior.

+ - Characterization tests before risky changes to legacy code capture current visible behavior, quirks included, at the public boundary.

+ - TDD: each slice has one test that failed for the expected reason before the smallest passing code, and refactoring happens only while green. No bulk suites for imagined behavior.

+ - If a code-graph tool (GitNexus, codegraph) is installed and fresh, use it to find affected flows and high fan-in code that needs regression coverage.

- ```text

- TEST IMPROVEMENT COMPLETE

- =========================

- Tests changed: N

- Waste removed: N

- Coverage: before → after | not measured

- Performance: baseline → after | not measured

- Status: CLEAN | NEEDS ATTENTION

+ ## Speed

- Key improvements:

- - path:line — change

+ Performance work records the same command's wall time before and after, names

+ the bottleneck (discovery, import or compile, setup, test body, external

+ boundary, runner config, or parallel balance), and leaves a guard against

+ regression (durations output, per-test ceiling, slow marker, or focused command).

- Verification:

- - <command> — pass/fail/skipped with reason

- ```

+ - Remove waste before removing checks: real sleeps, real external I/O, repeated setup, expensive imports, broad discovery, coverage-on-default.

+ - Keep coverage, race, mutation, browser, live, and end-to-end modes off the default loop; run them in their own tier.

+ - Parallelism needs isolated state: per-test or per-worker ports, temp dirs, databases, and filenames. Treat parallel-only failures as isolation bugs.

+ - Never make a number look better by hiding failures, deleting edge cases, or skipping fast tests.

- Reviewer or blocked:

+ Done when the relevant build/test/lint checks pass on what you changed, or you

+ name each check that did not run and why.

- ```text

- ## Proposed Changes | BLOCKED

+ ## Report

- Blocker:

- - <missing artifact, framework, tool, permission, or safe seam>

+ Mode, tests changed, key changes (`path:line — change`), coverage and timing

+ before → after when measured, and each check with pass, fail, or skipped and why.

- ### Change 1: <brief description>

+ ## Platform additions

- File: `path/to/test_file`

- Action: CREATE | MODIFY | DELETE

- Code: <complete test code or changed region with enough context>

- Rationale: <weak, missing, brittle, slow, or duplicate test this fixes>

- Verification: <command the applier should run>

- ```

+ No target-specific additions.

- If no test framework exists, ask before adding one. Do not claim clean without a

- passing check or an explicit skipped-check reason.