v1.0 to v1.0

---

name: coverage-analysis

description: >-

Measure and improve test coverage meaningfully. Covers Istanbul/V8/c8/coverage.py

configuration, coverage gap analysis methodology, coverage-as-ratchet in CI (never

let it decrease), PR coverage diff checks, and distinguishing meaningful from vanity

coverage. Use when: "code coverage," "coverage gap," "Istanbul," "coverage threshold,"

"coverage report," "branch coverage."

Related: unit-testing, ci-cd-integration, qa-metrics, ai-qa-review.

license: MIT

metadata:

author: kindlmann

version: "1.0"

category: metrics

---

<objective>

Measure coverage to find gaps, not to chase numbers.

</objective>

---

## Discovery Questions

1. **What coverage tooling is configured?** Check for `jest.config.*` (coverageProvider), `vitest.config.*` (coverage), `.nycrc`, `c8` in scripts, or `[tool.coverage]` in `pyproject.toml`.

2. **What is the current coverage level?** Run the existing coverage command and note line, branch, and function percentages. This is the baseline for the ratchet.

3. **Is coverage gated in CI?** Check GitHub Actions or GitLab CI for `--coverage`, `coverageThreshold`, `fail_under`, or `--cov-fail-under` flags.

4. **What is the target, and who set it?** A target without rationale ("the VP said 80%") leads to gaming. Targets should reflect risk tolerance and codebase maturity.

5. **Check `.agents/qa-project-context.md` first.** Respect existing coverage conventions and thresholds.

---

## Core Principles

**1. Coverage measures breadth, not depth.** A line of code being executed does not mean it is tested correctly. `expect(true).toBe(true)` executes the function but asserts nothing meaningful. Coverage tells you what code ran, not whether the tests would catch a bug.

**2. Branch coverage matters more than line coverage.** A ternary `condition ? a : b` on one line counts as "covered" in line coverage even if only one branch executes. Branch coverage exposes untested paths.

```typescript

// Line coverage: 100% (the line executed)

// Branch coverage: 50% (only the 'true' branch ran)

function discount(price: number, isPremium: boolean): number {

return isPremium ? price * 0.8 : price;

}

// Test only covers isPremium = true

expect(discount(100, true)).toBe(80);

// Missing: expect(discount(100, false)).toBe(100);

```

**3. Ratchet pattern: never decrease, only increase.** Record the current coverage as the minimum threshold. Every PR must meet or exceed it. Coverage goes up over time without forcing artificial targets.

**4. Focus on gaps, not numbers.** A project at 85% coverage is not "better" than one at 75%. What matters is whether the untested 15% or 25% contains critical business logic. Analyze gaps by risk, not by percentage.

**5. New code has a higher bar than legacy code.** Require 90%+ coverage on new code in PRs even if the overall project is at 65%. This prevents coverage decay without demanding a rewrite of legacy code.

---

## Coverage Tools

> **Node baseline:** `c8` v11 and `nyc` v18 (both released 2026-02-22) require **Node 20 || >= 22**. If your project still runs on Node 18, pin `c8@^10` / `nyc@^17` until you can upgrade. New projects should standardize on Node 20+ and the current majors.

### V8 / c8 (Node.js Built-in)

V8's built-in code coverage is faster than Istanbul because it does not instrument source code. Use `c8` as the CLI wrapper.

```bash

npm i -D c8@^11 # Node 20+ baseline

```

```json

// package.json

{

"scripts": {

"test": "vitest run",

"test:coverage": "vitest run --coverage"

}

}

```

```typescript

// vitest.config.ts

import { defineConfig } from "vitest/config";

export default defineConfig({

test: {

coverage: {

provider: "v8",

reporter: ["text", "html", "lcov", "json-summary"],

reportsDirectory: "./coverage",

include: ["src/**/*.ts"],

exclude: [

"src/**/*.test.ts",

"src/**/*.spec.ts",

"src/**/*.d.ts",

"src/**/index.ts", // Barrel exports

"src/**/types.ts", // Type-only files

"src/**/*.stories.ts", // Storybook

"src/generated/**", // Generated code

],

thresholds: {

lines: 80,

branches: 80,

functions: 80,

statements: 80,

},

},

},

});

```

### Istanbul / nyc (JavaScript/TypeScript)

Istanbul instruments source code for coverage tracking. Slower than V8 but more widely compatible.

```bash

npm i -D nyc@^18 # Node 20+ baseline

```

```json

// .nycrc.json

{

"all": true,

"include": ["src/**/*.ts"],

"exclude": [

"src/**/*.test.ts",

"src/**/*.spec.ts",

"src/**/*.d.ts",

"src/**/index.ts",

"src/generated/**"

],

"reporter": ["text", "html", "lcov", "json-summary"],

"report-dir": "./coverage",

"check-coverage": true,

"branches": 80,

"lines": 80,

"functions": 80,

"statements": 80,

"watermarks": {

"lines": [70, 90],

"functions": [70, 90],

"branches": [70, 90],

"statements": [70, 90]

}

}

```

```json

// package.json (with Jest)

{

"scripts": {

"test:coverage": "jest --coverage"

},

"jest": {

"coverageProvider": "v8",

"collectCoverageFrom": [

"src/**/*.ts",

"!src/**/*.{test,spec,d}.ts",

"!src/**/index.ts",

"!src/generated/**"

],

"coverageThreshold": {

"global": {

"branches": 80,

"functions": 80,

"lines": 80,

"statements": 80

}

}

}

}

```

### coverage.py (Python)

```bash

pip install pytest-cov # current coverage.py 7.13.x

```

> coverage.py 7.13.0 added `.coveragerc.toml` as a TOML-first standalone config — prefer it for new Python projects when you want config separated from `pyproject.toml`. 7.12.0 split statements vs branches totals in HTML and JSON reports.

```toml

# pyproject.toml (or .coveragerc.toml in 7.13+)

[tool.coverage.run]

source = ["src"]

branch = true

omit = [

"src/**/test_*.py",

"src/**/conftest.py",

"src/**/__init__.py",

"src/generated/*",

]

[tool.coverage.report]

fail_under = 80

show_missing = true

skip_covered = true

precision = 1

exclude_lines = [

"pragma: no cover",

"if TYPE_CHECKING:",

"if __name__ == .__main__.",

"raise NotImplementedError",

"@overload",

"\\.\\.\\.", # Ellipsis in abstract methods

]

[tool.coverage.html]

directory = "coverage/html"

[tool.coverage.xml]

output = "coverage/coverage.xml"

```

```bash

# Run tests with coverage

pytest --cov=src --cov-report=term-missing --cov-report=html --cov-report=xml

# Fail if coverage drops below threshold

pytest --cov=src --cov-fail-under=80

```

### Coverage Report Types

| Reporter | Output | Use Case |

|----------|--------|----------|

| `text` | Terminal table | Quick local check |

| `html` | Interactive HTML | Detailed local analysis, clicking through files |

| `lcov` | `lcov.info` file | SonarQube, Codecov, Coveralls integration |

| `json-summary` | `coverage-summary.json` | CI scripts, PR comments, dashboard metrics |

| `cobertura` | `cobertura-coverage.xml` | GitLab CI coverage visualization |

---

## Gap Analysis

### Identify Untested Code Paths

Coverage reports show which lines and branches are not executed. But not all gaps are equal. Prioritize by risk.

**Step 1: Generate the coverage report.**

```bash

npm run test:coverage

# Open coverage/index.html in a browser

```

**Step 2: Sort files by uncovered lines.** Parse `coverage-summary.json`, sort files by `(total - covered)` descending, and focus on the top 20. A script that reads the JSON summary and outputs a table of file, line%, branch%, and uncovered count makes this repeatable.

**Step 3: Map gaps to risk areas.**

| Gap Location | Risk Level | Action |

|-------------|-----------|--------|

| Payment processing | Critical | Write tests immediately |

| Auth/permissions | Critical | Write tests immediately |

| Data validation | High | Add to next sprint |

| Error handling paths | High | Add to next sprint |

| Utility functions | Medium | Cover when modifying |

| UI formatting | Low | Skip unless regression-prone |

| Generated code | None | Exclude from coverage |

### Coverage Diff on PRs

Show coverage change per PR so reviewers see the impact of each change.

```yaml

# GitHub Actions: coverage diff comment

- name: Run tests with coverage

run: npm run test:coverage

- name: Coverage Report

uses: davelosert/vitest-coverage-report-action@v2

if: github.event_name == 'pull_request'

with:

json-summary-path: coverage/coverage-summary.json

json-final-path: coverage/coverage-final.json

vite-config-path: vitest.config.ts

```

Alternative: use `marocchino/sticky-pull-request-comment@v2` with a script that reads `coverage-summary.json`, filters to changed files (via `git diff --name-only origin/main...HEAD`), and renders a markdown table showing per-file line and branch coverage.

---

## Coverage as CI Gate

### Threshold Configuration

**Global threshold (minimum for the entire project):**

```typescript

// vitest.config.ts

coverage: {

thresholds: {

lines: 80,

branches: 80,

functions: 80,

statements: 80,

},

}

```

**Per-directory thresholds (stricter for critical code):**

```typescript

// vitest.config.ts

coverage: {

thresholds: {

// Global baseline

lines: 75,

branches: 75,

// Stricter for critical paths

"src/payments/**": { lines: 95, branches: 90 },

"src/auth/**": { lines: 90, branches: 85 },

"src/utils/**": { lines: 80, branches: 80 },

},

}

```

**Jest per-file thresholds:**

```javascript

// jest.config.js

module.exports = {

coverageThreshold: {

global: { branches: 75, functions: 75, lines: 75, statements: 75 },

"./src/payments/": { branches: 90, functions: 90, lines: 95, statements: 95 },

"./src/auth/": { branches: 85, functions: 85, lines: 90, statements: 90 },

},

};

```

### Ratchet Pattern

Never let coverage decrease. Record the current level as the minimum and update it upward when coverage improves.

```typescript

// scripts/coverage-ratchet.ts

import fs from "fs";

import coverageSummary from "../coverage/coverage-summary.json";

const RATCHET_FILE = ".coverage-ratchet.json";

interface Ratchet {

lines: number;

branches: number;

functions: number;

statements: number;

updatedAt: string;

}

// Read current ratchet or initialize

let ratchet: Ratchet;

try {

ratchet = JSON.parse(fs.readFileSync(RATCHET_FILE, "utf-8"));

} catch {

ratchet = { lines: 0, branches: 0, functions: 0, statements: 0, updatedAt: "" };

}

const current = (coverageSummary as any).total;

const metrics = ["lines", "branches", "functions", "statements"] as const;

let failed = false;

for (const metric of metrics) {

const currentPct = current[metric].pct;

const ratchetPct = ratchet[metric];

if (currentPct < ratchetPct) {

console.error(

`FAIL: ${metric} coverage dropped from ${ratchetPct}% to ${currentPct}% (delta: ${(currentPct - ratchetPct).toFixed(1)}%)`

);

failed = true;

} else if (currentPct > ratchetPct) {

console.log(`IMPROVED: ${metric} coverage increased from ${ratchetPct}% to ${currentPct}%`);

ratchet[metric] = Math.floor(currentPct); // Floor to avoid ratcheting on decimals

} else {

console.log(`STABLE: ${metric} coverage at ${currentPct}%`);

}

}

if (failed) {

console.error("\nCoverage ratchet failed. Coverage must not decrease.");

console.error("If this is intentional (e.g., removing dead code), update .coverage-ratchet.json manually.");

process.exit(1);

}

// Update ratchet file

ratchet.updatedAt = new Date().toISOString();

fs.writeFileSync(RATCHET_FILE, JSON.stringify(ratchet, null, 2) + "\n");

console.log(`\nRatchet updated: ${JSON.stringify(ratchet)}`);

```

Commit `.coverage-ratchet.json` to the repo (e.g., `{ "lines": 82, "branches": 78, ... }`). In CI, run the ratchet script after tests. On main branch merges, auto-commit the updated ratchet file if coverage improved.

### PR Diff Coverage Gate

Require that new code in a PR meets a higher threshold (e.g., 90%) than the project baseline. In CI, use `git diff --numstat origin/main...HEAD` to identify changed files, then check their coverage from `coverage-summary.json`. Fail the pipeline if the average coverage of changed files falls below the threshold. This prevents coverage decay without demanding a rewrite of legacy code.

**Hosted alternatives:** **Codecov**, **Coveralls**, and **Trunk Coverage** all ship first-class differential PR coverage with merge-blocking gates and inline annotations. Most teams prefer these over hand-rolled diff scripts — pick one if you don't already have a coverage host. Codecov + GitHub: `codecov/codecov-action@v5` reads `lcov.info` and posts a PR comment with diff coverage automatically.

---

## Mutation Testing

Mutation testing measures *assertion quality*, not just code execution. With Stryker JS v9.6+ and Vitest 4.1+, the cost is low enough to make mutation score realistic on PR-changed files.

### Stryker (JS/TS)

```json

// stryker.config.json

{

"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",

"testRunner": "vitest",

"coverageAnalysis": "perTest",

"incremental": true,

"incrementalFile": ".stryker-tmp/incremental.json",

"mutate": ["src/**/*.ts"],

"thresholds": { "high": 80, "low": 60, "break": 50 }

}

```

Run on PR-changed files only with `--mutate '$(git diff --name-only origin/main...HEAD | grep "src/.*\.ts")'`. Pair `incremental: true` with the JSON cache so consecutive runs only re-mutate touched files.

### mutmut (Python)

```bash

mutmut run --paths-to-mutate=src/

mutmut results

```

mutmut 3.x is a from-scratch rewrite (still active 2026); pin via `pip install mutmut==3.*`.

### Targeting

Mutation testing is expensive on whole codebases. Restrict to:

- Pure business logic (validators, calculators, transformers)

- Critical paths (payment, auth, data integrity)

- Code with low confidence in test quality (look at branch-coverage > 90% but few assertion variants)

Skip on UI rendering code, glue code, and generated code.

### Reading the score

A mutation score of 80% means 80% of mutated bugs were caught by your tests. Lower than your coverage % is normal — many mutations land in untested branches the coverage report already flagged. The interesting gap is **high coverage + low mutation score**: code is executed but assertions don't constrain it.

---

## Meaningful vs Vanity Coverage

### Why 100% Coverage Is Usually Wrong

100% coverage requires testing every branch of every line, including:

- Error handling for impossible states

- Default cases in exhaustive switches

- Framework lifecycle methods that are never called directly

- Defensive checks against corrupted data

Tests written to hit 100% are often trivial, brittle, and do not catch real bugs.

### Diminishing Returns

| Coverage Range | Value | Effort |

|---------------|-------|--------|

| 0% to 60% | High -- covers main paths, catches obvious regressions | Low |

| 60% to 80% | Medium -- covers error paths, edge cases | Medium |

| 80% to 90% | Lower -- covers unusual combinations, defensive code | High |

| 90% to 100% | Minimal -- covers unreachable code, framework internals | Very high |

The sweet spot is 75-85% for most projects. Critical paths (payments, auth) should aim higher (90%+).

### What NOT to Cover

Exclude these from coverage calculations. They inflate the denominator without adding value.

```typescript

// vitest.config.ts or jest.config.js -- exclude patterns

exclude: [

"**/*.d.ts", // Type definitions

"**/index.ts", // Barrel exports (re-exports only)

"**/*.stories.{ts,tsx}", // Storybook stories

"**/generated/**", // Auto-generated code (GraphQL, Prisma)

"**/migrations/**", // Database migrations

"**/__mocks__/**", // Test mocks

"**/types/**", // Type-only modules

]

```

### Quality Indicators Beyond Percentage

Coverage percentage alone is insufficient. Combine it with:

| Indicator | What It Measures | How to Get It |

|-----------|-----------------|---------------|

| **Mutation score** | Would tests catch a real bug? | Stryker / mutmut |

| **Branch coverage** | Are all conditional paths tested? | V8/Istanbul with branch reporting |

| **Critical path coverage** | Are payment/auth/data flows fully covered? | Per-directory thresholds |

| **Defect escape rate** | Do production bugs occur in tested code? | Post-incident analysis |

| **Coverage delta** | Is coverage improving or declining? | Ratchet pattern tracking |

---

## Anti-Patterns

**Treating coverage as proof of quality.** "We have 90% coverage so we are well-tested" is a dangerous statement. Coverage says code executed, not that behavior was verified. A test with no assertions contributes to coverage but catches zero bugs.

**Excluding files to inflate numbers.** Adding files to the exclude list because they are hard to test (e.g., error handlers, integration modules) hides the most important gaps. Only exclude genuinely untestable code: generated files, type definitions, and barrel exports.

**Writing trivial tests to hit targets.** Tests like `it("should exist", () => { expect(MyClass).toBeDefined(); })` add coverage without value. Every test should verify a behavior that, if broken, would affect users.

**Global threshold without per-module analysis.** An 80% global threshold can pass even if the payment module is at 30% coverage -- as long as utility functions inflate the average. Use per-directory thresholds for critical paths.

**Coverage threshold set once, never adjusted.** The ratchet should increase over time. A team that has been at 78% for 6 months is not improving. Review the ratchet quarterly and investigate why it stagnates.

**Ignoring branch coverage.** Line coverage reports 100% on `const result = condition ? a : b` even if only one branch runs. Always report and gate on branch coverage alongside line coverage. Branch coverage is the more honest metric.

**Coverage in E2E tests only.** E2E tests execute many lines but cover broad, shallow paths. A single E2E test might touch 60% of the codebase without testing any edge case. Unit tests provide targeted, deep coverage of logic branches. Measure coverage from unit and integration tests separately from E2E.

---

## Done When

- Coverage is instrumented and running automatically in CI on every push (no manual steps to generate the report).

- Coverage threshold is enforced in CI configuration (build fails when line or branch coverage drops below the defined minimum).

- Coverage report is published as a CI artifact (HTML report and `json-summary`) and linked from PR comments showing per-PR coverage delta.

- Meaningful vs. vanity coverage distinction is documented for the codebase: excluded paths are listed and justified (generated code, barrel exports, type definitions), and per-directory thresholds are set higher for critical paths such as payments and auth.

- Coverage-as-ratchet is configured so the threshold only goes up: `.coverage-ratchet.json` is committed, the ratchet script runs in CI, and the build fails if coverage regresses from the recorded baseline.

## Related Skills

- **ci-cd-integration** -- Pipeline configuration for coverage gates, artifact storage, and PR comments.