awesome-regression-sweep git:20260912.e5449ebA

Immutable. This exact content never changes and is served at /api/v1/blob/d6f1d44593e4caa1.

---
name: awesome-regression-sweep
description: "Read-only regression sweep over a codebase and its live surfaces, answering in deltas against a recorded baseline: a layered aspect pass (typecheck, lint, generated-artifact drift, unit and integration suites, black-box wire contract, cross-implementation parity, downstream consumer build) plus nine rotating deep angles from shuffled test order to deployed-vs-committed drift. Never deploys, never writes, never calls a mutating route. Use when asked to verify everything before or after a deploy, check for regressions, or 'проверь что ничего не сломалось'. Do not use to design tests (awesome-test-writing), diagnose one failure (awesome-bug-fix), or profile latency (awesome-performance-audit)."
license: MIT
metadata:
  author: Khasky
  tags: ["verification", "regression", "release-readiness", "wire-contract", "baseline", "read-only"]
  documentation: "https://github.com/khasky/awesome-agent-skills/tree/main/skills/awesome-regression-sweep"
---

# Regression Sweep

One question — *did anything break* — answered in layers, because the layers fail
differently. A green unit suite says nothing about a byte format two independent
implementations have to agree on; a passing suite says nothing about whether the
code that is deployed is the code you tested; and a green working tree says nothing
about the scheduled job that stopped running last night.

Everything here is read-only. No deploy, no database write, no mutating admin
call, no destructive suite against production. When a check would need one, say so
and stop rather than doing it.

Reference files (load the one the run needs):
- [`references/live-contract-checks.md`](references/live-contract-checks.md) — the black-box checklist for a public read surface, the invariants of append-only and derived data, and the golden-vector method for cross-implementation parity.
- [`references/deployment-and-infrastructure.md`](references/deployment-and-infrastructure.md) — proving the deployed code is the tested code, and the infrastructure layer that fails with no code change at all.

Running the sweep is yours to arrange: the aspect table below is the contract, and whatever you drive it with — a
shell loop, a small program you write for this repository, or the commands one at a time — is an implementation
detail that never leaves your side of the boundary. What the run has to produce is fixed: one line per aspect, its
delta against the stored baseline, and a verdict.

- Any stack: the aspects are the project's own commands, whatever the language (`cargo test`, `go vet`, `pytest`,
  `mvn verify`, `dotnet build`). Keep the baseline in a plain text file beside the run so the next sweep has something
  to subtract from — the table is the contract, the way it is driven is not.

## Scope and prerequisites

State what is in scope before the first command: which repos, which environment the
live checks point at, and what is unavailable.

Everything optional degrades to a documented SKIP, never a failure. A laptop
with no local server running must not read as a regression. A `SKIP` is reported,
never silently dropped — the reader has to know which layer was not exercised. The
inverse holds too: a *required* working copy that is absent is a failure, not a
skip, or the pass reports green over a layer nobody ran.

Detect the stack before assuming commands: the manifest and its scripts
(`package.json`, `Makefile`, `justfile`, `pyproject.toml`, `go.mod`, `Cargo.toml`,
`build.gradle`) name the real typecheck, test, lint, and build entry points. Use
those, never a guessed equivalent.

## Phase 1 — the aspect sweep

Every aspect is read-only: it runs what the project already defines and never deploys, writes or mutates. Four rules
hold the sweep together, and each exists because a sweep without it reported green over something nobody ran:

- A prerequisite that is absent makes the aspect a SKIP, reported with its reason, never a failure and never dropped
  silently. A required working copy that is absent is a failure instead.
- An exit code is not enough. Wrappers, shims and some suites print a failure and still exit 0, so an aspect whose
  output carries a result line is judged on that line as well as on the code.
- Every aspect has a ceiling on how long it may run, so one hung command cannot swallow the sweep.
- A count is not a result. Each aspect is reported against the baseline from the previous pass: the same number, or
  the delta, plus what moved. Update the baseline only from a pass you are willing to call the new reference.

Nine aspects, one line each, then a verdict. Include the ones that exist:

| Aspect | What a failure means |
| --- | --- |
| typecheck / compile | a contract broke at a type boundary |
| lint / format | a rule the repo enforces, or a formatter drift |
| generated-artifact drift | a committed artifact (API schema, client, migration, lockfile, docs table) no longer matches its generator |
| unit suite | in-process behaviour changed |
| integration / selftests | a component's own invariants broke |
| live wire contract | the published contract changed on the wire |
| cross-implementation parity | two independent implementations disagree on bytes |
| end-to-end audit | the system does not verify end to end |
| downstream consumer build | a consumer of the contract no longer builds |

Baseline discipline. The first pass writes `baseline.json`; every later pass
compares against it and prints a `DELTAS` block, or `no deltas`. "846 passed" is not
a result; "846 passed, same as the baseline" is. Only a line the aspect declares as
its `tally` is compared — a bare last line carries timings and would report a delta
every pass. Re-baseline deliberately with `--update-baseline`, never to make a
red pass look green.

Four config options that decide whether the sweep can be trusted:

- `expect` — a pattern the output must contain. Exit codes lie: wrapper
  scripts, shell shims, and some suites report `0` while printing a failure, and a
  filtered summary has reported "clean" while the underlying tool failed. When an
  aspect prints its own verdict, assert on that verdict; and when a wrapper
  compresses output, run the tool's binary directly rather than the wrapper.
- `timeoutMs` — a ceiling per aspect. A hung live check otherwise stalls the pass
  forever, and a sweep that never finishes verified nothing.
- `failLines` — the pattern that marks a failing case, so a red aspect prints the
  failing names instead of a blind tail.
- `env` — derive environment parameters from the application's own config
  rather than re-typing them in the harness. A key, an endpoint, or a limit copied
  into the sweep drifts from the code it is supposed to check, and then the harness
  is what is wrong.

Running it repeatedly is the point. One pass proves the code compiles. Several
passes spread over time catch flakes, state leaking between tests, and drift caused
by things outside the repo — a scheduled job that ran, a machine that slept, someone
else's deploy. For a long watch, schedule the sweep and rotate the angles below
through the iterations.

Skip ≠ fail. Suites that drive live third-party surfaces turn anti-bot walls,
login walls, and missing credentials into skips, and the skip set varies run to
run. Attribute every non-green line to *code*, *environment*, or *harness* before
reporting it.

Done when: every aspect in the baseline has been compared, and each
non-green line is attributed to code, environment or harness.

## Parallelizing the sweep — with care

Split the aspects into two lanes by resource contention, not just independence
— this is the one skill where careless fan-out manufactures the very failures it
hunts.

- Safe to fan out concurrently: the static/read-only aspects (typecheck, lint,
  generated-artifact drift, docs-vs-code constants — angle 7) and the black-box
  HTTP probes, one agent per endpoint or per malformed shape.
  They share no port, database, or browser.
- Must stay serial: the unit / integration / e2e suites and any live browser
  suite. Concurrent runs fight over ports, test databases, and browsers and
  manufacture the exact flakes angle 1 exists to catch — so run heavy suites
  one at a time, and angle 1 (order-dependent flakes) always runs alone.

The barrier is the verdict: one agent merges all deltas against the single
`baseline.json` and assigns SHIP/FIX/BLOCK — a sub-agent never re-baselines or
emits a verdict. Resource preflight (before fan-out): cap concurrency at
`min((cores−1)×0.75, free_gb×0.7/per_agent, 6)`, `per_agent` ≈ 0.7 GB for
the static/HTTP agents; go serial if CPU load > 85% or free RAM <
2×per_agent; recompute before each wave; where the runtime caps sub-agent
concurrency itself, defer to it.

## Phase 2 — the nine angles

The sweep is the same every time; these are not. Rotate one per iteration so a long
watch keeps producing new information instead of the same green line.

1. Order-dependent flakes. Run the unit suite twice back to back, then shuffled
   under two different seeds. Prove the flag took effect — compare the first
   files reported under each seed; a green run with a silently ignored flag has
   proven nothing. This matters wherever a cache, a module-level singleton, or a
   temp directory is shared across a worker pool: an entry written by one file
   answers another file's request.
2. The same data by its other path. When a system publishes the same data twice
   — an API and a mirror, a CDN and an export, a database and a message stream —
   verify from the second path. It cross-checks both publication paths against one
   source of truth, and usually unlocks checks the first path cannot do.
3. The downstream consumer. Typecheck and test the client, SDK, or CLI that has
   to stay in lockstep with the contract. Name the one test that pins the shared
   derivation; that is the check that silently splits data when it drifts.
4. Coverage ratchet. Run coverage against the floors the repo already sets. A
   ratchet, not a target: it fires when a change *removes* coverage. Never lower a
   floor to make a run pass.
5. Deploy dry-run. Whatever the platform's `--dry-run` / `plan` / `validate`
   is, for every environment. It confirms each binding, secret, and variable
   resolves in all of them, and that the artifact is the same size from each — a
   configuration that only assembles in one environment is a deploy-time surprise
   otherwise.
6. One number by two independent paths. A cached counter versus a fold from
   zero, a dashboard total versus a `SELECT count(*)`, a reported balance versus a
   replay of the ledger. The only check where a wrong number cannot hide.
7. Docs versus code. Every constant a doc states must be greppable in the
   source — limits, timeouts, retry counts, enum vocabularies, defaults. Then grep
   for the phrasings a recent change made false. Highest hit rate of the nine; for
   the full public-copy pass, call the Skill tool with "awesome-claims-audit".
8. Deployed versus committed. `references/deployment-and-infrastructure.md`.
9. Infrastructure and scheduled jobs. Same file.

Done when: every angle picked for this round has been run and its delta
recorded against the single baseline.

## Phase 3 — the invariants checklist

Run the checklist in `references/live-contract-checks.md` against whatever the
system exposes: the public read surface, the durable data behind it, and any format
two implementations serialize independently. Each item there is cheap to re-check
and expensive to discover in production.

Probe the live surface with plain read requests — GET, HEAD and OPTIONS only, nothing that writes. Against a
stable public endpoint, check what no in-process test can see: that two spellings of the same query land on one cache
key, that a malformed request is rejected before any work is done, that the CORS answer is the one the endpoint
intends for an unlisted origin and for a route that does not exist, that an ETag round-trips into a 304, that HEAD
agrees with GET on headers, that an open-ended view is never served as immutable, and that the security headers the
service promises are actually on the response.

Send the malformed request once per shape rather than once per endpoint: non-numeric, reversed range, zero or
negative, oversize range, too many items, an over-long value, and a value missing its separator each take a different
branch through the validator, and the branch that forgets to reject is the one that reaches the database before
validation finishes. Where a path separator would be rewritten by the shell you are in, quote it or escape the
rewriting — a probe that silently asked for the wrong path proves nothing.

Done when: every invariant has been checked or marked NOT ASSESSED with
its reason.

## What not to flag

- A skip caused by the environment. No local server, no credential, an
  anti-bot wall on a third-party page. Report it as a skip with the reason.
- A flake called a regression before it is isolated. But the reverse is not
  allowed either: a test that passes only on re-run is a defect — report the flake,
  never silently retry until green.
- A harness bug called a product bug. Both happen and both look identical at
  first. Two of this sweep's historical failures were harness bugs; conflating them
  wastes the run. Diagnose before the next iteration.
- An absolute tally. A number with no baseline is not a finding.
- Style a formatter owns, when the formatter itself is green.
- A latency number without a baseline — that is `awesome-performance-audit`.
- A known-broken layer re-reported every pass. Carry it as a standing gap.

## Output

Deltas from the baseline, not a re-listing of what passed.

```text
Regression Sweep — <scope> — pass N — <date> — BASE=<environment>
Verdict: SHIP | FIX | BLOCK

Deltas vs baseline:
  <aspect> — <what changed> — code | environment | harness — <evidence>
  unit suite — 846 → 844 passed, 2 failed — code — <test name>, <file:line>

Unchanged: <aspects matching the baseline, one line total>
Skipped:   <aspect> — <why the prerequisite was absent>
Angle this pass: <which of the nine, and what it found>
Not verified: <standing gaps, and what it would take to close each>
```

- SHIP — no delta, or only deltas explained by the environment; the layers
  that matter for this change were exercised.
- FIX — a real regression with a named owner and a reproduction, or a
  deployed-versus-committed mismatch.
- BLOCK — a broken invariant in durable or published data, a wire contract that
  changed under existing consumers, or a state where the sweep cannot tell whether
  the deployed code is the tested code.
- Severity per finding — `Critical / High / Medium / Low`. Critical: durable
  data or a published contract is already wrong. `Informational` is not used.
- Prove a claim before making it. A failure gets diagnosed, not re-run until
  green.
- Say what was not verified and why. A boundary only covered by a stubbed unit
  test, a code path with no production traffic behind it, a suite that refuses to
  run against production by design — each is a standing gap, not a pass.

## Hygiene — own what you start

A sweep starts servers, browsers, tails, and background jobs. Close them.

- A stopped background task is not a stopped process. Dev servers spawn
  children that survive the parent; browser suites leave a browser alive after a
  crashed run.
- Never kill by name. Several agent sessions and editors run on one machine;
  `node`, `pwsh`, and `chrome` are not yours to kill wholesale. Walk the parent
  chain, confirm the process descends from your own command, and stop only that
  subtree.

```powershell
$p = <pid>; while ($p) { $x = Get-CimInstance Win32_Process -Filter "ProcessId=$p"; if (-not $x) { break }; "$($x.Name)($($x.ProcessId)) :: $($x.CommandLine)"; $p = $x.ParentProcessId }
```

```bash
pid=<pid>; while [ -n "$pid" ] && [ "$pid" != 0 ]; do ps -o pid=,ppid=,comm=,args= -p "$pid" || break; pid=$(ps -o ppid= -p "$pid" | tr -d ' '); done
```

- Confirm the port is closed after stopping a server, and stop every tail and
  monitor the run armed.
- Disk counts too. Browser profiles, downloaded fixtures, and coverage output
  grow without bound. Prune the ignored caches when no run is active.