ci-speedup · diff
git:20261022.dc3f816 to git:20261022.f974cf9
92 added, 51 removed. Audit B to A.
---
name: ci-speedup
- description: Cuts the wait from push to green by measuring a pipeline's critical path from run timestamps, then splitting, sharding, trimming setup and sharing test module state, with a before/after ledger. Use when asked to "speed up CI", "CI is slow", "why does a PR take 15 minutes", "CI is the bottleneck", or when agents merge faster than checks finish.
- compatibility: Reads run and job timings through the GitHub CLI or a GitHub MCP server; other CI hosts need their equivalent API. Local re-timing needs the repo's toolchain installed.
+ description: 'Speeds up CI and the merge path: finds where changes wait between push and deploy, then fixes the critical path, check placement, merge queue or post-merge checks with auto-revert, flaky tests, and deploy rate. Use when "CI is slow", "speed up CI", "do we need a merge queue", "main keeps going red", "flaky tests block merges", agents open PRs faster than they merge, or building a self-healing merge loop.'
+ compatibility: Reads PR, run, job and deployment history through the GitHub CLI and jq (or a GitHub MCP server); other hosts need their equivalent API. Local re-timing needs the repo's toolchain.
---
# CI Speedup
- Make the wait from push to green shorter, measured, and written down.
+ Find the stage where changes wait between push and deploy, fix that stage with numbers, and write down what changed and when to look again.
- - **IS:** measuring a pipeline's critical path from run timestamps, choosing the levers that shorten it, changing workflow and test configuration, and recording before and after with the trend that says when the next round is due.
- - **IS NOT:** getting a red build green (`pr-babysitter`), writing a pipeline for a package that has none (`scaffold-cli`), deleting low-value tests (`test-audit`), or general repository hygiene.
+ - **IS:** measuring where changes wait from push to deploy; a pipeline's critical path; check placement per lane and the required-check set; merge queue sizing or the post-merge design; main-health detection, attribution and revert; the agentic merge loop that reverts, re-lands and repairs without a human; flake policy; deploy throughput; cost per merged change; the ledger.
+ - **IS NOT:**
+ - one PR's red build, conflict or single rerun (`pr-babysitter`)
+ - package releases (`autoship`)
+ - which checks exist, enforcing a rule, hook wiring, protected paths, or the sandbox rules for an unattended agent job (`codebase-architecture`); this skill owns the merge controller that dispatches those jobs and decides revert, merge and deploy
+ - pruning tests (`test-audit`)
+ - reviewing a workflow diff (`tidy`)
+ - repositories with no pipeline yet (`scaffold-cli`, `scaffold-nextjs`)
+ - how many agents to run, or writing the repository's merge-risk policy (this skill reads that policy; it does not author it)
- ## What Changes the Number
+ ## The Model
- Wall-clock is the slowest chain of dependent jobs, not their sum. The number is push to green for one pull request, median and p90 over recent runs. Around five minutes an agent loop stops waiting on checks and the bottleneck moves to deploy and review. A lever that saves runner minutes off the critical path is a cost saving, and the ledger records it as one.
+ The objective is **X = changes merged and deployed per hour**, held inside three budgets and one hard limit:
- Three ratios decide what to do next:
+ | Symbol | Budget |
+ |---|---|
+ | R | main-red minutes per week |
+ | C | runner-minutes per merged change: PR runs × pushes, queue runs including discarded speculation, the main run, reruns, and per-push review bots |
+ | F | p90 time from push to a verdict the author can act on; an agent's context decays while it waits |
- - **Setup tax.** Checkout, runtime, dependencies, and service boot, per job. 110 seconds of setup around 60 of work is a batching candidate; around 400 it is a sharding candidate. Each shard adds a setup, so cut setup before adding shards.
- - **Import versus test.** When Vitest's import plus environment figures exceed its tests figure, the suite pays to boot the module graph per file. A shared registry or a lighter environment moves that; more shards only spread it.
- - **Gate versus work.** Change detection, matrix computation, and cache saves sit on the chain. Anything they do beyond deciding is paid by every job behind them.
+ Irreversible damage (destructive migrations, infrastructure, leaked secrets) stays at zero whatever it costs X.
+ Symbols used across every file: λ = arrival rate (merges/hour); T = merge-path CI minutes; p = failure rate on the merged tree, flakes included; p_mb = merge-breakage rate given the head was green; B = batch size; D = speculation depth.
+
+ **Finding the constraint.** Pull per-PR timestamps for open, ready, head green, approved, enqueued, merged, main verdict and deployed. Label each stage WAIT or WORK; review and queue time are wait. The constraint is the stage with the most queued minutes per week, and Little's law (WIP = λ × lead time) checks the arithmetic. Exploit it, subordinate the other stages to it, then elevate it, and re-measure after every fix because the constraint moves. Cutting CI duration when CI is not the constraint is a cost lever, not a throughput lever.
+
+ **Pipeline duration.** When the pipeline is the constraint, its wall-clock is the slowest chain of dependent jobs, not their sum. A lever that saves runner minutes off that chain lowers C, not F, and the ledger records it as a cost saving.
+
+ **Lanes and placement.** Every check runs in one lane: author pre-push, PR gate (exact head), merge path (the merged tree, or nothing), post-merge main (coalesced runs; drives auto-revert), pre-deploy (the exact artifact) or nightly (never a merge verdict). Place each check by asking, in order: irreversible if it ships (block before merge, human tier); merge-breakable, meaning its result can change when merged with other PRs (run it on the merged tree; otherwise at the PR head only); deterministic (only these may drive auto-revert); and what it adds to T against what it catches. The classification table and the YAML are in `references/lanes.md`.
+
+ **Queue or no queue.** Start from utilisation u = λ·T/60. Below 0.5, a merge queue with batch 1 to 4 that re-proves only merge-breakable checks gives the strongest main health cheaply. At 0.5 and above, measure p from `merge_group` runs before sizing anything, because flakes count as failures: they shrink the best batch size and make bisection eject innocent PRs. When λ outruns what batching and speculation can carry inside C, the agent-scale default is no global queue: merge on evidence for the exact head, run the full suite post-merge with coalesced runs, auto-revert deterministic failures only and hand each re-land to an agent as a new task, deploy only verified ancestors, and keep a pre-merge guard and a human merge for irreversible classes. Neither "require branches up to date" nor "require green main before merging" is ever the answer: both serialise merges. The batching math, the flip conditions back to a queue, the host settings and the calibration points live in `references/merge-path.md`.
+
## Reference Files
| File | Read when |
|------|-----------|
- | `references/measuring.md` | Step 1: run, job, and step timings; percentiles; the critical path; test-runner duration lines; the baseline table. Step 4: the rules that decide whether the after-run credits a lever |
- | `references/levers.md` | Step 2: the lever catalogue by class, what each needs to be true, expected gain, how each has failed |
- | `references/ledger.md` | Step 5: the ledger, its trend table, and the instruction-file lines that stop the layout regressing |
- | `scripts/ci-timings.sh` | Step 1: one run's jobs, steps and critical path, or `--runs N` for median and p90 per job; `--help` gives the forms |
+ | `references/flow.md` | Step 1, or the user asks where PRs wait: stage definitions, WAIT vs WORK, Little's law, health metrics, cost per merged change, reading `ci-flow.sh` output |
+ | `references/measuring.md` | Pipeline duration is the constraint, or Step 5 is crediting an after-run: run, job and step timings, the three ratios, the critical path, `merge_group` runs, failure and rerun share, queue time, after-run rules |
+ | `references/levers.md` | Choosing duration levers: setup, execution, toolchain and image-build levers, what each needs to be true, and the traps |
+ | `references/lanes.md` | Placing a check; editing `on:`, `if:`, `needs:` or `concurrency:`; changing required checks; a gate that passes while checking nothing |
+ | `references/merge-path.md` | Queue or no queue, queue sizing, a backed-up queue, ejections, conflicts, merge policy |
+ | `references/main-health.md` | Main is red, or designing detection, attribution, auto-revert or re-land |
+ | `references/merge-loop.md` | Designing or building the end-to-end agentic merge system: what code decides versus what agents repair, post-merge batching, the agent task contract, the autonomy ladder, loop health, and choosing a runtime (Actions with Claude, eve, Devin). Read with `main-health.md` and `bots.md` |
+ | `references/bots.md` | Building or changing any bot that writes: revert, repair, re-queue, re-land |
+ | `references/flakes.md` | Intermittent failures, retry or quarantine policy, and before batching or bisecting |
+ | `references/deploy.md` | Deploy is the constraint, or deploys trail merges |
+ | `references/ledger.md` | Step 6: the ledger, flow-metric trend, merge-path decision record, instruction-file lines |
+ | `scripts/ci-flow.sh` | Step 1: per-stage median, p90 and minutes per week for merged PRs with a CONSTRAINT line; `--since`, `--base`, `--workflow`, `--environment`, `--json` |
+ | `scripts/ci-timings.sh` | Step 2, pipeline: one run's jobs, steps and critical path, or `--runs N` for median, p90, failure, rerun and queue-wait per job; `--event merge_group` for queue runs |
+ | `scripts/ci-health.sh` | Step 2, main: red episodes on main with first-bad and last-good SHAs, red minutes per week, and flake candidates |
- `evals/evals.json` holds scenarios and the with-versus-without comparison for changing this skill; it never loads during a task.
+ Every script is read-only, takes `--repo owner/name` (or `GH_REPO`), and prints its forms with `--help`. `evals/evals.json` holds scenarios and the with-versus-without comparison for changing this skill; it never loads during a task.
## Workflow
Copy this checklist to track progress:
```text
CI speedup progress:
- - [ ] Step 1: Baseline (median and p90 over recent runs, per-step table for the longest job, critical path)
- - [ ] Step 2: Choose levers by gain on the critical path, with a predicted chain
- - [ ] Step 3: Change the pipeline; re-time locally what can be re-timed
- - [ ] Step 4: Push, read the branch's own runs as the after-measurement, iterate
- - [ ] Step 5: Ledger with trend, instruction-file lines, and the queue of what is left
+ - [ ] Step 1: Find the constraint (per-stage WAIT and WORK minutes per week)
+ - [ ] Step 2: Measure the constraint stage (pipeline, merge path, main, or deploy)
+ - [ ] Step 3: Decide with numbers (predicted X, R, C; one option; its flip condition)
+ - [ ] Step 4: Change it on a branch; re-time locally; dry-run every bot
+ - [ ] Step 5: After-measurement from the branch's own runs or a bot replay
+ - [ ] Step 6: Ledger, decision record, instruction-file lines, re-measure date
```
- Done means a mergeable pipeline change, a ledger whose after column comes from runs on the branch, and instruction-file lines for the layout and the test defaults. Reading runs, re-timing locally, and pushing to the working branch need no check-in; spending money (paid runners, more parallel jobs on a metered plan) and merging are the user's.
+ ### Step 1: Find the constraint
- ### Step 1: Baseline from timestamps
+ Run `scripts/ci-flow.sh` against the default branch and name the constraint in a sentence: which stage, WAIT or WORK, minutes per week, and how it compares with the next stage. A scoped ask (one suite, one job) skips to Step 2's pipeline branch. With fewer than about 20 merged PRs in the window, reason from the workflow config and the volumes the user states, and label every number a projection.
- Use the run and job API, never memory or the workflow file. `scripts/ci-timings.sh --runs 20 --workflow ci.yml --branch main` gives median and p90 per job; the single-run form on one default-branch run and one pull request gives the steps and the chain. Name the critical path in a sentence: which jobs, in what order, minutes from run start to the last job's end. Pull the slowest test step's log for the runner's own breakdown, and classify every minute on the path as toolchain, gating, setup, or execution before proposing anything.
+ ### Step 2: Measure the constraint stage
- ### Step 2: Choose levers
+ - **Pipeline:** `scripts/ci-timings.sh --runs 20 --workflow ci.yml` for median and p90 per job, then the single-run form for the steps and the chain. Classify every minute on the critical path as toolchain, gating, setup or execution before proposing anything.
+ - **Merge path:** `merge_group` runs give T, p and ejections; arrivals from Step 1 give λ.
+ - **Main:** `scripts/ci-health.sh` gives red episodes, R, and which checks failed on the same SHA they passed on.
+ - **Deploy:** deployment timings against merge times, and how many merges each deploy carries.
- Pick from `references/levers.md` by gain on the chain from Step 1. Write the predicted new chain with its minutes; a plan without a number cannot be checked. Prefer levers the suite itself proves safe (a sharded run passes or does not) over ones that need judgement per file. Runner minutes may rise when wall-clock falls; say by roughly how much, and whether the repository pays for minutes.
+ ### Step 3: Decide with numbers
- ### Step 3: Change the pipeline and re-time locally
+ State the predicted change in X, R and C, one recommended option, and the measurement that would flip it. When wall-clock falls and runner minutes rise, say by roughly how much and whether the repository pays for minutes.
- Edit the workflow and test configuration in one branch. Measure locally whatever can be measured without pushing, with the current and the new configuration. A change that makes a shard fail is a finding, not a reason to skip that shard.
+ ### Step 4: Change it on a branch
- Keep the guards. Change detection, merge-DAG checks, concurrency groups, and deploy gates exist because of a past incident; move them, do not drop them, and keep the comment that cites the incident next to the guard.
+ Edit workflow, test and bot configuration in one branch, and re-time locally whatever can be measured without pushing. A change that makes a shard fail is a finding, not a reason to skip the shard. Dry-run every bot that writes by replaying the red episodes from Step 2 and listing what it would have reverted, repaired or re-queued. Building the agentic merge loop: build the controller from `references/merge-loop.md`, ship it at rung 0 with the kill switch off, and promote only on replay numbers.
- ### Step 4: Push and read the after-measurement
+ Keep the guards. Change detection, concurrency groups, merge checks and deploy gates usually exist because of a past incident; move them, never drop them, and keep the comment that cites the incident next to the guard.
- The branch's own runs are the after-measurement. Read them as in Step 1, put the numbers beside the baseline, and apply the verification rules in `references/measuring.md` before crediting a lever. When predicted and measured disagree, find out why before adding another lever. Repeat until the path stops moving or the remaining levers need a decision the user owns.
+ ### Step 5: Read the after-measurement
- ### Step 5: Ledger and guardrails
+ The branch's own runs are the after-measurement; apply the after-run rules in `references/measuring.md` before crediting a lever. When predicted and measured disagree, find out why before adding another lever. A change that waits on a repository setting the user owns gets its after-measurement on the re-measure date, and the ledger says so. Repeat Steps 2 to 5 until the constraint moves to another stage or the remaining options need a decision the user owns.
- Write the ledger from `references/ledger.md` into the repository's docs: before, changes, after with run ids, a trend row with when the next round is due, and the queue of levers not taken with what each waits on. Add the instruction-file lines for the layout and for how a new test lands in the fast path. Done when the after column comes from real runs, not the prediction.
+ ### Step 6: Ledger and decision record
+ Write the ledger from `references/ledger.md` into the repository's docs: before, changes, after with run ids, the flow-metric trend row, the decision record with its flip conditions, the queue of options not taken with what each waits on, and the re-measure date. Add the instruction-file lines that stop the layout regressing.
+
+ ## Permissions and Done
+
+ Granted without a check-in, because each is read-only, branch-local or reversible: `gh` reads, running the scripts, local re-timing, pushing the working branch, and opening PRs, revert PRs included.
+
+ The user's call, because each spends money or changes what every other change must pass:
+
+ - spending: paid runners, higher queue concurrency or speculation depth on a metered plan
+ - repository settings: rulesets, required checks, enabling the merge queue
+ - turning a bot's kill switch on
+ - merging anything to main
+
+ Done means a mergeable change, after-numbers from real runs or a bot replay, a decision record with its flip conditions, and a ledger row with a re-measure date.
+
## Gotchas
- - `cancel-in-progress: true` on the default branch cancels the superseded run's deploy mid-rollout; cancel on pull requests only. On a pull request the same setting cancels your own measurement: a push while its run is in flight throws that run away. Hold docs-only commits until the measured run completes.
- - A queued default-branch run that gets replaced never runs its checks, so change detection against the previous commit skips work that never passed. Compare against the last commit that finished successfully.
- - Every workspace's tests in one job through a task runner with no remote cache is oversubscription, not caching: twenty test processes on four cores, each with its own workers. A 0:47 suite took 3:04 there. Read the per-workspace duration lines before blaming the tests.
- - A restored dependency cache can be slower than a fresh install (Linear measured restore at four times the install). Time both; a 2.6 GB `node_modules` is not worth trying.
- - Overlapping two IO-bound steps on a hosted runner returns less than their sum: a background service boot cut the post-install wait from 2:11 to 0:34 while the install grew from 1:32 to 2:28.
- - A background boot must resolve its project the way the reading step does. `supabase start --workdir <dir>` booted a default project under another container name and `supabase status` found nothing.
- - Vitest and Playwright shard by file count, so two shards differ by a minute and the same shard varies by a minute between runs. Read median and p90 before adding a shard; splitting the long files rebalances, another shard does not.
- - A job that `needs` a job skipped by an `if` skips too. Where a deploy must run past an optional job, gate on `always()` and the specific results.
- - Matrix jobs default to `fail-fast: true`; one red shard cancels the rest and hides how many are red. Set `fail-fast: false` on test shards.
- - A format check with `--check .` on the whole tree makes docs-only commits ineligible for skipping; only extensions the formatter never matches can be skipped, and only through the `changes` job's job-level `if` (`references/levers.md`, Gating). A workflow-level `paths:` or `paths-ignore:` filter on a workflow with a required check leaves that check pending forever when nothing listed changes.
- - A Docker registry cache pushed to the same tag as the image never hits: the push overwrites the cache manifests and `cache-from` finds a regular image. The cache ref is its own tag.
- - `RUN --mount=type=cache` without an `id` shares one directory between parallel builds of different images and corrupts it under the default `sharing=shared`. Name every mount.
- - Editing the deploy line in the workflow does nothing when it calls a wrapper script that owns the real command. Find where the command runs before re-timing.
- - `turbo run typecheck`, `lerna run`, and `pnpm -r run <script>` silently skip workspaces that lack the script; a green root command proves nothing about a workspace that never ran. Validate per workspace.
+ - Sharding a pipeline whose PRs go green in minutes and then wait hours for review or merge changes nothing in X. Measure open to deployed with `ci-flow.sh` before touching the pipeline.
+ - After a fix the constraint moves: re-timing only the stage just fixed reports a win while X stays flat. Re-run `ci-flow.sh` after every change.
+ - `cancel-in-progress: true` on main cancels a superseded run mid-rollout and leaves the merged commits with no verdict; on a PR it throws away your own measurement run when a later push lands. Cancel superseded runs on PRs only, and hold pushes until the measured run completes.
+ - `turbo run`, `pnpm -r run` and `lerna run` skip workspaces that lack the script, so a green root command proves nothing about them. Check the per-workspace output.
+ - A workflow step that calls a wrapper script: editing the workflow line changes nothing, because the wrapper owns the real command. Find where the command runs before re-timing.
## Related Skills
- - `pr-babysitter` for a run that is red rather than slow, and for watching the pull request this work opens.
- - `pr-creator` for the pull request; the ledger's before and after tables belong in its body.
- - `agents-md` when the instruction-file lines from Step 5 grow into a wider AGENTS.md pass.
- - `scaffold-cli` and `scaffold-nextjs` when the repository has no pipeline yet; this skill starts from timestamps, which a new repository lacks.
+ - `pr-babysitter` for one PR that is red, conflicted or needs a rerun, and for watching the PR this work opens.
+ - `autoship` for package releases and their CI.
+ - `codebase-architecture` for which checks should exist, enforcing a rule, hook wiring, protected paths, and the sandbox an unattended agent job runs in.
+ - `test-audit` when the suite is slow because it holds tests that prove nothing; pruning is its job, timing is this one's.
+ - `tidy` for reviewing the workflow diff this skill produces.
+ - `scaffold-cli` and `scaffold-nextjs` when the repository has no pipeline yet; this skill starts from timestamps a new repository lacks.