ci-fix · diff
git:20261002.65d1cd0 to git:20261002.6fceb2e
45 added, 35 removed. Audit A to A.
---
name: ci-fix
description: Debug unsuccessful CI on the default branch. Use when CI fails or is cancelled on main.
argument-hint: "[run-id and context]"
metadata:
internal: true
---
# Fix CI on Default Branch
CI ended unsuccessfully on the default branch. Diagnose the result, fix any
durable cause, and create a PR when code or configuration needs to change.
**Run:** $ARGUMENTS
## Workflow
### 0. Load required skills
- Load `/tend-ci-runner:run-tend` first, including its repo-specific overlay.
+ - Load `/tend-ci-runner:ground-claims` before reading and matching the current failure.
- Load `/tend-ci-runner:fix-a-bug` before attempting a fix.
- Load `/tend-ci-runner:post-to-github` before composing a comment, issue, or PR body.
- Load `/tend-ci-runner:open-pr` before opening an issue or PR.
- Load `/tend-ci-runner:push-commits` before pushing a fix and `/tend-ci-runner:monitor-ci` after it.
- ### 1. Check for existing fixes
+ ### 1. Read the current failure
+ 1. Read the run's conclusion and jobs: `gh run view <run-id> --json conclusion,jobs,url`
+ 2. For a failure, read its logs: `gh run view <run-id> --log-failed`. If the output is truncated or only lists failed tests, save it and read the deciding diagnostic before matching prior work.
+ 3. For a cancellation, use the job and step conclusions from the first command. Its log
+ archive may not exist; a failed job inside the cancelled run is the signal
+ to diagnose, not a log fetch.
+
+ Establish the current failure per **Claims of recurrence** in
+ `/tend-ci-runner:ground-claims` before reading earlier diagnoses.
+
+ A cancellation takes this same diagnostic path; do not presume it is transient
+ or ignore it. First establish why it stopped. If concurrency replacement or a
+ deliberate cancellation stopped a run that had no failure, exit silently — do
+ not use the transient-tracker path below. Otherwise diagnose the completed
+ steps and current branch normally.
+
+ ### 2. Check for existing fixes
+
Check whether any PR already addresses the same failure — a prior bot attempt, a prior bot fix a maintainer rejected, or a maintainer's in-flight fix under any branch name. List the open ones; search the closed ones, since a recency listing drops an older rejection past any `--limit`:
```bash
gh pr list --state open --limit 200 --json number,title,state
# <shape>: the failing test or job name, or a distinctive phrase from the error;
# search matches titles and bodies, so run one per term
gh pr list --state closed --search "<shape>" --limit 100 --json number,title,state
```
- Match by **failure shape** — the diagnostic snippet in the bot's PR body, or the diff for a maintainer-authored PR — not branch name; branch names encode run IDs and never repeat. The listing carries titles only; pull the body and author per candidate (`gh pr view <n> --json body,author`).
+ Match the current diagnostic evidence against the candidate PR's diagnosis and diff, per **Claims of recurrence** in `/tend-ci-runner:ground-claims`. Branch names encode run IDs and never repeat. The listing carries titles only; pull the body and author per candidate (`gh pr view <n> --json body,author`), then read its diff.
- If an existing **open** PR addresses the same failure, read its current head, checks, and conversation. For the configured bot's PR, continue the repair per `/tend-ci-runner:continue-pr`, then stop this workflow without opening another PR. For a human's in-flight fix, leave the work with them; post only new diagnostic evidence per `/tend-ci-runner:post-to-github`.
- If a **closed** PR with a maintainer rejection covers the same failure, exit silently; check the closure comment for the rationale before referencing it. Re-deriving the same fix forces a maintainer to close it twice.
- Also check for open tracking issues left by a prior unfixable diagnosis (see 3b) — if one matches the current failure shape, the fix PR you eventually open should reference it via `Fixes #<n>` so the issue closes when the PR merges:
+ Also check for open tracking issues left by a prior unfixable diagnosis (see 4b) — compare candidate diagnoses per **Claims of recurrence** in `/tend-ci-runner:ground-claims`. If one covers the current failure, the fix PR you eventually open should reference it via `Fixes #<n>` so the issue closes when the PR merges:
```bash
BOT_LOGIN=$(gh api user --jq '.login')
gh issue list --state open --author "$BOT_LOGIN" --search "ci-fix: in:title" \
--json number,title,body --limit 100
```
- ### 2. Diagnose and fix
+ ### 3. Diagnose and fix
The gates every fix clears are in `/tend-ci-runner:fix-a-bug`; read it before writing one.
- 1. Read the run's conclusion and jobs: `gh run view <run-id> --json conclusion,jobs,url`
- 2. For a failure, read its logs: `gh run view <run-id> --log-failed`
- 3. For a cancellation, use the job and step conclusions from step 1. Its log
- archive may not exist; a failed job inside the cancelled run is the signal
- to diagnose, not a log fetch.
- 4. Identify the root cause — don't just fix the symptom
- 5. Search for the same pattern elsewhere in the codebase
- 6. Reproduce locally using test commands from the project's instruction files
- 7. Fix at the right level (shared helper > per-file fix)
-
- A cancellation takes this same diagnostic path; do not presume it is transient
- or ignore it. First establish why it stopped. If concurrency replacement or a
- deliberate cancellation stopped a run that had no failure, exit silently — do
- not use the transient-tracker path below. Otherwise diagnose the completed
- steps and current branch normally.
+ 1. Identify the root cause — don't just fix the symptom
+ 2. Search for the same pattern elsewhere in the codebase
+ 3. Reproduce locally using test commands from the project's instruction files
+ 4. Fix at the right level (shared helper > per-file fix)
- ### 3. Create PR
+ ### 4. Create PR
- Re-run step 1's author-agnostic PR query, then follow **Dedup recheck immediately before `gh pr create`** in `/tend-ci-runner:open-pr` to check for a fix committed to the default branch. If the failure no longer reproduces there, don't open the PR. Before the push, review the change per **Review the change before the push** in `/tend-ci-runner:push-commits`.
+ Re-run step 2's author-agnostic PR query, then follow **Dedup recheck immediately before `gh pr create`** in `/tend-ci-runner:open-pr` to check for a fix committed to the default branch. If the failure no longer reproduces there, don't open the PR. Before the push, review the change per **Review the change before the push** in `/tend-ci-runner:push-commits`.
```bash
git checkout -b fix/ci-<run-id>
git add <files>
git commit -m "fix: <description>"
git push -u origin fix/ci-<run-id>
```
Create the PR with `gh pr create`, composing its body in a file per `/tend-ci-runner:post-to-github`. Write for a maintainer deciding whether the current fix addresses the failure: explain the causal finding, why the change fixes it at the right level, and the verification relevant to that decision. Link the failed run as supporting evidence and follow **Reader-facing prose** in `/tend-ci-runner:run-tend`.
- ### 3a. Diagnosis without a fix (transient causes)
+ ### 4a. Diagnosis without a fix (transient causes)
If the diagnosis identifies the failure as transient — runner-disk corruption, an isolated network blip, an upstream incident that has since resolved — there is no fix PR to create. Don't post the diagnosis as a commit comment (it surfaces on whatever commit triggered CI, including release commits where it's visibly off-topic).
- Use this path only when the evidence points to ephemeral infrastructure, not anything the project's code does. Signals (examples, not a checklist): the same code path passed on recent prior runs with no relevant change; the failure shape is filesystem/network-level; an upstream status incident matches the timing and components. If you can't tell whether it's transient, treat it as durable — create a fix PR, or follow 3b if a safe fix can't be produced.
+ Use this path only when the evidence points to ephemeral infrastructure, not anything the project's code does. Signals (examples, not a checklist): the same code path passed on recent prior runs with no relevant change; the failure shape is filesystem/network-level; an upstream status incident matches the timing and components. If you can't tell whether it's transient, treat it as durable — create a fix PR, or follow 4b if a safe fix can't be produced.
#### Repeat-occurrence escalation
Before filing the tracker below, check whether the same failure shape has already been classified transient recently. The single-shot criteria above don't catch an intermittent upstream regression — each rerun-pass reinforces the wrong classification.
```bash
BOT_LOGIN=$(gh api user --jq '.login')
gh issue list --state all --label tend-outage --author "$BOT_LOGIN" \
--search "ci-fix: transient failure in:title created:>=$(date -u -d '7 days ago' +%F)" \
--json number,title,body,createdAt --limit 100
```
- Both filters are load-bearing: the label keeps 3b's durable trackers out, and the title keeps out the `report_failure.py` **"Bot temporarily unavailable"** issues, which carry the same label and author and vastly outnumber these — without it the first page is all outage rows and the count reads zero.
+ Both filters are load-bearing: the label keeps 4b's durable trackers out, and the title keeps out the `report_failure.py` **"Bot temporarily unavailable"** issues, which carry the same label and author and vastly outnumber these — without it the first page is all outage rows and the count reads zero.
- Match by failure-shape keyword against the issue body (e.g. `rustup-init`, `composer connect timeout`, `docker pull rate limit`) — not by job name. The same root cause can surface on multiple jobs.
+ Use failure-shape keywords (e.g. `rustup-init`, `composer connect timeout`, `docker pull rate limit`) to find candidates. Read their diagnoses and compare the evidence per **Claims of recurrence** in `/tend-ci-runner:ground-claims` before counting occurrences. A keyword match finds evidence to inspect, not an occurrence to count; the same proven cause can surface on multiple jobs.
If the current failure shape has 2+ prior occurrences on separate days within the past 7, escalate to durable: a fault that keeps coming back within a week is not transient even when individual reruns pass. Count occurrences, not trackers — the same root cause taking down several jobs in one afternoon files several trackers and is still one occurrence.
- An escalated fault still reruns green, so a mitigation prevents future red runs rather than fixing a bug. Open a fix PR proposing the smallest mitigation that holds, usually one setting in one place (pin the runner image, skip the affected leg, disable the relevant cache layer), preferring an upstream-documented workaround — `gh issue search` against the action's repo, the action's README, GitHub Community threads — and linking the upstream issue if the search surfaced one. If the fault has no such mitigation, treat it as a durable cause without a safe fix and follow 3b.
+ An escalated fault still reruns green, so a mitigation prevents future red runs rather than fixing a bug. Open a fix PR proposing the smallest mitigation that holds, usually one setting in one place (pin the runner image, skip the affected leg, disable the relevant cache layer), preferring an upstream-documented workaround — `gh issue search` against the action's repo, the action's README, GitHub Community threads — and linking the upstream issue if the search surfaced one. If the fault has no such mitigation, treat it as a durable cause without a safe fix and follow 4b.
#### File the transient tracker
If the failure stays classified transient, open an issue with the diagnosis per `/tend-ci-runner:open-pr` and close it immediately. The closure records "diagnosed, no further action" while keeping the analysis discoverable and off the commit timeline. Apply the `tend-outage` label — the job-level `if:` in `tend-triage` and `tend-mention` skip labelled issues, suppressing the no-op cascade runs (`opened` → silent-exit; `closed`-comment → silent-exit) that would otherwise fire on every transient tracker:
```bash
gh label create tend-outage --description "Tracks bot outage incidents" --color "d93f0b" 2>/dev/null || true
gh issue create --title "ci-fix: transient failure on <run-id>" --label tend-outage --body-file "$TMPDIR/diagnosis.md"
gh issue close <issue-number> --reason "not planned" --comment "Transient — closing as diagnosed."
```
- Skip step 4 — there's no PR to monitor.
+ Skip step 5 — there's no PR to monitor.
- ### 3b. Diagnosis without a fix (durable causes)
+ ### 4b. Diagnosis without a fix (durable causes)
If the diagnosis identifies a durable root cause but a safe fix can't be produced — the cause is in an external system the bot can't change, the fix requires judgment the bot shouldn't make unilaterally, or an attempted fix didn't validate locally — leave a tracking issue. Without one, a durable failure that the bot can't fix lives only on the workflow-run page and is invisible in the issues list.
- Leave the issue **open**, so maintainers have a durable "still broken" signal until a fix ships. A subsequent fix PR closes it via `Fixes #<n>` in the PR body (see step 1 — search for a matching open tracking issue before opening the fix PR).
+ Leave the issue **open**, so maintainers have a durable "still broken" signal until a fix ships. A subsequent fix PR closes it via `Fixes #<n>` in the PR body (see step 2 — search for a matching open tracking issue before opening the fix PR).
- **Dedup first.** Search for an open tracking issue covering the same failure shape; if one exists, post an update on it rather than opening a duplicate. Match by failure shape (workflow name + diagnostic snippet), not run ID — each run ID is unique and won't dedup:
+ **Dedup first.** Search for an open tracking issue covering the same failure shape; if one exists, post an update on it rather than opening a duplicate. Use failure-shape keywords to find candidates, then match their diagnostic evidence per **Claims of recurrence** in `/tend-ci-runner:ground-claims`, not run ID — each run ID is unique and won't dedup:
```bash
BOT_LOGIN=$(gh api user --jq '.login')
gh issue list --state open --author "$BOT_LOGIN" --search "ci-fix: in:title" \
--json number,title,body --limit 100
```
- If an open tracking issue matches, read its whole thread first — the listing above returns only the body:
+ Read each candidate tracker's whole thread before matching or composing a recurrence update — the listing above returns only the body:
```bash
gh issue view <issue-number> --json body,comments
- gh issue comment <issue-number> --body-file "$TMPDIR/recurrence.md"
```
- The comment is an update for people following the tracker, not a log entry for this run. They already know CI is failing and have read the earlier reports, so tell them what they don't know yet: what changed since the thread's last update — failures that are new or now pass, a fix that merged or is still in flight, a revised diagnosis — or, when nothing did, that the same failures continue and this run is the latest case. Write it from that delta rather than from the shape of the earlier comments; the run link and supporting diagnosis back it up rather than open it.
+ When the diagnostic comparison establishes a match, update that tracker. The comment is an update for people following the tracker, not a log entry for this run. They already know CI is failing and have read the earlier reports, so tell them what they don't know yet: what changed since the thread's last update — failures that are new or now pass, a fix that merged or is still in flight, a revised diagnosis — or, when nothing did, that the same failures continue and this run is the latest case. Write it from that delta rather than from the shape of the earlier comments; the run link and supporting diagnosis back it up rather than open it. Compose `$TMPDIR/recurrence.md` and post per `/tend-ci-runner:post-to-github`:
+ ```bash
+ gh issue comment <issue-number> --body-file "$TMPDIR/recurrence.md"
+ ```
+
Otherwise, open a new tracking issue per `/tend-ci-runner:open-pr`. Use a title prefix that future runs can search on (`ci-fix: <workflow-name> failing`) with a short root-cause suffix for human readability:
```bash
gh issue create \
--title "ci-fix: <workflow-name> failing — <short root cause>" \
--body-file "$TMPDIR/diagnosis.md"
```
Compose `$TMPDIR/diagnosis.md` as a durable account for a maintainer deciding what happens next. Include the failed workflow and run, the root cause and mechanism, why no safe automated fix was produced, and the current blocker or decision. Follow **Reader-facing prose** in `/tend-ci-runner:run-tend`; the issue should preserve the conclusion, not the diagnostic transcript.
- Skip step 4 — there's no PR to monitor.
+ Skip step 5 — there's no PR to monitor.
- ### 4. Monitor CI
+ ### 5. Monitor CI
Wait for CI per `/tend-ci-runner:monitor-ci`.