git:20260719.2d0d9d2 to git:20260728.1bce424

---

name: ci-failure-triage

description: "Decides what kind of failure a red CI test is before anyone starts fixing it. Extracts seven failure signals from the runner output, stack trace, run history, and environment metadata, then walks an ordered first-match-wins rule set to exactly one verdict: flaky-known, environment-drift, defect, timeout, flaky-pre-incident, or flake-of-unknown-cause. Emits the verdict together with the alternatives that were rejected and the specific condition each one failed, so the triage decision is auditable rather than asserted. Use when a test has just gone red and the next action depends on whether the cause is a product defect, a non-deterministic test, or drifted infrastructure."

---

# CI failure triage

One test is red. Before anyone opens a ticket, reruns the job, or edits a

line, someone has to answer a prior question: what kind of failure is this?

This is the method for answering it, and for writing the answer down in a

form a second person can check.

The cost of getting it wrong is measurable. Google reports that about 84% of

the pass-to-fail transitions its CI observes involve a flaky test, and that

## What this owns, and what it does not

This owns **classification**: turning one failed run into one verdict, with

the rejected alternatives recorded.

It deliberately does not own:

- **Report formatting.** A bug-report schema (summary, environment, steps to

reproduce, expected, actual, severity, priority, reproducibility) is a

separate job that starts *after* you already know the failure is a defect.

This decides which kind of failure you are looking at; that fills in the

form once you know.

- **Remediation.** No fix hypothesis, no patch, no rerun, no quarantine

action. The verdict names the next step; it does not take it.

- **Suite-wide prediction.** One failure at a time. Screening a whole green

suite for tests likely to flake next is a different technique with

different inputs.

- **Retrieval.** How you obtain the log, the run history, or the environment

metadata is out of scope. This assumes they are in front of you.

## Terminology

`flaky-test`, `flaky`, and `non-deterministic-test` returns 404 in all three

underneath a test that did not.

## Step 1 - Extract the seven signals

Record all seven before applying any rule. A rule that fires on a signal you

never extracted is a guess.

| # | Signal | What to record |

|---|---|---|

| S1 | **Failure mode** | One token from the pattern table below. |

| S2 | **Reproducibility window** | Pass-to-fail ratio over the last 50 runs; length of the current red streak; the first red commit. |

| S3 | **Co-failure pattern** | Did other tests fail in the same run? Same suite? Same shard? |

| S4 | **Time clustering** | Are failures concentrated in one window (a deploy, off-hours, peak load)? |

| S5 | **Environment delta** | Did the runner image, container tag, or base build hash change inside the failure window? |

| S6 | **Change-set proximity** | Did files in the test's call graph change inside the window? |

| S7 | **Quarantine status** | Is the test on a formal quarantine list (an annotation, a decorator, a CI quarantine tag, a checked-in flake list)? |

The 50-run and 7-day windows are **practitioner conventions**, not values from

any standard. Pick windows your history actually covers and state them in the

verdict, because every rule below is relative to them.

### S1: the failure-mode pattern table

Match in the order listed. The order is by decreasing specificity, and crash

signatures come first because a dying process frequently emits a trailing

timeout that would otherwise capture the match.

| Mode token | Pattern in the runner output or trace |

|---|---|

| `runner-crash` | `SIGSEGV`, `Segmentation fault`, `OutOfMemoryError`, `panic:`, an unhandled exception that kills the process. |

| `setup-error` | A `BeforeEach`, `setUp`, `@BeforeAll`, or fixture frame in the trace. |

| `environmental` | `ENOENT`, `FileNotFoundException`, a missing environment variable, `connection refused`. |

| `timed-out` | `Timeout` or `exceeded N ms` with no other exception attached. |

| `unclassified` | None of the above matched. |

`timed-out` is the **mode**. `timeout` is a **verdict**. They are not the same

claim: the mode says the clock ran out, the verdict says the CI infrastructure

is why. R3 and R4 below decide between them.

### S6: read the diff, do not skim it

Change-set proximity means files on the test's call graph changed inside the

window, and you read the change. Google's engineering practices put the

obligation plainly for reviewers: read and understand the change before acting

## Step 2 - Walk the rules, first match wins

The verdicts are mutually exclusive and single-valued. Evaluate R1 through R6

in order and stop at the first rule whose conditions all hold. A failure with

several plausible causes gets the earliest matching verdict, not two verdicts.

### R1 - `flaky-known`

Fires when S7 shows the test on a **formal** quarantine list **and** the

observed S1 mode matches the flake category already recorded for it.

Next step: rerun, and let the existing quarantine process handle it. No new

triage. Fowler's advice is to quarantine non-deterministic tests out of the

Informal evidence does not count. A code comment saying "this one sometimes

fails" is not a quarantine list. If the project has no detectable quarantine

convention, R1 can never fire and you fall through to R2.

### R2 - `environment-drift`

Fires when **all** hold:

- S5 shows a runner image, container tag, or base build hash change inside the

window, and

- the test passed on the old environment and fails on the new one, and

- S1 is `runner-crash`, `setup-error`, or `environmental` (never

`assertion-mismatch`).

Next step is **not** a defect ticket. It is a re-pin of the image or an

investigation of how the environment is provisioned, routed to whoever owns

the CI platform.

An `environmental` mode with **no** S5 delta is not drift. The environment did

not change; the test depends on state nobody controls. That case falls through

to R5 or R6 as a test-isolation problem.

### R3 - `defect`

Fires when **all** hold:

- the test was green for the previous N runs (N=5 is a common default, a

convention rather than a standard; state the N you used), and

- S6 shows files on the call graph changed inside the window, and

- S1 is `assertion-mismatch`, `selector-breakage`, or `setup-error`; or S1 is

`timed-out` **and** the diff touched a timing constant or the code path under

test, and

- the failure reproduces on a rerun of the same commit, where the history

contains such a rerun.

This is the highest-confidence verdict available, because four independent

signals agree. Next step: capture whatever trace or artifact the runner can

produce, draft the defect report against the schema the team uses, and lock

the reproduction in a committed failing test before anyone edits production

code.

### R4 - `timeout`

Fires when S1 is `timed-out` with no other exception **and** either:

- S3 or S2 show similar timing-edge failures on *different* tests in the same

window, which points at shared infrastructure rather than this test's logic,

or

- the runner's CPU or memory profile for the run shows resource saturation.

Next step is a suite time-budget review or runner resource tuning, not a

defect ticket. R3 has already claimed the timeouts that have code-change

proximity, so anything reaching R4 has none.

### R5 - `flaky-pre-incident`

Fires when **all** hold:

- S2 shows at least one prior failure of this same test inside the last 50

runs, so the pattern is intermittent rather than a clean break, and

- S6 shows no change-set proximity, and

- the S1 mode and the trace are shaped like async-wait, concurrency, or

test-order dependency.

Those three are the top root-cause categories in the largest published study

of flaky-test fixes: of 161 classified fixes drawn from 201 flaky-test commits

across 51 Apache projects, 45% were async wait, 20% concurrency, and 12% test

This is the "flaking, not yet quarantined" verdict. Next step: pattern

attribution, then quarantine if the pattern is confirmed.

### R6 - `flake-of-unknown-cause`

The fallback. Nothing above matched, or S1 came back `unclassified`. Emit a

low-confidence verdict and recommend bisection-style narrowing across run

conditions. R6 is an honest answer, and it is a better one than a `defect`

verdict propped up on two of four signals.

### Why this order

R2 sits ahead of R3 on purpose. Both can look plausible when a test starts

failing after an image bump that also landed near a code change, and routing

drift to the defect tracker sends the work to the wrong team while the tracker

fills with false positives. Putting the environment check first makes the

precedence explicit rather than leaving it to the mode preconditions.

R3 sits ahead of R5 on purpose too: when there is code-change proximity, an

async-wait timeout is more likely a product that got slower than a test that

got unlucky.

## Step 3 - Emit the verdict

One fixed shape. The **Not classified as** block is mandatory, including when

the verdict is high-confidence.

```markdown

## Failure classification - `<test-id>`

**Verdict:** <flaky-known | environment-drift | defect | timeout | flaky-pre-incident | flake-of-unknown-cause>

**Confidence:** <high | medium | low>

**Windows used:** <e.g. last 50 runs, 7 days of history, N=5 green-run threshold>

**Evidence:**

- <one line per signal that carried the verdict, naming the signal (S1..S7) and the observed value>

**Recommended next step:**

1. <action>

2. <action>

**Not classified as:**

- `<verdict>` - <the specific condition that failed, with the observed value>

- `<verdict>` - <the specific condition that failed, with the observed value>

```

### The Not-classified-as block is the load-bearing part

A bare verdict is an assertion. A verdict plus the rejected alternatives and

the exact condition each one failed is a decision another person can audit,

reopen, and overturn when a signal later turns out to be wrong.

Rules for the block:

- List **every** verdict that was evaluated and not returned. Do not list only

the near miss.

- For each, name the **one specific condition** that failed and the value you

observed, not a restatement of the rule. "`environment-drift` - runner image

and container tag unchanged across the window" is auditable. "Does not look

like environment drift" is not.

- When a signal was unavailable rather than negative, say so: "`environment-drift`

- no environment metadata captured for this run, S5 could not be evaluated."

That distinction is what tells a reader whether to re-triage once the missing

signal arrives.

- Keep it even at high confidence. The high-confidence verdicts are the ones

nobody re-examines later, which is exactly when a silent misroute is

expensive.

## Worked example

## Anti-patterns

## Confidence and limits

- **Confidence tracks signal availability, not conviction.** Without S5, R2

cannot be evaluated and no verdict may exceed medium. Without S6, R3 cannot

reach high. Say which signal was missing rather than lowering the label

silently.

- **New tests cannot be classified for the first window.** A test merged two

days ago has no S2, so R3 and R5 are both unavailable and R6 is usually the

only honest answer.

- **Co-failure detection is heuristic.** S3 reports that other tests failed in

the same run, suite, or shard. It does not prove shared-state coupling; that

needs targeted isolation analysis.

- **`unclassified` is a real outcome.** When S1 matches nothing in the pattern

table, record `unclassified` and let it flow to R6. Forcing a mode token to

make a rule fire is how a wrong verdict acquires false confidence.

- **One failure per pass.** For an overnight run with forty reds, run the

method forty times. It is not a report-summarising technique, and batching

it destroys the per-failure signal set the rules depend on.