quality-engineering-flaky-triage git:20260913.6db6862A

Immutable. This exact content is served forever at /api/v1/blob/43aec22376d71b15.

---
name: quality-engineering-flaky-triage
description: Quarantines an intermittently failing test behind a ticket with an owner and expiry, assigns a root-cause bucket from isolated reruns, and defines when the test leaves quarantine. Use when a test is a QUARANTINE_CANDIDATE or fails without a code change; not for deterministic failures.
guardrail: true
metadata:
  triggers:
    files:
      - "**/quarantine.json"
      - "**/.flaky-tests*"
    keywords:
      - flaky test
      - flake triage
      - quarantine test
      - intermittent failure
      - unquarantine
      - retry budget
      - passes on rerun
---
# Quality Engineering: Flaky Triage

## **Priority: P0 (CRITICAL)**

## Entry

Only a `QUARANTINE_CANDIDATE` verdict enters triage: the test fails intermittently across isolated reruns with no code change between runs. A deterministic failure is `REAL_BUG_DO_NOT_HEAL` and goes to `dev-fix`, never here.

## Quarantine Contract

- Quarantine means the test still runs, its result is reported, and it does not gate merge; it never means `test.skip`.
- Every quarantined test carries a ticket with an owner and an expiry of at most 14 days. No ticket, no quarantine.
- On expiry: fixed and un-quarantined, or expiry extended once with a written reason, or the test is deleted with the coverage gap recorded in the coverage report. Silent expiry is forbidden.
- Record each entry in `flake_quarantine[]` as `{test, ticket, expiry, bucket}` so `test-loop` carries it in its handoff.

## Root-Cause Buckets

`ORDER_DEPENDENCE` · `SHARED_STATE` · `TIMING` · `ENVIRONMENT` · `DATA` · `PRODUCT_NONDETERMINISM`. Assign exactly one from the evidence in [Root-Cause Buckets](references/root-cause-buckets.md); `UNKNOWN` is allowed only with the reruns that were tried.

## Evidence Required

Run the test alone 10 times sequentially in the foreground. Record pass count, the failing run's artifact, and whether the failure reproduces with a fresh environment. Fewer than 10 isolated runs is not evidence; a pass on plain CI rerun is not evidence.

## Un-quarantine

A test leaves quarantine only when the root cause is fixed and it passes 10 consecutive isolated green runs after the fix. Retries, longer timeouts, or a wider matcher are not fixes; they hide the bucket.

## Reliability Impact

Each flaky test counts against `suite_reliability_pct` in `quality-engineering-automation-health` until its root cause is fixed; quarantine removes its reds from the merge gate, not from the report. State the quarantine count and the oldest expiry next to `release_confidence`.

## Red Flags

"just add retries: 3" · "skip it for now, we'll come back" · "it passes locally" · "extend the expiry again" — each hides a bucket instead of naming it. Stop, run the 10 isolated reruns, assign the bucket.

## Anti-Patterns

- **No skip as quarantine**: a skipped test reports nothing; quarantine still runs and reports.
- **No ticketless quarantine**: a quarantine without owner and expiry is a permanent skip in disguise.
- **No retry as fix**: `retries: N` is not a fix; it raises the pass rate and leaves the bucket unnamed.
- **No un-quarantine on a single green**: 10 consecutive isolated green runs after the fix, or it stays.

## References

- [Quarantine Ticket Template](references/quarantine-ticket-template.md)
- [Root-Cause Buckets](references/root-cause-buckets.md)
- [Reliability Math](references/reliability-math.md)