failure-models git:20260927.9ecadfdA

Immutable. This exact content never changes and is served at /api/v1/blob/d42c95b923767428.

---
name: failure-models
description: >
  Stating a system's fault model before designing against it: crash-stop, crash-recovery,
  omission, timing and Byzantine faults; partial failure and the third outcome of every
  remote call (unknown); gray failure and the slow node whose health check stays green; the
  eight fallacies as a checklist; blast radius, correlated versus independent failure, and
  the availability arithmetic of a dependency chain. Use when a design says "if the service
  is down" without defining down, when a retry is added to a call whose outcome is unknown,
  when a node is slow rather than dead, when replicas share a host, an AZ or a database, or
  when an availability target is quoted for a service built on ten others. Does not cover
  what the model implies about messages (delivery-semantics) or reads (consistency-models),
  how a failure spreads (cascading-failures), the named shapes
  (distributed-failure-catalogue), what an orchestrator does with a failed pod
  (kubernetes-service-lifecycle), or failures as types (java-exception-design).
---

# Failure Models

## Purpose

Make the relevant fault model explicit before relying on it in a design. Every downstream decision —
whether a retry is safe, whether a read may be stale, how many replicas are enough — is an
answer to "which faults do we tolerate?", and a design that never asked the question has
answered it by accident.

The failure this prevents is the one-word fault model. "The service can go down" is not a
model: it leaves restart behavior, outstanding effects and shared failure domains unspecified.
Even a crash-stop peer may have applied a write before its acknowledgement was lost; the
fault class alone does not resolve that outcome. Naming the class and its assumptions makes
recovery, retry and redundancy claims reviewable.

## Workflow

Match the depth to the requested diagnosis, design or review. A narrow outcome or arithmetic
question can finish with the relevant card fields inline, the supporting evidence and its limits.
Reuse adequate models and tests; preserve findings-only scope and an already sound design.

1. **Inspect the contract and write or update the relevant boundary's fault-model card.**
   Start with the requested operation, its success/degraded-success criteria, existing model,
   client/driver behavior and deployment evidence. Reuse established facts; ask only for gaps
   that change safety, recovery or the recommendation. Name the fault classes, failure domains,
   synchrony assumption, recovery source, detection mechanism, and maximum tolerated
   combination. Crash-stop, crash-recovery, omission, timing and Byzantine are not labels
   for the whole system: a trusted database replica may be crash-recovery while an
   Internet-facing client is arbitrary or hostile. A process that restarts from durable
   state is crash-recovery; any in-flight operation whose completion was not durably
   recorded must be retried, reconciled, or abandoned by an explicit rule.
2. **Distinguish known applied, known not applied and unknown effects.** Name the effect
   boundary: a multi-effect operation can be partly complete, so preserve each effect's
   status instead of calling the whole operation rejected or rolled back. A read timeout,
   reset after dispatch or missing broker ack normally leaves the effect unknown. Decide
   what happens on unknown: repeat safely, reconcile, or escalate. Proven non-application
   can permit retry even for a non-idempotent operation, subject to cause and budget; a
   rejected later attempt does not clear an earlier unknown effect. See
   `references/the-unknown-outcome.md`.
3. **Check a relevant gray failure.** For example, a node passes its health check while its
   responses miss the caller's deadline. Assess detection, resource occupancy and remaining
   capacity under the actual load; distinguish a tolerable slowdown from a violated contract.
4. **Draw the failure domains.** For a process, a host, a rack, an AZ, a dependency and a
   deploy, write what a specified fault can affect. Replicas sharing a domain fail together for that
   cause; they may still tolerate independent process faults.
5. **Do conditional availability arithmetic** on the request path before promising a
   number. Label inputs as observations, modeling assumptions or objectives; an SLA/SLO target
   is not an observed probability or an availability ceiling. Required dependencies in series multiply availability only when their events are
   independent and their SLI windows and success definitions align; genuinely independent
   redundant alternatives multiply _unavailability_. Correlated and conditional failure
   needs a measured joint distribution or an explicit common-cause model for a point estimate.
   Without dependence evidence, use bounds when they can resolve the decision. See
   `references/failure-domains-and-arithmetic.md`.
6. **Walk the eight fallacies as a checklist** — reliable network, zero latency, infinite
   bandwidth, secure network, unchanging topology, one administrator, zero transport cost,
   homogeneous network. Use code patterns as investigation leads: check whether an overall
   deadline bounds a client without a separate read timeout, admission bounds queue growth,
   and discovery/connection refresh handles topology changes. Report the effective behavior
   and violated contract, not a defect inferred from one missing API call.

Inspect the actual client/driver, retry, durability and deployment configuration before assigning
outcomes. The conceptual Java type uses sealed classes/records (Java 17); exhaustive pattern
switch without preview requires Java 21. These examples do not set a platform requirement;
preserve the project's target. Deliver the relevant card, evidence versus assumptions, outcome
and recovery policies, and a recommendation compared with retaining the current design and
any materially relevant alternative. Name what evidence would change it. When a recovery or
tolerance claim needs experimental evidence, include a focused fault-injection case with an
observable invariant; distinguish proposed checks from executed
results. Missing protocol or topology evidence means a conditional claim, not a replica count
or availability promise.

## Fault classes

```text
Assume crash-stop when:
- the algorithm may treat a stopped participant as never returning. Replacing its process
  identity with a fresh replica does not make the original participant crash-recovery.
Assume crash-recovery when:
- the same logical participant can return after a crash and recover durable state — local
  log, database rows, committed offsets, epochs or leases. Volatile state is lost; durable
  state may lag acknowledged work unless the durability contract proves otherwise.
Assume omission (a message or a response silently lost) when:
- the underlying transport, queue, proxy or load balancer can drop sends or receives.
  A higher-level reliable-channel abstraction may mask omissions, but its retry,
  deduplication and terminal-failure assumptions then become part of the model.
Assume timing/performance failure when:
- correctness or usefulness depends on a deadline. A correct-but-late response can be a
  failure to the caller. Allocate the caller's end-to-end deadline across attempts,
  queueing and cleanup; a timeout is not automatically useful merely because it is shorter.
Include Byzantine faults when:
- input crosses a trust boundary — a client, another tenant, a third party — where a
  participant may send arbitrary, inconsistent or hostile data. Input validation protects
  an API but does not make its replication protocol Byzantine-fault tolerant. Under common
  quorum protocols, tolerating `f` crash failures typically needs `2f+1` voting members and
  Byzantine agreement commonly needs `3f+1`; the exact bound depends on synchrony,
  authentication, quorum and protocol assumptions. State those assumptions instead of
  transplanting a replica count.
```

## Fault-model card

For every important operation, make these fields reviewable:

| Field              | Question that must have an answer                                                                    |
| ------------------ | ---------------------------------------------------------------------------------------------------- |
| Safety invariant   | What must remain true even during a partition, retry or recovery?                                    |
| Liveness condition | Under which timing and quorum assumptions must progress resume?                                      |
| Faults tolerated   | Crash-stop, crash-recovery, send/receive omission, delay, corruption, arbitrary peer?                |
| Bound              | How many simultaneous faults, and in which independent domains?                                      |
| Detector           | Timeout, lease, heartbeat, quorum observation, operator signal? Which false suspicion is acceptable? |
| Durable truth      | Which log, row, offset, epoch or manifest reconstructs state after restart?                          |
| Ambiguous effect   | How is an unknown outcome deduplicated, queried, reconciled or escalated?                            |
| Recovery objective | What RTO/RPO and backlog-drain time are required, and under what load?                               |
| Re-entry           | How is a recovered or partitioned participant fenced before it can mutate state again?               |

Do not merge **fault**, **error** and **failure**. A fault is the hypothesised cause; an error
is incorrect internal state; a failure is externally visible deviation from the service
contract. One host reboot can cause ten failed requests: count the incident once and the ten
request failures in a request-based SLI. A masked reboot may cause no service failure. State
the counting unit, and do not dismiss a latency SLO failure because responses were eventually
correct.

## Rules

- **Partial failure creates outcome uncertainty.** A local exception does not imply rollback
  either; remote calls additionally decouple caller observation from peer execution. A remote
  call can leave you not knowing — and that third outcome is where many
  distributed bugs arise. Code that maps a timeout onto "the effect never applied" has erased it:
  a timeout alone does not establish the callee's state, and the callee may complete the
  work after the caller gave up.
- Crash-recovery makes the recovery path a correctness surface. For each recovery step,
  state whether it is idempotent and under which key. The mechanics are `idempotency`; the
  requirement to have an answer is here.
- **A slow node can be more damaging than a dead one.** A definitively stopped endpoint is
  eventually excluded; a slow endpoint may retain traffic and consume caller threads,
  connections and deadline budget. But aggressive suspicion can eject a healthy node and
  destroy quorum or capacity. That the system's view of health can differ from the client's
  — _differential observability_ — is why "fail fast" is a policy, not a fact. In a fully
  asynchronous network a detector cannot distinguish crash from unbounded delay; practical
  systems assume some eventual timing bound and trade false suspicion against detection
  delay. Record that trade-off for readiness checks, leases and failover.
- **Redundancy must name the fault it tolerates.** Three replicas on one host cannot survive
  losing that host, but may tolerate an individual process crash. Ask what they share: host, rack, AZ, control plane,
  image, config, deploy, certificate, downstream dependency. A shared deploy is the one most
  often missed — rolling one bad artefact to every replica introduces a common cause, which
  makes deploy strategy an availability control rather than a release convenience.
- **Adding a required dependency multiplies availability under an independence model and
  therefore increases total unavailability.** Ten independent dependencies at 99.9% in
  series produce about 99.0% path availability. For a time-based SLI on a 365-day year,
  that corresponds to roughly 87 unavailable hours, not 8.8; request failure fractions do
  not directly convert to outage hours. Real incidents are often correlated, so use this as a comparison model, not a
  forecast. Either define a tested degraded mode that removes the dependency from the
  required path, or stop quoting the higher number. Redundant alternatives multiply
  unavailability only under independence; common causes set an unavailability floor.

## Decision framework

```text
If the operation crosses a process boundary:
  classify timeout/cancellation/disconnect as Unknown unless protocol or authoritative
  state establishes the relevant effect's outcome.

If progress requires suspecting a peer:
  separate suspicion from authority to act; preserve the protocol's safety assumptions.
  Tune detection for recovery speed and false suspicion, and state any timing/clock bounds
  required for safety, especially with leases.

If replicas share any host, zone, control plane, deploy, credential or dependency:
  model that cause once as a common failure domain;
  do not multiply replica availability as if independent.

If stale or recovered writers could violate the invariant:
  identify how the protected resource rejects their effects, for example through a validated
  fencing protocol or atomic conditional write. Verify the actual claim/effect contract;
  a client-side check followed by a pause and an unchecked write is not enforcement.
  Detailed lease/fencing protocol design belongs to distributed-locks-and-leases.

If the design claims availability during partition:
  state which operations remain safe, which side may progress, and what reconciliation
  occurs after healing. "The service stays up" is not a consistency contract.
```

## Failure injection and recovery proof

Select fault cases relevant to the card. Before injection, establish an isolated or explicitly
authorized target, observable invariants, blast radius, abort conditions and recovery steps:

- inject loss separately before send, after apply/before acknowledgement, and during
  response transfer; assert downstream state and duplicate count;
- pause a process and add latency/jitter rather than testing only clean termination;
- partition asymmetrically (`A` reaches `B`, `B` cannot reach `A`) and isolate data plane
  from control plane;
- crash after every durable-write boundary, restart from persisted state, and verify the
  safety invariant plus bounded recovery;
- expire credentials, deploy incompatible versions, exhaust pools/disk/file descriptors,
  and restore backups into an isolated environment;
- run at realistic load: failover that takes 20 seconds when idle can create hours of
  recovery backlog at saturation.

Observe detection latency, false-positive rate, unknown outcomes, duplicate/reconciliation
counts, quorum loss, recovery backlog, RTO and recovered data point (RPO). A test that only
asserts the client exception does not validate the distributed outcome.

## Anti-patterns

| Anti-pattern                         | Why dangerous / symptom                                                 | Better alternative                                                          |
| ------------------------------------ | ----------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| One fault model for the whole system | Trust and durability assumptions leak across boundaries                 | Model each operation and participant role, then compose them                |
| Timeout means rollback               | Retried writes duplicate after lost acknowledgements                    | Preserve `Unknown`; use idempotency, status lookup or reconciliation        |
| Health check means truth             | Gray/asymmetric failures stay green or healthy nodes flap               | Compare client-view signals; define detector error costs                    |
| Replica count means availability     | Common deploy, zone or datastore defeats all replicas                   | Draw domains and measure joint/common-cause failures                        |
| Failover equals recovery             | Traffic moves but stale owners write, data is missing, backlog explodes | Fence old owners; prove state recovery and capacity during catch-up         |
| Chaos without invariants             | Generates outages but no falsifiable learning                           | Declare safety/liveness hypotheses, blast radius and abort conditions first |

## References

- [The unknown outcome](references/the-unknown-outcome.md) — the three-outcome model in Java,
  how a JDBC, HTTP and Kafka call each maps onto it, and what an unknown write forces the
  design to provide. Read when adding a retry, handling a timeout, or writing across a
  process boundary.
- [Failure domains and availability arithmetic](references/failure-domains-and-arithmetic.md)
  — series and parallel composition worked through, correlated failure, and the questions
  that expose a hidden shared dependency. Read when promising an availability number, sizing
  replicas, or reviewing a topology.
- [Chubby locks and sequencers](https://static.usenix.org/events/osdi06/tech/full_papers/burrows/burrows_html/)
  and [Redis lock assumptions](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/)
  — consult when checking stale-owner enforcement and a lease protocol's timing assumptions.