git:20260911.37b054b to git:20260923.832673e

---

name: failure-models

description: >

Stating a system's fault model before designing against it: crash-stop, crash-recovery,

omission, timing and Byzantine faults; partial failure and the third outcome of every

remote call (unknown); gray failure and the slow node whose health check stays green; the

eight fallacies as a checklist; blast radius, correlated versus independent failure, and

the availability arithmetic of a dependency chain. Use when a design says "if the service

is down" without defining down, when a retry is added to a call whose outcome is unknown,

when a node is slow rather than dead, when replicas share a host, an AZ or a database, or

when an availability target is quoted for a service built on ten others. Does not cover

what the model implies about messages (delivery-semantics) or reads (consistency-models),

how a failure spreads (cascading-failures), the named shapes

(distributed-failure-catalogue), what an orchestrator does with a failed pod

(kubernetes-service-lifecycle), or failures as types (java-exception-design).

---

# Failure Models

## Purpose

Make the relevant fault model explicit before relying on it in a design. Every downstream decision —

whether a retry is safe, whether a read may be stale, how many replicas are enough — is an

answer to "which faults do we tolerate?", and a design that never asked the question has

answered it by accident.

The failure this prevents is the one-word fault model. "The service can go down" is not a

## Workflow

1. **Inspect the contract and write or update the relevant boundary's fault-model card.**

Start with the requested operation, its success/degraded-success criteria, existing model,

client/driver behavior and deployment evidence. Reuse established facts; ask only for gaps

that change safety, recovery or the recommendation. Name the fault classes, failure domains,

synchrony assumption, recovery source, detection mechanism, and maximum tolerated

combination. Crash-stop, crash-recovery, omission, timing and Byzantine are not labels

for the whole system: a trusted database replica may be crash-recovery while an

Internet-facing client is arbitrary or hostile. A process that restarts from durable

state is crash-recovery; any in-flight operation whose completion was not durably

recorded must be retried, reconciled, or abandoned by an explicit rule.

2. **Distinguish known applied, known not applied and unknown effects.** Name the effect

boundary: a multi-effect operation can be partly complete, so preserve each effect's

status instead of calling the whole operation rejected or rolled back. A read timeout,

reset after dispatch or missing broker ack normally leaves the effect unknown. Decide

what happens on unknown: repeat safely, reconcile, or escalate. Proven non-application

can permit retry even for a non-idempotent operation, subject to cause and budget; a

rejected later attempt does not clear an earlier unknown effect. See

`references/the-unknown-outcome.md`.

3. **Check a relevant gray failure.** For example, a node passes its health check while its

responses miss the caller's deadline. Assess detection, resource occupancy and remaining

capacity under the actual load; distinguish a tolerable slowdown from a violated contract.

4. **Draw the failure domains.** For a process, a host, a rack, an AZ, a dependency and a

deploy, write what a specified fault can affect. Replicas sharing a domain fail together for that

cause; they may still tolerate independent process faults.

5. **Do conditional availability arithmetic** on the request path before promising a

number. Required dependencies in series multiply availability only when their events are

independent and their SLI windows and success definitions align; genuinely independent

redundant alternatives multiply _unavailability_. Correlated and conditional failure

needs a measured joint distribution or an explicit common-cause model. See

`references/failure-domains-and-arithmetic.md`.

6. **Walk the eight fallacies as a checklist** — reliable network, zero latency, infinite

bandwidth, secure network, unchanging topology, one administrator, zero transport cost,

homogeneous network. Use code patterns as investigation leads: check whether an overall

deadline bounds a client without a separate read timeout, admission bounds queue growth,

and discovery/connection refresh handles topology changes. Report the effective behavior

and violated contract, not a defect inferred from one missing API call.

Inspect the actual client/driver, retry, durability and deployment configuration before assigning

outcomes. The conceptual Java type uses sealed classes/records (Java 17); exhaustive pattern

switch without preview requires Java 21. These examples do not set a platform requirement;

preserve the project's target. Deliver the relevant card, evidence versus assumptions, outcome

and recovery policies, and a recommendation compared with retaining the current design and

any materially relevant alternative. Name what evidence would change it. Include a focused

fault-injection case with an observable invariant; distinguish proposed checks from executed

results. Missing protocol or topology evidence means a conditional claim, not a replica count

or availability promise.

## Fault classes

```text

Assume crash-stop when:

- the algorithm may treat a stopped participant as never returning. Replacing its process

identity with a fresh replica does not make the original participant crash-recovery.

Assume crash-recovery when:

- the same logical participant can return after a crash and recover durable state — local

log, database rows, committed offsets, epochs or leases. Volatile state is lost; durable

state may lag acknowledged work unless the durability contract proves otherwise.

Assume omission (a message or a response silently lost) when:

- the underlying transport, queue, proxy or load balancer can drop sends or receives.

A higher-level reliable-channel abstraction may mask omissions, but its retry,

deduplication and terminal-failure assumptions then become part of the model.

Assume timing/performance failure when:

- correctness or usefulness depends on a deadline. A correct-but-late response can be a

failure to the caller. Allocate the caller's end-to-end deadline across attempts,

queueing and cleanup; a timeout is not automatically useful merely because it is shorter.

Include Byzantine faults when:

- input crosses a trust boundary — a client, another tenant, a third party — where a

participant may send arbitrary, inconsistent or hostile data. Input validation protects

an API but does not make its replication protocol Byzantine-fault tolerant. Under common

quorum protocols, tolerating `f` crash failures typically needs `2f+1` voting members and

Byzantine agreement commonly needs `3f+1`; the exact bound depends on synchrony,

authentication, quorum and protocol assumptions. State those assumptions instead of

transplanting a replica count.

```

## Fault-model card

For every important operation, make these fields reviewable:

| Field | Question that must have an answer |

| ------------------ | ---------------------------------------------------------------------------------------------------- |

| Safety invariant | What must remain true even during a partition, retry or recovery? |

| Liveness condition | Under which timing and quorum assumptions must progress resume? |

| Faults tolerated | Crash-stop, crash-recovery, send/receive omission, delay, corruption, arbitrary peer? |

| Bound | How many simultaneous faults, and in which independent domains? |

| Detector | Timeout, lease, heartbeat, quorum observation, operator signal? Which false suspicion is acceptable? |

| Durable truth | Which log, row, offset, epoch or manifest reconstructs state after restart? |

| Ambiguous effect | How is an unknown outcome deduplicated, queried, reconciled or escalated? |

| Recovery objective | What RTO/RPO and backlog-drain time are required, and under what load? |

| Re-entry | How is a recovered or partitioned participant fenced before it can mutate state again? |

Do not merge **fault**, **error** and **failure**. A fault is the hypothesised cause; an error

is incorrect internal state; a failure is externally visible deviation from the service

contract. One host reboot can cause ten failed requests: count the incident once and the ten

request failures in a request-based SLI. A masked reboot may cause no service failure. State

the counting unit, and do not dismiss a latency SLO failure because responses were eventually

correct.

## Rules

- **Partial failure creates outcome uncertainty.** A local exception does not imply rollback

either; remote calls additionally decouple caller observation from peer execution. A remote

call can leave you not knowing — and that third outcome is where many

distributed bugs arise. Code that maps a timeout onto "the effect never applied" has erased it:

a timeout alone does not establish the callee's state, and the callee may complete the

work after the caller gave up.

- Crash-recovery makes the recovery path a correctness surface. For each recovery step,

state whether it is idempotent and under which key. The mechanics are `idempotency`; the

requirement to have an answer is here.

- **A slow node can be more damaging than a dead one.** A definitively stopped endpoint is

eventually excluded; a slow endpoint may retain traffic and consume caller threads,

connections and deadline budget. But aggressive suspicion can eject a healthy node and

destroy quorum or capacity. That the system's view of health can differ from the client's

— _differential observability_ — is why "fail fast" is a policy, not a fact. In a fully

asynchronous network a detector cannot distinguish crash from unbounded delay; practical

systems assume some eventual timing bound and trade false suspicion against detection

delay. Record that trade-off for readiness checks, leases and failover.

- **Redundancy must name the fault it tolerates.** Three replicas on one host cannot survive

losing that host, but may tolerate an individual process crash. Ask what they share: host, rack, AZ, control plane,

image, config, deploy, certificate, downstream dependency. A shared deploy is the one most

often missed — rolling one bad artefact to every replica introduces a common cause, which

makes deploy strategy an availability control rather than a release convenience.

- **Adding a required dependency multiplies availability under an independence model and

therefore increases total unavailability.** Ten independent dependencies at 99.9% in

series produce about 99.0% path availability. For a time-based SLI on a 365-day year,

that corresponds to roughly 87 unavailable hours, not 8.8; request failure fractions do

not directly convert to outage hours. Real incidents are often correlated, so use this as a comparison model, not a

forecast. Either define a tested degraded mode that removes the dependency from the

required path, or stop quoting the higher number. Redundant alternatives multiply

unavailability only under independence; common causes set an unavailability floor.

## Decision framework

```text

If the operation crosses a process boundary:

classify timeout/cancellation/disconnect as Unknown unless protocol or authoritative

state establishes the relevant effect's outcome.

If progress requires suspecting a peer:

separate suspicion from authority to act; preserve the protocol's safety assumptions.

Tune detection for recovery speed and false suspicion, and state any timing/clock bounds

required for safety, especially with leases.

If replicas share any host, zone, control plane, deploy, credential or dependency:

model that cause once as a common failure domain;

do not multiply replica availability as if independent.

If stale or recovered writers could violate the invariant:

identify how the protected resource rejects their effects, for example through a validated

fencing protocol or atomic conditional write. Verify the actual claim/effect contract;

a client-side check followed by a pause and an unchecked write is not enforcement.

Detailed lease/fencing protocol design belongs to distributed-locks-and-leases.

If the design claims availability during partition:

state which operations remain safe, which side may progress, and what reconciliation

occurs after healing. "The service stays up" is not a consistency contract.

```

## Failure injection and recovery proof

Select fault cases relevant to the card. Before injection, establish an isolated or explicitly

authorized target, observable invariants, blast radius, abort conditions and recovery steps:

- inject loss separately before send, after apply/before acknowledgement, and during

response transfer; assert downstream state and duplicate count;

- pause a process and add latency/jitter rather than testing only clean termination;

- partition asymmetrically (`A` reaches `B`, `B` cannot reach `A`) and isolate data plane

from control plane;

- crash after every durable-write boundary, restart from persisted state, and verify the

safety invariant plus bounded recovery;

- expire credentials, deploy incompatible versions, exhaust pools/disk/file descriptors,

and restore backups into an isolated environment;

- run at realistic load: failover that takes 20 seconds when idle can create hours of

recovery backlog at saturation.

Observe detection latency, false-positive rate, unknown outcomes, duplicate/reconciliation

counts, quorum loss, recovery backlog, RTO and recovered data point (RPO). A test that only

asserts the client exception does not validate the distributed outcome.

## Anti-patterns

| Anti-pattern | Why dangerous / symptom | Better alternative |

| ------------------------------------ | ----------------------------------------------------------------------- | --------------------------------------------------------------------------- |

| One fault model for the whole system | Trust and durability assumptions leak across boundaries | Model each operation and participant role, then compose them |

| Timeout means rollback | Retried writes duplicate after lost acknowledgements | Preserve `Unknown`; use idempotency, status lookup or reconciliation |

| Health check means truth | Gray/asymmetric failures stay green or healthy nodes flap | Compare client-view signals; define detector error costs |

| Replica count means availability | Common deploy, zone or datastore defeats all replicas | Draw domains and measure joint/common-cause failures |

| Failover equals recovery | Traffic moves but stale owners write, data is missing, backlog explodes | Fence old owners; prove state recovery and capacity during catch-up |

| Chaos without invariants | Generates outages but no falsifiable learning | Declare safety/liveness hypotheses, blast radius and abort conditions first |

## References

- [The unknown outcome](references/the-unknown-outcome.md) — the three-outcome model in Java,

how a JDBC, HTTP and Kafka call each maps onto it, and what an unknown write forces the

design to provide. Read when adding a retry, handling a timeout, or writing across a

process boundary.

- [Failure domains and availability arithmetic](references/failure-domains-and-arithmetic.md)

— series and parallel composition worked through, correlated failure, and the questions

that expose a hidden shared dependency. Read when promising an availability number, sizing

replicas, or reviewing a topology.

- [Chubby locks and sequencers](https://static.usenix.org/events/osdi06/tech/full_papers/burrows/burrows_html/)

and [Redis lock assumptions](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/)

— consult when checking stale-owner enforcement and a lease protocol's timing assumptions.