distributed-systems-testing git:20260906.e6ad4bdA

Immutable. This exact content never changes and is served at /api/v1/blob/17f52d1c4781eb66.

---
name: distributed-systems-testing
description: >
  Testing the failure behaviour a distributed system claims: injecting latency, errors,
  partitions and process death; verifying that timeouts, retries, breakers and fallbacks do
  what their configuration says; proving idempotency against duplicate delivery; and running
  a controlled experiment in production rather than a chaos tool. Use when resilience
  configuration exists but has never been exercised, when a timeout or retry budget is being
  chosen, when an incident was caused by a dependency being slow rather than down, when a
  consumer is assumed idempotent, when a rollout is protected by a probe nobody has failed on
  purpose, or when chaos engineering is proposed without a hypothesis. Does not cover the
  in-process test pyramid and architecture rules (architecture-testing), thread-level race
  testing (concurrency-testing), throughput and saturation measurement (load-testing), or the
  remedies themselves (retries-and-backoff, circuit-breakers, timeouts-and-deadlines).
---

# Distributed Systems Testing

## Purpose

Make the system's failure behaviour something that has been observed rather than configured.
Timeouts, retry budgets, circuit breakers, fallbacks, idempotency keys and readiness probes
are all claims; until each has been exercised against the failure it exists for, the system's
resilience is a set of YAML values that have never executed.

The gap this closes is specific. Functional tests exercise the happy path against a fast,
available dependency. Load tests exercise a healthy system at volume. Neither produces the
condition that actually causes outages: **a dependency that is slow rather than down**, a
duplicate delivered after a broker reconnect, a node that vanishes mid-transaction.

The two failures this exists to prevent: resilience settings that provably do nothing —
a retry budget exhausted by its first attempt, a breaker that never records the failures it
was meant to count; and chaos experiments run without a
hypothesis or a blast-radius limit, which produce an incident rather than a finding.

## Workflow

1. **Write the claim down first.** "A payment gateway timing out returns 503 within 2 s and
   does not double-charge." An untestable claim is a configuration you do not understand yet.
2. **Pick the cheapest level that can falsify it.** Most claims fall at the component level
   with one faulty dependency; very few need a whole environment.
3. **Select faults from the dependency contract and incident evidence.** Include slow,
   unavailable, duplicated, lost and partial outcomes where relevant; do not assume a
   universal frequency ranking or that connection refusal covers a blackhole.
4. **Assert the observable outcome**, not the mechanism: the status code, the elapsed time,
   the number of times the downstream was called, the number of rows written. Asserting that
   a breaker library was invoked tests the library.
5. **Assert the budget, not just the behaviour.** Retry counts and timeouts compose across
   hops; the property that matters is the total, and it is where retry storms come from.
6. **Promote to production only with a hypothesis and a limit** — expected outcome, blast
   radius, abort condition, and a way to stop.

## The failure taxonomy to test against

Use this coverage menu according to the workload and failure model, not as a frequency ranking:

```text
SLOW              Dependency responds, eventually. Threads/connections
                  pile up behind it. Tests: does the timeout fire, is the
                  pool bounded, does the caller shed rather than queue?
                  (cascading-failures).

DUPLICATED        The same message or request arrives twice. Tests: is
                  the effect applied once (idempotency, delivery-semantics)?

PARTIAL           One call in a fan-out fails; one write of two succeeds.
                  Tests: is the outcome consistent, is compensation
                  triggered (distributed-transactions-and-sagas)?

REORDERED         Messages arrive out of order across partitions.
                  Tests: does the consumer tolerate it, or silently
                  corrupt (message-ordering-and-partitioning)?

ERRORING          5xx, connection reset, malformed body. Tests: is the
                  classification right — retryable vs permanent?

DOWN              Connection refusal, unavailable endpoints or silent drops.
                  Distinguish fast errors from timeout-driven detection.

PARTITIONED       Both sides alive, cannot see each other. Tests: split
                  brain, duplicate leaders, lock expiry
                  (distributed-locks-and-leases, leader-election).

DEAD MID-FLIGHT   Process dies between the write and the acknowledgement.
                  Tests: is the work lost, duplicated, or recovered?
```

## Decision rules

```text
The claim is about how a response is classified or a policy decides
        → unit test the pure policy. No network needed, and every edge
          case is a one-line test (humble-objects-and-functional-core).

The claim is about the client's behaviour — timeout fires, retry count,
connection released
        → component test against a stub server that can delay, reset and
          return errors. This is the highest-value level and where most
          resilience claims belong.

The claim is about consumer idempotency
        → deliver the same message twice in a test and assert the effect
          once. This is cheap and almost never done.

The claim is about behaviour under a slow dependency at load
        → load test with latency injected into the dependency. Neither a
          plain load test nor a plain fault test finds this
          (load-testing, littles-law-and-queueing).

The claim is about the system surviving a node or pod dying
        → kill it in a real environment. No stub reproduces the
          combination of in-flight work, connection draining and probe
          timing (kubernetes-service-lifecycle).

The claim is about a partition between two stateful components
        → a network-level fault injector between real instances.
          Application-level stubs cannot produce a partition.

The proposal is "let us run chaos experiments"
        → require the hypothesis, the steady-state metric, the blast
          radius and the abort condition first. Without those it is an
          outage with better branding.

The system has no monitoring for the failure being injected
        → fix the observability first. An experiment you cannot observe
          produces no finding (slo-and-alerting, metrics-and-cardinality).
```

## Rules

- **Test slow before down.** A dependency returning in 30 s exhausts the caller's threads and
  connections and takes down healthy services; a dependency refusing connections fails fast
  and is usually survived. Every timeout in the system deserves one test that it actually
  fires.
- **Assert timing, not only outcome.** "Returns an error" passes whether the timeout fired at
  2 s or at 60 s. The elapsed time is the assertion that matters.
- **Do not mock the dependency you are testing the failure of.** A mocked client returns the
  exception you told it to and proves nothing about connection handling, pool exhaustion or
  socket timeouts. Use a stub server that can genuinely hang and reset
  (`architecture-testing`).
- **Retry and timeout budgets compose across hops and must be tested end to end.** Three
  layers making three total attempts each permit up to twenty-seven deepest calls; three
  retries plus the initial attempt at each layer permit sixty-four. This multiplication is
  the mechanism of most retry storms, and it is invisible in any single service's tests
  (`retries-and-backoff`, `cascading-failures`).
- A circuit breaker's history usually spans multiple logical calls. Check its scope, window,
  minimum sample count, recorded outcomes and timeout/retry ordering. A caller's shorter
  deadline does not imply that the shared breaker can never open
  (`circuit-breakers`).
- **Idempotency is a claim about duplicates, so test with duplicates.** Send the same request
  or message twice, concurrently as well as sequentially, and assert one effect. Concurrent
  duplicates find the missing unique constraint that sequential ones miss (`idempotency`).
- Kill the process at the awkward moment — between the database write and the acknowledgement,
  between two writes, mid-batch. This is where at-least-once semantics stop being theoretical
  and where the outbox either works or does not.
- **Fault injection needs a seam.** A gateway behind an interface, a proxy, or a service mesh
  can be made to fail; a static call buried in business logic cannot. Testability of failure
  is an argument for the adapter boundary, independent of portability
  (`framework-coupling-and-independence`).
- Use controlled fault points and bounded execution for regression tests. Seeded property
  tests and deterministic simulations can run in CI; retain seeds, traces and failing inputs.
  A seed alone does not reproduce uncontrolled network or thread scheduling.
- Run experiments in production only with a hypothesis, a steady-state metric, a bounded blast
  radius and an abort condition — and only where the failure is already observable. Anything
  else is not an experiment.
- **Every finding becomes a regression test at the cheapest level that reproduces it.** The
  value of an experiment is the test it leaves behind, not the incident it simulated.

## References

Before implementing Java tests, inspect compiler/runtime and resolved test, client and resilience
libraries. The virtual-thread example requires Java 21; preserve lower targets using their
existing executors. No upgrade or dependency change is implied. Examples are partial fixture
shapes, not standalone suites. A passing run establishes only the exercised fault/interleaving.

- [Failure scenario audit](references/failure-scenarios.md) — read when checking missing coverage
  or defining the invariant and an assertion that would otherwise hide failure.
- [Techniques and controlled time](references/techniques.md) — read when selecting infrastructure,
  implementing concurrent duplicate tests, or controlling local time.

- [Injecting failure in a Java system](references/fault-injection.md) — the tooling ladder
  from a stub server through a TCP-level proxy to mesh and node-level faults; what each can
  and cannot produce; concrete test shapes for timeout, retry, breaker, duplicate delivery and
  mid-flight death; and asserting budgets across hops. Read when writing a specific failure
  test.
- [Experiments in a real environment](references/chaos-experiments.md) — turning a resilience
  claim into a hypothesis with a steady-state metric, choosing blast radius and abort
  conditions, the readiness checklist a system must pass before an experiment is worth running,
  game days, and what to do with a finding. Read before proposing or running chaos engineering.