v1.0.0 to v2.0.0
181 added, 25 removed. Audit A to A.
---
name: chaos-engineering-for-trading-infrastructure
- description: Quantitative infrastructure testing tool that injects controlled network
- latency, packet drops, and process terminations to validate trading system resilience.
- domain: Infrastructure
- subdomain: Reliability
+ description: Use when a trading system's failover, timeout and gap-recovery paths have
+ never actually been exercised — injecting reproducible latency, message loss and
+ simulated process death at an I/O boundary, behind a fail-closed activation gate, and
+ counting what was injected so a green run cannot be mistaken for a resilient one.
+ domain: algorithmic-trading
+ subdomain: deployment-ops
tags:
+ - deployment-ops
- chaos-engineering
- resilience
- - latency
+ - fault-injection
- failover
- - infrastructure
- brokers_frameworks:
- - Generic Infrastructure
- version: "1.0.0"
+ - grey-failure
+ brokers_frameworks: []
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Use this skill to proactively validate the resilience of a trading system. In quantitative trading, network jitter, dropped FIX packets, and unexpected process crashes are inevitable. Chaos engineering involves intentionally injecting these faults in a controlled environment (or carefully in production) to ensure failover mechanisms (like secondary gateways or heartbeat timeouts) trigger correctly before a real outage causes financial loss.
+ Invoke this skill when a trading system has recovery paths that have never been
+ executed under fault: a secondary FIX session that has never taken over, a
+ sequence-gap recovery that has only ever run in a unit test, a heartbeat timeout
+ nobody has watched fire. Network jitter, dropped messages and process death are not
+ hypothetical in production — the question is only whether the first real occurrence
+ is also the first *observed* occurrence.
+ `ChaosInjector` wraps a callable that stands in for a network or IPC boundary (a FIX
+ send, a REST call, a websocket read, a queue publish) and injects three fault classes
+ against it:
+
+ 1. **Added latency and jitter** — the "grey failure" case: the connection stays open
+ and keeps answering, just far too slowly. This is the fault that bypasses TCP
+ disconnect handling entirely and is usually more dangerous than a hard failure.
+ 2. **Message loss** (`ConnectionAbortedError`) — exercises reconnect logic, sequence
+ gap-fill, and the decision about whether an unacknowledged order was actually sent.
+ 3. **Simulated process death** (`SimulatedProcessCrash`) — exercises heartbeat
+ detection, circuit breakers, and whatever is supposed to flatten or halt when a
+ dependency disappears.
+
+ Use it in CI, against an isolated integration environment, as a standing regression
+ suite: the value comes from experiments running on every build, not from a quarterly
+ exercise.
+
+ ## When NOT to Use
+
+ - **Against anything that can reach a live gateway or real capital.** This tool
+ creates faults; it does not contain them. The activation gate is a last-resort
+ backstop, not a substitute for environment isolation. For firms in scope of MiFID II
+ RTS 6, testing that does not affect the production environment is a regulatory
+ requirement, not a preference — see `references/standards.md`.
+ - **Before the recovery path exists.** Chaos engineering *validates* a hypothesis
+ about recovery. If there is no failover, no heartbeat timeout and no kill switch,
+ the experiment will simply confirm that; write the control first.
+ - **For network-layer realism.** This is an application-level wrapper. Kernel
+ buffering, TCP retransmission, half-open sockets, and true network partitions need
+ `tc`/`netem`, a proxy such as toxiproxy, or venue-provided test facilities.
+ - **For sub-millisecond latency work.** `time.sleep()` resolution is OS-dependent
+ (roughly 1-2 ms on Windows, finer on Linux). Latency targets below ~1 ms are not
+ meaningful here; see `colocation-latency-budget-accounting`.
+ - **As a load or capacity test.** Injecting faults is not the same as injecting
+ volume. RTS 6 Article 10 stress testing is a separate exercise — see
+ `load-testing-before-scaling-to-new-instrument-universe`.
+
## Prerequisites
- - A staging or paper-trading environment that perfectly mirrors production infrastructure.
- - High-resolution observability (monitoring) to verify the system's reaction to the injected faults.
+ - An environment that is separated from production and cannot route to a live venue:
+ distinct credentials, distinct endpoints, distinct accounts.
+ - A **written, measurable steady state** to compare against — "the order gateway
+ sustains 100 orders/sec at p99 < 5 ms and zero unacknowledged orders" — captured
+ *before* the experiment starts. Without it the experiment has no verdict.
+ - Observability with resolution finer than the faults being injected. Injecting 100 ms
+ of latency into a system whose metrics are 1-minute averages tells you nothing.
+ - A working kill switch and pre-trade risk layer, independent of the system under
+ test (`kill-switch-and-drawdown-circuit-breakers`).
+ - Somewhere to record the experiment: hypothesis, seed, fault profile, outcome. The
+ seed is what makes a failure debuggable.
+ - An agreed abort condition and the ability to stop the experiment immediately.
## Workflow
- 1. **Baseline Definition**: Define the steady-state of the system (e.g., "Order gateway processes 100 orders/sec with < 5ms latency").
- 2. **Fault Injection**: Use the `ChaosInjector` to simulate a specific failure:
- - **Jitter**: Inject random latency (e.g., 50-200ms) into the simulated network layer to test backpressure handling.
- - **Drop**: Simulate a severed TCP connection to test reconnect logic and sequence number gap-fills.
- - **Crash**: Forcibly terminate a simulated downstream service to test circuit breakers.
- 3. **Observation**: Monitor the system. If it fails to recover (e.g., it hangs indefinitely waiting for a dropped packet), the chaos experiment has surfaced a bug.
+ 1. **Write the steady state and the hypothesis first, as a falsifiable statement.**
+ "If the feed handler stops responding, the trading engine detects the missing
+ heartbeat within 3 s, cancels working orders through the OMS, and enters HALTED."
+ A hypothesis you cannot fail is not an experiment.
+ 2. **Bound the blast radius before configuring the fault.** Confirm the target
+ environment cannot reach a live gateway. Then construct the injector explicitly:
+ `ChaosInjector(config, enabled=True, name="fix-session-a")` in a test harness, or
+ leave `enabled=None` and let CI set `CHAOS_ENGINEERING_ENABLED`. Left alone, the
+ injector is inert and passes calls straight through.
+ 3. **Always set a seed.** `ChaosConfig(..., seed=20260821)`. An unreproducible failure
+ in a chaos run costs more time than the run saved. The injector logs a warning when
+ a probabilistic profile has no seed.
+ 4. **Start with the grey failure, not the crash.** `latency_ms=50, jitter_ms=150`
+ models a 50-200 ms degraded link. A connection that answers slowly bypasses
+ disconnect handling completely, so it reaches code paths a hard kill never touches.
+ 5. **Escalate one fault class at a time.** Latency, then loss, then death. Injecting
+ all three at once produces a failure you cannot attribute.
+ 6. **Set `latency_ms` deliberately relative to the client's own timeout.** A drop
+ raises *after* the configured delay, so a latency above the client timeout is what
+ exercises the timeout branch; a latency below it exercises the slow-but-successful
+ branch. These are different tests with different bugs.
+ 7. **Treat an unacknowledged order as ambiguous, never as unsent.** When a send is
+ dropped, the correct recovery is to query order state by client order ID, not to
+ resubmit — see `order-placement-idempotency`. An experiment that produces a
+ duplicate order has found a real defect, not a harness artefact.
+ 8. **Check `injector.stats` before believing the result.** `faults_injected == 0`
+ means the run proved nothing: a 10% drop rate over 20 calls injects no drop about
+ 12% of the time. Assert on the counters, not just on the absence of exceptions.
+ 9. **Compare against the hypothesis and record the verdict**, including the seed and
+ fault profile, whether it passed or failed. A passing experiment is evidence for
+ the annual review; a failing one is a defect ticket with a reproducer attached.
+ 10. **Automate it into CI and re-run it after every material change** to the recovery
+ path, the broker adapter, or the venue's session configuration.
- > Full procedure: see `references/workflows.md`.
- > Standards reference: see `references/standards.md`.
+ > Full procedure, including environment topology and verdict criteria: see
+ > `references/workflows.md`.
+ > Regulatory touchpoints (EU/UK RTS 6 Articles 7, 10, 14) and engineering standards,
+ > with their jurisdictional limits: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Testing in Production Without Limits**: Injecting a network partition on a live FIX connection during market hours without a "blast radius" constraint, leading to unmanaged open positions.
- - **Ignoring Jitter**: Only testing hard crashes (process death). In trading, a "grey failure" (a connection that stays open but becomes extremely slow) is often more dangerous than a hard crash, as it bypasses standard TCP disconnect handlers.
- - **Manual Execution**: Running chaos tests manually once a quarter. They must be automated in CI/CD pipelines against integration testing environments.
+ - **Running the experiment where it can reach real capital.** The discipline's own
+ canonical guidance ("Chaos strongly prefers to experiment directly on production
+ traffic") was written for consumer web services whose worst case is a failed page
+ load. In trading the worst case is an unmanaged open position, and for firms in
+ scope of RTS 6 the regulation resolves the tension explicitly: tests must not affect
+ the production environment.
+ - **Testing only hard crashes.** Process death is the *easy* failure — the socket
+ closes and every handler notices. The expensive failure is the connection that
+ stays open and responds in 30 seconds, because it defeats disconnect handling and
+ quietly fills queues until something upstream blocks.
+ - **Reading "no exception" as "resilient".** With a 10% drop rate and a short run, the
+ most likely outcome is that nothing was injected at all. Assert on
+ `injector.stats.faults_injected`.
+ - **Re-seeding the process RNG to get determinism.** Calling `random.seed()` to make a
+ chaos run reproducible re-seeds the generator the *system under test* uses for retry
+ backoff and jitter, so every client retries in lockstep and the experiment measures
+ an artefact of its own instrumentation. This injector seeds only generators it owns.
+ - **Simulating a crash with `SystemExit`.** Raised inside a worker thread it is
+ swallowed silently by `threading` — no traceback, no failed test, a green run that
+ injected a crash nobody saw. Reaching the interpreter, it looks like a clean
+ shutdown in the logs. `SimulatedProcessCrash` derives from `BaseException` so it
+ still bypasses `except Exception`, but it is reported and attributable.
+ - **Resubmitting a dropped order.** A dropped send means the outcome is unknown, not
+ that the order was not sent. Retrying without an idempotent client order ID is how a
+ chaos experiment creates a real duplicate position.
+ - **Running chaos experiments during a deployment freeze or a market event.** See
+ `deployment-freeze-windows-around-market-events`.
+ - **Quarterly manual exercises.** An experiment that runs once a quarter validates a
+ system that no longer exists. Automate it.
## Verification
- - Initialize the `ChaosInjector` and wrap a mock network connection. Inject 100ms of latency and a 10% packet drop rate. Verify the downstream consumer experiences the delay and handles the missing packets gracefully.
- - Run `python scripts/test_chaos_injector.py`.
+ Run the unit suite:
+ ```
+ python -m unittest discover -s skills/chaos-engineering-for-trading-infrastructure/scripts
+ ```
+
+ Then confirm the two safety properties by hand:
+
+ ```python
+ from chaos_monkey_trading_simulator import ChaosConfig, ChaosInjector, MockFixClient
+
+ client = MockFixClient()
+
+ # 1. Fail-closed: with CHAOS_ENGINEERING_ENABLED unset, the wrapper is transparent.
+ inert = ChaosInjector(ChaosConfig(latency_ms=100, drop_probability=1.0))
+ assert inert.execute(client.send_order, "ORD-1") == "ACK-ORD-1"
+ assert inert.stats.faults_injected == 0
+
+ # 2. Enabled: 100 ms of latency and a 10% drop rate, reproducibly.
+ chaos = ChaosInjector(
+ ChaosConfig(latency_ms=100, drop_probability=0.10, seed=20260821),
+ enabled=True, name="fix-session-a")
+ delivered = 0
+ for i in range(100):
+ try:
+ chaos.execute(client.send_order, f"ORD-{i}")
+ delivered += 1
+ except ConnectionAbortedError:
+ pass # the consumer's gap-recovery path belongs here
+ print(chaos.stats, delivered)
+ ```
+
+ The run must show a non-zero `drops_injected`, `total_delay_ms` of at least
+ `100 * calls`, and identical results on a second run with the same seed.
+
+ **Migration from v1 (breaking):** a simulated crash now raises `SimulatedProcessCrash`
+ instead of `SystemExit`; injection requires `enabled=True` or
+ `CHAOS_ENGINEERING_ENABLED` in the environment; `ChaosConfig` validates its arguments
+ and no longer touches the global `random` module.
+
## Related Skills
- `circuit-breaker-for-downstream-service-calls`
+ - `kill-switch-and-drawdown-circuit-breakers`
+ - `order-placement-idempotency`
+ - `sequence-number-gap-detection-for-feeds`
+ - `websocket-reconnection-with-state-recovery`
+ - `exchange-gateway-redundancy-and-failover-testing`
+ - `disaster-recovery-runbook-for-full-region-outage`
+ - `graceful-degradation-priority-during-partial-outage`
+ - `position-limit-breach-simulation-fire-drills`
+ - `load-testing-before-scaling-to-new-instrument-universe`
- `feed-handler-canary-deployment`
+ - `deployment-freeze-windows-around-market-events`