v1.0.0 to v2.0.0
169 added, 24 removed. Audit A to A.
---
name: circuit-breaker-for-downstream-service-calls
- description: Resilience engineering pattern implementing a Circuit Breaker to prevent
- cascading failures when downstream trading services or data APIs experience degraded
- performance.
- domain: Infrastructure
- subdomain: Reliability
+ description: Use when a trading process calls a downstream service — reference data,
+ historical or alt-data vendors, internal microservices — and a slow or failing
+ dependency must be cut off before it exhausts threads and connection pools.
+ Covers the thread-safe CLOSED/OPEN/HALF_OPEN state machine, single-probe recovery,
+ escalating backoff, slow-call detection, and which exceptions must never trip a
+ breaker.
+ domain: algorithmic-trading
+ subdomain: real-time-architecture
tags:
+ - real-time-architecture
- circuit-breaker
- resilience
- - microservices
- - api
- fail-fast
+ - downstream-dependency
+ - graceful-degradation
+ - thread-safety
brokers_frameworks:
- Generic Infrastructure
- version: "1.0.0"
+ - Python
+ - requests
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Use this skill when your trading system relies on external or downstream microservices (e.g., historical data APIs, alternative data vendors, or non-critical risk checks). If that downstream service becomes slow or unresponsive, a Circuit Breaker immediately fails fast, preventing your main trading threads from blocking, exhausting connection pools, and causing a cascading system-wide outage.
+ Use this skill when a trading process makes **synchronous calls to a downstream service
+ it does not control** — a reference-data or corporate-action API, a historical/alt-data
+ vendor, a sentiment endpoint, an internal risk-annotation microservice — and that service
+ can become slow or unavailable without the trading system being obliged to stop.
+ The failure this prevents is not the failed call. It is the *queue behind it*: N threads
+ blocked on a dependency that takes 30 seconds to time out will hold N connections, N
+ threads and N slots in whatever pool they came from, and the outage propagates into
+ components that never touched the sick service. The breaker converts a slow failure into
+ a fast, explicit one that the caller can handle — cached value, degraded mode, skip.
+
+ `CircuitBreaker` in `scripts/circuit_breaker.py` is the reference implementation:
+ thread-safe, monotonic-clocked, with a single-probe HALF_OPEN, escalating backoff, and
+ optional slow-call detection.
+
+ ## When NOT to Use
+
+ - **On the order path.** Never wrap order submission, order cancellation, or a kill
+ switch. Fast-failing a *cancel* is strictly worse than failing slowly: it turns a slow
+ dependency into an uncancelled live order. For an EU/UK firm in scope of MiFID II RTS 6,
+ Article 12 requires the ability to cancel unexecuted orders immediately as an emergency
+ measure; a client-side breaker in front of that path works against the obligation. See
+ `references/standards.md`.
+ - **In front of mandatory pre-trade risk controls.** A US broker-dealer's SEC Rule
+ 15c3-5 controls must be applied, not skipped because a service was slow. If a required
+ check cannot run, the correct behaviour is to stop trading, not to fail open.
+ - **When the caller has no fallback.** A breaker that raises into code which has no
+ degraded path has only moved the outage earlier. Decide what happens on
+ `CircuitBreakerOpenException` *before* installing the breaker.
+ - **When the failure is deterministic.** HTTP 400, malformed symbol, bad credentials,
+ insufficient funds — these fail identically forever. Opening a circuit on them delays
+ the fix and hides the real error.
+ - **Across independently failing resources.** One breaker over several venues, shards or
+ accounts blocks healthy ones because an unhealthy one failed. One breaker per resource
+ that can fail on its own.
+ - **Instead of a timeout.** The breaker cannot interrupt a call already in flight; see
+ Prerequisites.
+
## Prerequisites
- - A solid understanding of the State Machine pattern (Closed, Open, Half-Open).
- - An asynchronous or multi-threaded trading architecture where non-critical service calls can fail gracefully.
+ - **A client-side timeout on every wrapped call.** This is the hard prerequisite. The
+ breaker only learns of a failure when the call raises; with no timeout the first
+ `failure_threshold` threads block indefinitely and the circuit never opens. In
+ `requests`, that is an explicit `timeout=(connect, read)` — the library has no default.
+ - **A decided fallback** for the open state: last-known-good cached value, degraded
+ feature, skipped enrichment, or a deliberate halt.
+ - **The correct exception tuple.** `expected_exceptions` must name infrastructure faults
+ only. Note that `requests` exceptions do *not* derive from the builtin `TimeoutError` /
+ `ConnectionError`, so the default tuple will not catch them.
+ - **Per-dependency instances**, one per independently failing resource, held for the
+ process lifetime — a breaker constructed per call remembers nothing.
+ - **Somewhere for state changes to go**: a metrics gauge and an alert. A circuit that
+ opens silently is an outage nobody is investigating.
## Workflow
- 1. **Closed State**: The circuit is closed, and requests flow normally. If a request fails (timeout or HTTP 5xx), the failure counter increments.
- 2. **Open State**: If the failure threshold is reached (e.g., 5 consecutive failures), the circuit "trips" to the Open state. Further requests immediately raise a `CircuitBreakerOpenException` without actually hitting the network, saving resources and allowing the downstream service to recover.
- 3. **Half-Open State**: After a configurable timeout (e.g., 30 seconds), the circuit transitions to Half-Open. A limited number of test requests are allowed through.
- - If they succeed, the circuit resets to Closed.
- - If they fail, the circuit returns to Open.
+ 1. **Confirm the call is optional.** If the trading loop cannot proceed without the
+ answer, a breaker converts a slow degradation into a fast one — that may still be the
+ right call, but the correct response to the open circuit is then to halt, not to
+ continue with missing data. Order and cancel paths are out of scope entirely.
+ 2. **Set the client timeout first**, and set it shorter than the loop's tolerance. The
+ breaker is what stops you *making* the call; the timeout is what stops it hanging.
+ 3. **Choose the exception tuple against the actual client library.** With `requests`,
+ pass `(requests.Timeout, requests.ConnectionError)`. Do **not** pass
+ `requests.RequestException` or `OSError`: both subsume `requests.HTTPError`, so a
+ deterministic HTTP 400 raised by `raise_for_status()` would open the circuit.
+ 4. **Size the threshold against the call rate, not against a feeling.** A 3-failure
+ threshold on a service called 500 times a second opens in milliseconds of noise; the
+ same threshold on a once-a-minute call takes three minutes to react. Use
+ `failure_window_sec` when the call is infrequent, so evidence from an hour ago does
+ not combine with today's.
+ 5. **Decide whether slow counts as failed.** For a service whose whole value is
+ timeliness, set `slow_call_duration_sec` below the client timeout. Without it, a
+ dependency that answers correctly but far too late never trips anything.
+ 6. **Wrap the call** with `call()`, `decorate()`, or the `guard()` context manager.
+ Never hold a position-, order- or lock-bearing resource across the wrapped call.
+ 7. **Handle `CircuitBreakerOpenException` distinctly from the dependency's own errors.**
+ The former means *nothing was attempted* — the fallback is safe and no request reached
+ the service. The latter means the request may well have been received. For anything
+ with a side effect, that distinction decides whether a retry is safe.
+ 8. **Do not retry into an open circuit.** If a retry layer sits above the breaker, it
+ must treat `CircuitBreakerOpenException` as terminal for that attempt and back off,
+ not consume its retry budget against a breaker that is refusing by design.
+ 9. **Let recovery be a single probe.** `half_open_max_calls=1` is the default for a
+ reason: a recovering service that gets the entire backlog on the first successful
+ response goes straight back down. Raise `half_open_success_threshold` for a dependency
+ known to flap.
+ 10. **Publish `snapshot()` as a gauge and alert on the OPEN transition**, using
+ `on_state_change`. Include `retry_after_sec` and `total_short_circuits` — the count of
+ calls the breaker refused is the number your business logic silently degraded.
+ 11. **Keep a manual override.** `force_open()` takes a known-bad dependency out of the
+ path without a deployment; `reset()` restores service without waiting out an escalated
+ backoff. Both belong behind the same access control as any other live-trading control.
- > Full procedure: see `references/workflows.md`.
- > Standards reference: see `references/standards.md`.
+ > Full procedure, including tuning and monitoring: see `references/workflows.md`.
+ > Engineering standards and the regulatory boundary: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Wrapping Critical Path Functions**: Do not wrap your primary exchange execution gateway in a circuit breaker if it means silently dropping live orders. Circuit breakers are for services where degradation is preferable to total system failure.
- - **Infinite Half-Open Loops**: If the downstream service flickers, the circuit might thrash between Open and Half-Open. Ensure exponential backoff is implemented for the recovery timeout.
- - **Ignoring Exceptions**: Swallowing the `CircuitBreakerOpenException` without alerting the monitoring system.
+ - **No timeout on the wrapped client.** The single most common way a circuit breaker
+ provides no protection at all. Threads pile up on a hung socket, nothing raises, the
+ circuit stays CLOSED, and the process dies exactly as it would have without it.
+ Microsoft's Azure Architecture Center lists this failure explicitly.
+ - **Catching too much.** `expected_exceptions=(Exception,)` — a tempting default, and the
+ one this skill's implementation used to ship — counts a `ValueError` from your own
+ parsing code as evidence the dependency is down. The breaker then hides your bug behind
+ a fake outage.
+ - **Catching the wrong hierarchy.** `requests.ConnectionError` is **not** a subclass of
+ the builtin `ConnectionError`, and `requests.Timeout` is not a subclass of
+ `TimeoutError`; both derive from `RequestException(OSError)`. A tuple of builtins
+ silently never fires against `requests`. Conversely `OSError` catches everything
+ `requests` raises — including `HTTPError` for a 4xx.
+ - **Counting cumulative rather than consecutive failures.** A counter that never resets on
+ success trips on three unrelated blips spread across a session. Reset on success, and
+ age out stale evidence with `failure_window_sec`.
+ - **Using `time.time()` for the recovery timer.** A wall clock steps. An NTP correction
+ can make the recovery window appear to have elapsed instantly, or to never elapse.
+ Use `time.monotonic()`.
+ - **A HALF_OPEN state with no lock.** Without one, every thread waiting on the open
+ circuit becomes a probe the moment the timer expires, and the recovering service is
+ flooded by exactly the herd the pattern exists to prevent.
+ - **A fixed recovery timeout against a flapping dependency.** The circuit thrashes
+ OPEN → HALF_OPEN → OPEN forever at a constant rate. Escalate the timeout on each
+ re-open, and add jitter when many processes share the same dependency — otherwise the
+ whole fleet probes on the same second.
+ - **Treating a per-process breaker as a global one.** Twenty processes with
+ `half_open_max_calls=1` send twenty probes per window. If the dependency cannot take
+ that, the coordination has to live outside the process.
+ - **Nested breakers that cascade.** If an outer breaker counts the inner breaker's
+ `CircuitBreakerOpenException` as a failure, one sick leaf dependency opens every circuit
+ above it. The reference implementation never counts it, whatever the exception tuple
+ says.
+ - **Swallowing the open-circuit exception.** Degrading silently is how a strategy runs a
+ full session on stale reference data. Log it, count it, alert on the transition.
+ - **Assuming an open circuit means the request never happened *at the venue*.** It means
+ *this* process did not send it. That is only the same thing if nothing else — a retry
+ layer, a sibling process, an earlier attempt — already did.
## Verification
- - Wrap a mock API call that always throws a `TimeoutError`. Call it 5 times to trip the breaker. On the 6th call, verify that a `CircuitBreakerOpenException` is thrown immediately without executing the mock API logic.
- - Run `python scripts/test_circuit_breaker.py`.
+ - Run the unit suite and confirm every test passes:
+ `python -m unittest discover -s skills/circuit-breaker-for-downstream-service-calls/scripts`
+ - Wrap a callable that always raises `ConnectionError`, call it `failure_threshold` times,
+ then assert that the next call raises `CircuitBreakerOpenException` **and that the
+ callable's invocation counter did not increase** — proving no I/O was attempted.
+ - Interleave a success between failures and assert the circuit stays CLOSED; a breaker
+ that opens here is counting cumulatively.
+ - Drive the recovery timeout with an injected clock, not `sleep`. Assert the probe is
+ refused at `t = timeout - ε` and admitted at `t = timeout`.
+ - Fail the probe and assert the next window is `backoff_multiplier` times longer, and that
+ it returns to the base timeout once the circuit closes.
+ - Start a slow probe in one thread and assert a concurrent call is refused with
+ `state == HALF_OPEN` — this is the single-probe guarantee, and it is the property a
+ lock-free implementation silently loses.
+ - Raise a business exception through the breaker and assert `failure_count` is unchanged.
+ - Point the breaker at a real staging endpoint, kill the endpoint, and confirm the alert
+ fires from `on_state_change` and the gauge reflects OPEN in your dashboard — an untested
+ alert path is the normal reason an open circuit goes unnoticed.
## Related Skills
- - `chaos-engineering-for-trading-infrastructure`
+ - `graceful-degradation-to-polling-fallback`
+ - `vendor-outage-fallback-data-source-hierarchy`
+ - `graceful-degradation-priority-during-partial-outage`
- `smart-order-router-failover-on-venue-outage`
+ - `broker-status-page-monitoring-integration`
+ - `multi-broker-rate-limit-handling`
+ - `chaos-engineering-for-trading-infrastructure`
+ - `kill-switch-and-drawdown-circuit-breakers`
+ - `order-placement-idempotency`
+ - `log-aggregation-and-centralized-observability`