v1.1.0 to v3.0.0
186 added, 33 removed. Audit A to A.
---
name: broker-failover-secondary-account-routing
- description: Institutional-grade circuit breaker and failover router for high-availability
- algorithmic trading systems.
- version: "1.1.0"
+ description: Use when routing live orders to a backup broker account after the primary
+ degrades — classifying which failures are safe to fail over, refusing to re-send an
+ order that may already be working, keeping terminal rejections off the second account,
+ and pinning position-reducing orders to the account that actually holds the position.
domain: algorithmic-trading
- subdomain: general
+ subdomain: broker-integration
tags:
- - trading
- - algo
- - skill
+ - broker-integration
+ - failover
+ - circuit-breaker
+ - high-availability
+ - duplicate-order-prevention
+ - position-integrity
+ - resilience
brokers_frameworks:
- - Python
- - Dataclasses
+ - Binance Spot REST API
+ - RFC 9110 (503 / Retry-After)
+ - SEC Regulation SHO (17 CFR 242.200/203)
+ - SEC Rule 15c3-5
+ - MiFID II RTS 6 (Reg (EU) 2017/589)
+ version: "3.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
# Broker Failover & Secondary Account Routing
- This skill provides a rigorously engineered **Broker Failover Router** leveraging the Circuit Breaker design pattern. It ensures robust continuation of algorithmic order flow by dynamically rerouting to a secondary broker account when primary API connections degrade, experience latency spikes, or return critical HTTP 503/429 errors.
+ ## When to Use
- ## Core Features
- 1. **Circuit Breaker Pattern (CLOSED, OPEN, HALF_OPEN):** Protects primary connections from cascading failures.
- 2. **Concurrency Safe:** Built-in threading locks handle concurrent order streams from multiple alpha models without race conditions.
- 3. **Automated Half-Open Probing:** After a predefined exponential backoff/timeout period, the system probes the primary broker with a single test order.
- 4. **Symbol Translation Mapping:** Normalizes canonical tickers to broker-specific formats (e.g., `AAPL` -> `AAPL STK SMART` vs `AAPL.S`).
+ Invoke this skill when a trading system holds a backup broker relationship and needs
+ order flow to continue after the primary degrades — connection loss, sustained 5xx,
+ throttling, or an outage mid-session. It provides a circuit breaker over two broker
+ legs, with the failover decision gated on *why* the primary failed.
- ## Integration Checklist
- Check the `assets/checklist.md` for a comprehensive onboarding guide to integrate this into your institutional quant infrastructure.
+ A generic circuit breaker guards idempotent reads, where retrying elsewhere is free.
+ This one guards live order submission across two **accounts**, which breaks three of the
+ pattern's usual assumptions:
- ## Folder Structure
- - `scripts/`: Contains the core `failover_router.py` implementation and `test_failover_router.py` unit tests.
- - `references/`: Documentation on institutional routing standards and sequence workflows.
- - `assets/`: Integration checklists.
+ > **A failed call is not a failed order.**
+ > **Not every error is an outage.**
+ > **A probe is a real order.**
- *Note: For production deployment, ensure the timeout values and backoff mechanisms are tuned to the specific exchange latency characteristics.*
+ And one assumption that endpoint failover never has to think about at all: **positions
+ do not net across accounts.** Failing a *closing* order over to the backup account does
+ not flatten anything — it leaves you long in one account and short in the other, with
+ double the gross exposure.
+ ## When NOT to Use
- ## When to Use
+ - **As a retry mechanism.** It routes each order once, to one place. Retrying the same
+ intent is a different problem with different safety rules — see
+ `order-placement-idempotency`.
+ - **As reconciliation.** It *detects* that an outcome is ambiguous and refuses to act;
+ resolving "did the primary take my order?" needs the broker's order-state stream or a
+ status query, which you supply. See `webhook-based-order-fill-notifications`.
+ - **As a risk control.** Pre-trade limits belong above the router and must apply to both
+ accounts. A second broker is a second market-access path, not an exemption —
+ `sec-rule-15c3-5-risk-controls-us`.
+ - **As a kill switch.** Opening the circuit stops *new* flow to the primary; it does not
+ withdraw working orders. That is `kill-switch-and-drawdown-circuit-breakers`.
+ - **For failover between endpoints of the same account** (a regional DR flip, a second
+ gateway). There the position-integrity problem does not exist and the routing is
+ simpler — see `multi-region-failover-for-broker-connectivity` and
+ `smart-order-router-failover-on-venue-outage`.
- Documentation for When to Use.
+ ## Prerequisites
+ - **A funded, entitled, tested secondary account** with the same instruments enabled.
+ A backup you have never placed an order through is not a backup.
+ - **Stable `client_order_id`s per order intent.** This is the key you reconcile on when
+ a call is ambiguous. Note what it does *not* buy you here: a client order id
+ de-duplicates at the broker that issued it, and the secondary broker has never seen
+ it. **Cross-broker duplicates cannot be prevented by the broker** — only by not
+ sending the second order.
+ - **An order-status resolver**, ideally. Without one, every ambiguous primary failure
+ becomes a hard stop for that order. With one, the router can ask "is it working?" and
+ keep going.
+ - **Seeded positions per account.** The router's position map is a cache. Seed it from
+ each broker's own position report at session start, or a REDUCE order will be judged
+ on stale information.
+ - **Symbol mappings for every tradable instrument**, and `strict_symbol_mapping=True` in
+ production.
+ - **Documented business continuity arrangements** where required. EU-authorised
+ investment firms are subject to MiFID II RTS 6 (Reg (EU) 2017/589) Article 14, which
+ requires business continuity arrangements for algorithmic trading systems,
+ proportionate to the business and documented in a durable medium.
- ## Prerequisites
+ ## Workflow
- Documentation for Prerequisites.
+ 1. **Classify the failure before deciding anything.** This is the whole skill. Four
+ classes, three different answers:
+ | Class | Meaning | Failover? |
+ |---|---|---|
+ | `UNAVAILABLE` | Connection refused, DNS failure — the request never left | **Yes**, safe |
+ | `RATE_LIMITED` | 429/418 — not accepted, but back off this leg | **Yes**, and honour `Retry-After` |
+ | `AMBIGUOUS` | Timeout, connection reset, 5xx — it may be working | **No** — resolve first |
+ | `REJECTED` | Bad symbol, insufficient buying power, size limit | **No** — the order is dead |
- ## Workflow
+ Anything unrecognised is `AMBIGUOUS`. The errors are not symmetric: a wrong
+ "ambiguous" costs one status query, a wrong "failed" costs a duplicate order in an
+ account that will never net it off.
- Documentation for Workflow.
+ 2. **Never fail over an ambiguous outcome.** Binance's REST documentation states the
+ general case for 5xx: "It is important to **NOT** treat this as a failure operation;
+ the execution status is **UNKNOWN** and could have been a success." Either resolve it
+ through a status query, or stop and make the caller reconcile. A resolver that itself
+ errors means *still ambiguous* — never read a failed status check as "not there".
+ 3. **Never fail over a terminal rejection.** "Insufficient buying power" is the primary's
+ pre-trade risk control working. Routing it to a second account is shopping for a
+ broker that will say yes. Re-raise it, and do **not** count it against broker health —
+ otherwise the strategy's own bad orders trip the breaker.
- ## Common Pitfalls
+ 4. **Pin reducing orders to the account that holds the position.** Mark every closing or
+ reducing order `PositionEffect.REDUCE`. In an account with no position the identical
+ instruction opens new exposure. In US equities it is also a compliance event: Reg SHO
+ Rule 200(c) deems a person to own securities "only to the extent that he has a net
+ long position", Rule 200(g) permits a "long" mark only where the seller owns the
+ security, and Rule 203(b)(1) requires a locate before a short sale order is accepted.
+ Netting the two accounts requires the independent-trading-unit conditions of Rule
+ 200(f), including a written plan. If the holding account is down, the correct answer
+ is to refuse and escalate — not to open a position somewhere else.
- Documentation for Common Pitfalls.
+ 5. **Bound the half-open probes.** Each probe is capital at risk against a broker you
+ believe is down. Generic implementations permit several concurrent trial calls
+ (resilience4j's `permittedNumberOfCallsInHalfOpenState` defaults to 10); here the
+ default is one, claimed under an explicit permit so concurrent callers cannot all
+ probe at once.
+ 6. **Ignore stale successes.** A slow call that started before the breaker tripped can
+ return after it. Honouring it closes a circuit that other threads opened for good
+ reason. Tag each attempt with the circuit generation and discard successes from an
+ older one.
- ## Verification
+ 7. **Measure the recovery timeout on a monotonic clock.** `time.time()` moves when NTP
+ steps it; a backward jump holds the circuit open indefinitely.
- Documentation for Verification.
+ 8. **Handle "both legs down" as its own outcome.** It is the case that matters most and
+ the one most likely to be untested. The caller needs a typed error naming the order,
+ not whatever the secondary happened to raise.
+ > Full failure-classification tables and phase procedure: see `references/workflows.md`.
+ > Cited standards, Reg SHO analysis, and thresholds: see `references/standards.md`.
+ > Printable go-live checklist: see `assets/checklist.md`.
+ ## Common Pitfalls
+
+ - **Failing over on a timeout.** The primary may have accepted the order. The secondary
+ cannot de-duplicate it, because it has never seen the client order id. You end up with
+ two live orders in two accounts and no automatic way to net them.
+ - **Treating every exception as a clean failure.** `except Exception: route_to_secondary`
+ is the single most damaging line this pattern attracts.
+ - **Failing over a business rejection.** An order the primary refused on buying power or
+ size gets filled at the backup — the failover has quietly become a risk-control bypass.
+ - **Counting rejections toward the breaker.** A strategy sending bad orders then trips
+ the circuit and pushes healthy flow to the backup for no reason.
+ - **Failing over a closing order.** It opens new exposure instead of reducing any, and
+ in US equities produces a sale in an account that owns nothing.
+ - **Assuming positions net across accounts.** They do not, absent a Rule 200(f)
+ aggregation-unit plan. Two accounts flat against each other is *double* gross exposure,
+ double margin, and two settlement obligations.
+ - **Unbounded half-open probing.** Every concurrent caller sends a live order to a broker
+ believed to be down.
+ - **Letting a stale success close the circuit.** One slow call returning late resets the
+ breaker and the failure counter, and flow returns to a broker that is still broken.
+ - **Using the wall clock for the recovery timeout.** An NTP step forward probes early; a
+ step backward never probes at all.
+ - **Silently passing an unmapped symbol through.** The same ticker can denote a different
+ instrument at the other broker. Use strict mapping in production.
+ - **Leaving the secondary leg untested.** A backup that has never taken an order is an
+ assumption, not a control. Exercise it on a schedule.
+ - **Forgetting the backup has its own limits.** Different rate limits, different order
+ types, different margin treatment, different pattern-day-trade and margin accounting
+ per account. Flow that fits the primary may not fit the backup.
+
+ ## Verification
+
+ - Run `python -m unittest discover -s skills/broker-failover-secondary-account-routing/scripts`
+ and confirm all tests pass.
+ - Make the primary accept an order and then raise `TimeoutError`. Confirm the router
+ raises `AmbiguousOrderStateError` carrying the `client_order_id`, and that the
+ secondary received **nothing**.
+ - Supply an `order_status_resolver` that finds the order; confirm the router returns the
+ real primary fill and still does not touch the secondary. Then make the resolver raise
+ and confirm the outcome stays ambiguous rather than becoming "not found".
+ - Make the primary raise a `REJECTED` `BrokerError`; confirm it is re-raised, the
+ secondary is untouched, and the circuit stays `CLOSED` with `primary_failures == 0`.
+ - Make the primary raise `UNAVAILABLE`; confirm the order reaches the secondary with the
+ secondary's symbol mapping applied.
+ - Fail both legs; confirm `AllBrokersUnavailableError` names the order.
+ - Seed a long position in the primary, take the primary down, and submit a
+ `PositionEffect.REDUCE` sell. Confirm `PositionAffinityError` and that no short was
+ opened at the secondary.
+ - Trip the circuit, wait past the recovery timeout, then fire ten concurrent orders and
+ confirm exactly **one** reached the primary.
+ - Start a slow successful call, trip the circuit from other threads while it is in
+ flight, then let it return. Confirm the circuit is still `OPEN`.
+ - Patch `time.time` to a fixed value and confirm recovery still happens on schedule.
+ - Call `manual_open`, wait past the recovery timeout, and confirm the primary is still
+ not probed until `manual_reset`.
+ - With `strict_symbol_mapping=True`, submit an unmapped symbol and confirm
+ `SymbolMappingError` — not an ambiguous outcome, and no failover.
+
## Related Skills
- Documentation for Related Skills.
+ - `order-placement-idempotency`
+ - `broker-api-idempotent-cancel-requests`
+ - `broker-agnostic-adapter-interface`
+ - `multi-region-failover-for-broker-connectivity`
+ - `smart-order-router-failover-on-venue-outage`
+ - `multi-broker-consolidated-position-view`
+ - `multi-broker-rate-limit-handling`
+ - `kill-switch-and-drawdown-circuit-breakers`
+ - `webhook-based-order-fill-notifications`
+ - `us-reg-sho-short-sale-locate-requirements`
+ - `sec-rule-15c3-5-risk-controls-us`
+ - `counterparty-and-broker-concentration-risk`