52 added, 18 removed. Audit A to A.
---
name: backpressure-drop-degrade-policy
description: Use when a real-time pipeline's consumers fall persistently behind producers
- and the system needs an explicit, chosen policy rather than an accidental one
+ and the system needs an explicit, chosen drop/sample/degrade policy per stream rather
+ than an accidental one inherited from a queue library default
domain: algorithmic-trading
subdomain: real-time-architecture
tags:
- real-time-architecture
- brokers_frameworks: []
- version: "1.0.0"
+ - backpressure
+ - queue-overflow
+ - drop-policy
+ - degradation
+ - tick-pipeline
+ brokers_frameworks:
+ - Python asyncio.Queue / collections.deque
+ - ZeroMQ
+ - Apache Kafka / Redis Streams
+ - RxPY / ReactiveX
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Invoke this once `tick-buffering-burst-handling` has established bounded buffers — this skill defines what happens when a buffer is persistently at capacity, i.e., sustained backpressure rather than a momentary burst. Every real-time trading system eventually falls behind producers at some point; the difference between a robust system and a fragile one is whether that moment is handled by a deliberate, chosen policy or by whatever the underlying queue/library happens to do by default (which is often "block the producer" — the worst outcome, since it propagates the slowdown back to the WebSocket read loop).
+ Invoke this once `tick-buffering-burst-handling` has established bounded buffers — this skill defines what happens when a buffer is persistently at capacity, i.e., sustained backpressure rather than a momentary burst. Every real-time trading system eventually falls behind producers; the difference between a robust system and a fragile one is whether that moment is handled by a deliberate, chosen policy or by whatever the underlying queue/library happens to do by default (often "block the producer" — the worst outcome, since it propagates the slowdown back to the WebSocket read loop).
+ ## When NOT to Use
+
+ - **For momentary bursts.** A short spike that a bounded buffer absorbs is a buffering problem, not a policy problem — see `tick-buffering-burst-handling`.
+ - **When the consumer is merely slow-to-start.** Fix the consumer or scale it out before deciding what to throw away; a drop policy applied to a fixable bottleneck permanently degrades data quality to paper over a solvable issue.
+ - **As a substitute for capacity planning.** If a stream is *always* at its watermark, the policy is masking undersized infrastructure — see `capacity-planning-for-symbol-universe-growth`.
+ - **For order/execution acknowledgements.** Those are not a droppable telemetry stream; reconciliation, not sampling, is the correct response to falling behind.
+
## Prerequisites
- Bounded buffers already in place (see `tick-buffering-burst-handling`)
- A per-data-type criticality classification: which data can be dropped, sampled, or degraded, and which cannot
+ ## The Caller's Contract
+
+ `handle_full()` returns a `BackpressureDecision`. **Check `decision.accepted`.**
+
+ A `NEVER_DROP` stream at capacity returns `accepted=False` with the item in `decision.rejected_item` — the manager cannot store it for you and will not pretend it did. Treating that return as "handled" silently discards risk-critical data, which is precisely the failure this skill exists to prevent. Wire `on_never_drop_overflow` to your emergency path, or set `strict_never_drop=True` to raise.
+
## Workflow
1. Classify each data stream by what backpressure response is acceptable:
- **Drop-oldest:** for streams where only the latest state matters (e.g., latest LTP for a position-monitoring display) — safe to discard stale ticks in favor of the newest.
- - **Sample/throttle:** for streams feeding non-critical downstream consumers (e.g., a dashboard chart) — reduce update frequency under load rather than dropping entirely.
- - **Degrade to lower-resolution data:** for streams that can fall back to a coarser representation (e.g., switch from tick-level to 1-second OHLC aggregation) under sustained load rather than processing every individual tick.
- - **Never drop:** for streams tied directly to risk decisions (e.g., position/margin updates feeding the kill-switch) — these must never silently degrade; if they cannot keep up, this is a system health emergency requiring an alert, not a quiet policy application.
- 2. Implement the chosen policy explicitly per stream rather than letting a generic queue library's default behavior (often blocking or exception-on-full) decide unintentionally.
- 3. Never let a "never drop" stream share a queue/thread pool with a "safe to drop" stream — resource contention between them means the safe-to-drop stream's load can degrade the never-drop stream's latency. Isolate resources (separate queues, separate worker pools) by criticality tier, mirroring the tier separation in `multi-broker-rate-limit-handling`.
- 4. Emit an explicit alert (not just a log line) when any "never drop" stream approaches its buffer capacity — this is a signal that the system is nearing a state where risk-critical logic cannot keep up with market data, which is materially different from a dashboard being a few seconds stale.
- 5. Record, per backpressure event, which policy fired and what was dropped/degraded/throttled, so post-session review can assess whether the chosen policies were appropriate for what actually happened.
+ - **Sample/throttle:** for streams feeding non-critical downstream consumers (e.g., a dashboard chart) — admit 1 of every N items under load, leaving the existing backlog intact. Throttling admission is not the same as flushing the buffer: discarding half the queue on every overflow destroys far more data than the load actually requires.
+ - **Degrade to lower-resolution data:** for streams that can fall back to a coarser representation (e.g., tick-level to 1-second OHLC) under sustained load.
+ - **Never drop:** for streams tied directly to risk decisions (e.g., position/margin updates feeding the kill-switch). If they cannot keep up, that is a system health emergency requiring an alert and an explicit rejection the caller must handle — not a quiet policy application.
+ 2. Declare **every** stream's policy explicitly. An undeclared stream raises `UnknownStreamError` by default rather than falling back to a drop policy — a mistyped risk-stream name must not silently become a data-loss path.
+ 3. Call `observe()` on every push, not just on overflow. `handle_full()` alone cannot warn you early, because by the time it runs the queue is already full — for a `NEVER_DROP` stream that is already too late.
+ 4. Never let a "never drop" stream share a queue/thread pool with a "safe to drop" stream — resource contention means the safe-to-drop stream's load can degrade the never-drop stream's latency. Isolate by criticality tier, mirroring the tier separation in `multi-broker-rate-limit-handling`.
+ 5. Emit an explicit alert (not just a log line) when any "never drop" stream approaches capacity. Pass a real out-of-band `alert_fn`; the default only writes a log warning.
+ 6. Record, per backpressure event, which policy fired and what was dropped/degraded/throttled, so post-session review can assess whether the chosen policies matched what actually happened.
> Full step-by-step procedure with broker-specific detail: see `references/workflows.md`.
> Broker/framework coverage table for this skill: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - Using a single generic bounded queue for all data types and accepting whatever the library does by default when full (commonly: block the caller, which is the worst option for anything upstream of a WebSocket read loop).
- - Applying a "safe to drop" policy to a risk-relevant stream by not distinguishing it clearly enough at design time — this usually happens when position/margin updates are lumped in with general market data rather than treated as their own tier.
- - Silent degradation with no alerting — a system that quietly downgrades from tick-level to OHLC-level processing during a busy session may be technically "working" while producing materially different (and possibly worse) signals than the strategy was validated against, and nobody notices until performance diverges from backtest.
- - Treating backpressure policy as a one-time architectural decision rather than something to revisit as strategies and data volumes change.
+ - **Ignoring the return value.** If `handle_full()` returns "nothing to do" for both a successful drop-oldest and a rejected risk item, the caller cannot distinguish them and the risk item is lost. Check `accepted`.
+ - **A bounded deque is already an implicit policy.** `collections.deque(maxlen=N)` silently discards from the opposite end when you append to a full deque — Python documents this. If you append directly, you have chosen drop-oldest by accident. Route pushes through the manager.
+ - **Compound queue operations are not atomic.** `deque` guarantees individual appends and pops are thread-safe, but reading `len(queue)` and then popping that many times races with a concurrent consumer and raises `IndexError: pop from an empty deque` — crashing the producer thread, which is usually the WebSocket read loop. Guard every pop.
+ - **Defaulting an unknown stream.** Applying a "safe to drop" policy to a risk-relevant stream, usually because position/margin updates were lumped in with general market data or a stream name was mistyped.
+ - **Flushing the backlog in the name of "sampling."** Discarding a fixed fraction of the queue per overflow event is a different, far more destructive behavior than reducing update frequency.
+ - **Silent degradation with no alerting** — a system that quietly downgrades from tick-level to OHLC-level processing may be technically "working" while producing materially different signals than the strategy was validated against, and nobody notices until live performance diverges from backtest.
+ - **Zero-filling missing tick fields.** Substituting `0.0` for an absent price corrupts every OHLC bar built from it. Reject the tick instead.
+ - **Misleading telemetry.** A drop rate computed against overflow events rather than total pushes reads near 100% under load and tells you nothing; a rate of `0.0` when nothing was observed reads as healthy when it means "no data".
+ - **Treating backpressure policy as a one-time decision** rather than something to revisit as strategies and data volumes change.
## Verification
- - Under sustained simulated overload (replay at multiples of peak historical tick rate), confirm risk-critical streams (position/margin/kill-switch) never drop data and instead trigger the defined alert.
- - Confirm dashboard/non-critical streams degrade gracefully (throttled updates, coarser data) without crashing or blocking the pipeline.
- - Confirm logs show which policy fired for each stream during the overload test, matching the intended design.
+ - Run `python scripts/test_backpressure_policy.py` — 100% pass rate (38 tests).
+ - Under sustained simulated overload (replay at multiples of peak historical tick rate), confirm risk-critical streams return `accepted=False` and trigger the defined alert rather than silently discarding.
+ - Confirm an undeclared stream name raises `UnknownStreamError` instead of being assigned a drop policy.
+ - Confirm the overflow path does not raise when a consumer drains the queue concurrently.
+ - Confirm dashboard/non-critical streams throttle admission while leaving the existing backlog intact.
+ - Confirm `get_metrics_summary()` reports drop rates against observed pushes, and `None` (not `0.0`) when nothing was observed.
## Related Skills
- `producer-consumer-tick-pipeline`
- `tick-buffering-burst-handling`
- `kill-switch-and-drawdown-circuit-breakers`
+ - `graceful-degradation-priority-during-partial-outage`
+ - `capacity-planning-for-symbol-universe-growth`