37 added, 30 removed. Audit A to A.
---
name: adaptive-sampling-under-extreme-tick-rates
- description: Use during extreme market volatility or flash-crash volume spikes to
- dynamically sample tick streams (1:N systematic sampling), preserving strategy engine
- throughput and cumulative volume/VWAP tracking while preventing memory buffer overflow.
+ description: Use for high-rate trade-tick streams when downstream processing capacity is bounded and systematic 1:N aggregation can preserve traded volume and price-volume notional.
+ The sampler is thread-safe and fail-closed for malformed or out-of-order input, but it is not a quote, order-book, microstructure, or compliance feed replacement.
domain: algorithmic-trading
subdomain: real-time-architecture
tags:
- real-time-architecture
- adaptive-sampling
- flash-crash
- tick-filtering
- volume-preservation
- throughput-protection
brokers_frameworks:
- Adaptive Tick Sampler
- Python Real-Time Engine
- version: "1.0.0"
+ version: "1.1.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Invoke this skill when operating real-time strategy engines during flash-crash events or extreme news volatility (e.g. 100,000+ ticks/second). When tick rates exceed system CPU capacity, queues overflow and drop random ticks across all symbols indiscriminately. An adaptive tick sampler monitors rolling tick frequency $F_t$ per symbol, dynamically engaging systematic sampling ($1:N$ reduction) while aggregating skipped volume to maintain accurate VWAP and total volume metrics.
-
- ## Prerequisites
+ Invoke this skill for a high-rate **trade-tick** stream when a downstream strategy or analytics consumer has a measured processing ceiling and cannot safely keep every input tick. The engine monitors per-symbol arrival rate, emits every `1:N` ticks during overload, and aggregates skipped trade volume and price-volume notional into the emitted sample.
- - Target maximum processing capacity per symbol (e.g., $F_{\text{target}} = 5,000$ ticks/sec).
- - Cumulative volume and VWAP tracking logic.
+ The default target of 5,000 ticks/second is an example capacity parameter, not a universal market-data standard. Calibrate it from measured CPU, queue, latency, and downstream correctness budgets.
- ## Workflow
+ ## When NOT to Use
- 1. **Monitor Rolling Tick Frequency**:
- - Compute 1-second rolling tick arrival frequency $F_t$ per symbol.
+ - Do not use sampled output for order-book reconstruction, quote protection, spread/microstructure signals, queue-position models, latency-sensitive execution, or regulatory/compliance records that require the complete feed.
+ - Do not use it to repair feed gaps, sequence gaps, duplicate messages, or venue recovery state. Detect and reconcile those upstream before sampling.
+ - Do not drop residual aggregates at shutdown, symbol removal, or feed restart; call `flush`, `flush_all`, or `reset_symbol(flush=True)` and persist the result.
+ - Do not treat aggregate VWAP as an OHLC bar or assume it preserves the path, timing, or extrema of skipped trades.
- 2. **Determine Sampling Rate**:
- - $F_t \le F_{\text{target}}$: `PASSTHROUGH` mode ($100\%$ ticks emitted).
- - $F_t > F_{\text{target}}$: Set sampling ratio $k = \lceil F_t / F_{\text{target}} \rceil$ (emit 1 out of every $k$ ticks).
+ ## Prerequisites
- 3. **Aggregate Skipped Tick Metrics**:
- - Accumulate volume $V_{\text{skipped}}$ and price-volume sum $\sum (P \cdot V)$ of skipped ticks.
+ - A trade-tick feed with documented symbol identity, sequence semantics, event timestamps, price units, and volume units.
+ - A measured per-symbol processing target and an explicit policy for overload, feed restart, sequence reset, and symbol lifecycle.
+ - Downstream consumers that understand `SampledTick.aggregated_tick_count`, `sampling_factor`, and synthetic `is_flush` records.
+ - A durable handoff or audit path for emitted samples, validation errors, and flush/recovery events.
+ - Python 3.10+.
- 4. **Emit Sampled Tick with Aggregated Volume**:
- - When emit tick is dispatched, attach $V_{\text{total}} = V_{\text{current}} + V_{\text{skipped}}$ to preserve cumulative volume and VWAP integrity.
+ ## Workflow
- > Full procedure: see `references/workflows.md`.
- > Standards reference: see `references/standards.md`.
- > Printable pre-flight checklist: see `assets/checklist.md`.
+ 1. **Set the capacity contract**: Construct `AdaptiveTickSamplerEngine(target_max_rate_per_sec=...)` with a positive integer target. Keep `enforce_monotonic_sequence=True` unless the feed contract explicitly permits another policy.
+ 2. **Validate feed identity upstream**: Verify symbol, sequence, timestamp, price, and volume semantics. The engine rejects empty symbols, non-finite values, non-positive prices/volumes, duplicate or decreasing sequences, and backwards event timestamps.
+ 3. **Ingest one trade tick at a time**: Call `ingest_tick(...)`. Under the target rate it emits a passthrough aggregate; above the target it computes `k = ceil(rate / target)` with a minimum sampled factor of 2.
+ 4. **Consume explicit metadata**: Use `mode` and `sampling_factor` to identify policy, `aggregated_tick_count` to identify how many raw trades are represented, and `is_flush` to distinguish synthetic residual output.
+ 5. **Handle errors fail-closed**: Route validation exceptions to the feed-integrity path. Do not retry the same malformed or duplicate tick as if it were new data.
+ 6. **Manage lifecycle**: Call `flush(symbol)` for an individual stream, `flush_all()` during shutdown/checkpointing, and `reset_symbol(flush=True)` when a symbol leaves the stream or its sequence domain resets.
+ 7. **Monitor quality**: Track input/output rates, sampling factor, aggregate counts, volume/notional reconciliation, validation failures, flush counts, queue latency, and downstream processing latency.
## Common Pitfalls
- - **Discarding Volume Data**: Dropping skipped ticks without accumulating their volume, corrupting strategy VWAP indicators.
- - **Applying Uniform Sampling Factor Across All Symbols**: Sampling a quiet illiquid stock at the same $1:10$ rate as a hyper-active index component.
- - **Oscillating Sampling Modes**: Lacking hysteresis on tick frequency thresholds, causing rapid mode switching between passthrough and sampled modes.
+ - **Discarding skipped volume**: Emitting only the selected tick loses traded volume and corrupts aggregate VWAP.
+ - **Calling the result a full feed**: A VWAP aggregate preserves volume and notional, not every price path, quote, or order-book event.
+ - **Accepting duplicate or out-of-order data**: Repeated trades double-count volume; backwards event time corrupts rolling-rate windows.
+ - **Using wall-clock flush timestamps**: A flush must remain deterministic in replay; the implementation defaults to the last event timestamp unless an explicit later timestamp is supplied.
+ - **Leaking symbol state**: Long-lived feeds must reset inactive symbols after flushing so per-symbol dictionaries do not grow without bound.
+ - **Sampling before feed recovery**: Sequence gaps and venue recovery require upstream reconciliation; sampling cannot infer missing trades.
## Verification
- - Simulate 20,000 ticks/sec with target capacity 5,000 ticks/sec, verifying $1:4$ sampling reduction.
- - Verify cumulative volume of sampled ticks matches exact total volume of raw input ticks.
- - Run `python scripts/test_tick_sampler.py` and confirm 100% pass rate.
+ Run the focused test suite:
+ ```text
+ python -m unittest discover -s skills/adaptive-sampling-under-extreme-tick-rates/scripts
+ ```
+
+ The tests cover passthrough, overload sampling, volume/notional preservation, zero timestamps, deterministic flushes, sorted `flush_all`, symbol reset, duplicate/out-of-order rejection, malformed tick values, invalid targets, and symbol isolation. Production sign-off additionally requires replaying calibrated bursts and verifying downstream reconciliation against raw input totals.
+
## Related Skills
- `backpressure-drop-degrade-policy`
- `tick-buffering-burst-handling`
- `adaptive-batch-size-tuning-under-load`
- ---
-