14 added, 5 removed. Audit A to A.
---
name: backtest-outlier-and-bad-tick-filtering
- description: >-
- Use when cleaning historical tick and bar data for backtesting to detect and filter erroneous price prints, fat-finger quotes, stale prices, and out-of-sequence ticks before they distort signal generation.
+ description: Use when cleaning historical tick and bar data for backtesting to detect
+ and filter erroneous price prints, fat-finger quotes, stale prices, and out-of-sequence
+ ticks before they distort signal generation.
domain: algorithmic-trading
subdomain: backtesting-methodology
- tags: ["backtesting-methodology", "bad-tick-filtering", "outlier-detection", "data-cleaning", "median-filter", "price-spikes"]
- brokers_frameworks: ["Outlier Bad Tick Filter Engine", "Python"]
- version: "1.0"
+ tags:
+ - backtesting-methodology
+ - bad-tick-filtering
+ - outlier-detection
+ - data-cleaning
+ - median-filter
+ - price-spikes
+ brokers_frameworks:
+ - Outlier Bad Tick Filter Engine
+ - Python
+ version: '1.0'
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
Invoke this skill during data ingestion prior to backtesting. Raw exchange tick files frequently contain anomalous data points: fat-finger bad prints ($10.00$ on a $100.00$ stock), test messages, corrupt quotes during opening auction, or zero-price trades. If unfiltered, a single erroneous low print can trigger a false $+900\%$ momentum signal or hit a stop loss in backtest simulation that could never occur in reality.
## Prerequisites
- Raw price time series (ticks or high-frequency bars).
- Rolling window size $W$ (e.g., 21 ticks) and outlier threshold $Z_{\text{max}}$ (e.g., 5.0 MAD deviations).
## Workflow
1. **Compute Rolling Median & Median Absolute Deviation (MAD)**:
$$\text{MAD} = \text{median}(|P_i - \text{median}(P)|)$$
2. **Evaluate Modified Z-Score**:
$$Z_i = \frac{0.6745 \cdot |P_i - \text{median}(P)|}{\text{MAD}}$$
3. **Filter Outliers**: Drop or interpolate price prints where $Z_i > Z_{\text{max}}$ or price change $> \Delta_{\text{max}}\%$ in single tick.
4. **Generate Data Cleanliness Report**: Report total raw ticks, bad ticks purged, and percent dataset sanitized.
> Full procedure: see `references/workflows.md`.
> Standards reference: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- **Using Mean/StdDev Instead of Median/MAD**: Outliers heavily distort mean and standard deviation, masking subsequent bad prints.
- **Over-filtering Real Volatility Spikes**: Setting $Z_{\text{max}}$ too low (e.g. 2.0) and truncating genuine flash crash price moves.
## Verification
- Inject bad tick ($P=10.0$ into $100.0$ series), verify bad tick detection and removal.
- Run `python scripts/test_outlier_filter.py` and confirm 100% pass rate.
## Related Skills
- `data-vendor-cross-validation-for-backtests`
- `backtest-determinism-and-reproducibility`
---