Immutable. This exact content is served forever at /api/v1/blob/78d128c84bf99a3f.
---
name: backtest-outlier-and-bad-tick-filtering
description: Use when cleaning historical tick and bar data for backtesting to detect
and filter erroneous price prints, fat-finger quotes, stale prices, and out-of-sequence
ticks before they distort signal generation.
domain: algorithmic-trading
subdomain: backtesting-methodology
tags:
- backtesting-methodology
- bad-tick-filtering
- outlier-detection
- data-cleaning
- median-filter
- price-spikes
brokers_frameworks:
- Outlier Bad Tick Filter Engine
- Python
version: "1.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
Invoke this skill during data ingestion prior to backtesting. Raw exchange tick files frequently contain anomalous data points: fat-finger bad prints ($10.00$ on a $100.00$ stock), test messages, corrupt quotes during opening auction, or zero-price trades. If unfiltered, a single erroneous low print can trigger a false $+900\%$ momentum signal or hit a stop loss in backtest simulation that could never occur in reality.
## Prerequisites
- Raw price time series (ticks or high-frequency bars).
- Rolling window size $W$ (e.g., 21 ticks) and outlier threshold $Z_{\text{max}}$ (e.g., 5.0 MAD deviations).
## Workflow
1. **Compute Rolling Median & Median Absolute Deviation (MAD)**:
$$\text{MAD} = \text{median}(|P_i - \text{median}(P)|)$$
2. **Evaluate Modified Z-Score**:
$$Z_i = \frac{0.6745 \cdot |P_i - \text{median}(P)|}{\text{MAD}}$$
3. **Filter Outliers**: Drop or interpolate price prints where $Z_i > Z_{\text{max}}$ or price change $> \Delta_{\text{max}}\%$ in single tick.
4. **Generate Data Cleanliness Report**: Report total raw ticks, bad ticks purged, and percent dataset sanitized.
> Full procedure: see `references/workflows.md`.
> Standards reference: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- **Using Mean/StdDev Instead of Median/MAD**: Outliers heavily distort mean and standard deviation, masking subsequent bad prints.
- **Over-filtering Real Volatility Spikes**: Setting $Z_{\text{max}}$ too low (e.g. 2.0) and truncating genuine flash crash price moves.
## Verification
- Inject bad tick ($P=10.0$ into $100.0$ series), verify bad tick detection and removal.
- Run `python scripts/test_outlier_filter.py` and confirm 100% pass rate.
## Related Skills
- `data-vendor-cross-validation-for-backtests`
- `backtest-determinism-and-reproducibility`
---