46 added, 12 removed. Audit A to A.
---
name: backtest-vs-live-performance-divergence-tracking
description: Use when monitoring promoted strategies to systematically measure, decompose,
and alert on divergence between backtested (hypothetical) performance and realized
live trading performance across Sharpe ratio, drawdown, fill rate, and slippage
- dimensions.
+ dimensions. Periodic snapshot comparison; not real-time monitoring and not an automatic
+ kill switch.
domain: algorithmic-trading
subdomain: backtesting-methodology
tags:
- backtesting-methodology
- backtest-live-divergence
- performance-tracking
- strategy-monitoring
- slippage-drift
- sharpe-decay
brokers_frameworks:
- Divergence Tracking Engine
- Python Statistics
- version: "1.0.0"
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
Invoke this skill after promoting a strategy from backtesting to live trading. Every strategy experiences some divergence between its backtested equity curve and realized live performance. Small divergence ($<20\%$ Sharpe decay) is expected due to execution friction. Large unexplained divergence ($>30\%$ Sharpe decay, or max drawdown $2\times$ backtest worst case) signals model overfitting, regime shift, or execution infrastructure failure. This skill provides a structured framework for tracking, decomposing, and alerting on backtest-vs-live divergence.
+ The decomposition is the point. A strategy whose Sharpe halved because fill rate collapsed and slippage tripled has an execution problem; one whose Sharpe halved with execution metrics intact has an alpha problem. Those need different responses, and the per-metric breakdown separates them.
+
+ ## When NOT to Use
+
+ - **Not real-time monitoring, and not a kill switch.** This compares two periodic snapshots. It cannot detect a runaway algorithm within a session. Firms subject to a real-time monitoring obligation — EU investment firms engaged in algorithmic trading fall under Article 16 of RTS 6 (Commission Delegated Regulation (EU) 2017/589) — do not discharge it with a divergence report. See `mifid-ii-algo-trading-compliance-eu`, `sec-rule-15c3-5-risk-controls-us`, and `kill-switch-and-drawdown-circuit-breakers`. *(Sourcing note: EUR-Lex was not retrievable during review; the article number and title are corroborated from secondary reproductions of RTS 6, not read from the primary text.)*
+ - **Not a cause attribution.** It reports which metrics moved, not why. Slippage amplification and Sharpe decay moving together is consistent with an execution problem *and* with a volatility regime change that widened spreads and hurt the signal simultaneously.
+ - **Not a validated threshold set.** No authoritative source prescribes backtest-vs-live divergence limits. Every default here is an implementation default. Calibrate against your own strategy population before wiring the output to a suspension workflow.
+ - **Not usable on a short live sample.** Comparing a multi-year backtest Sharpe against two weeks of live results measures noise. Supply `observation_periods` and set `min_live_observations` so the report flags it; the flag warns in *both* directions, since a short sample can hide a real problem as easily as invent one.
+ - **Not a substitute for a paired like-for-like backtest.** The backtest snapshot must cover a comparable regime and instrument universe, otherwise the divergence is an artefact of the comparison, not of the strategy.
+
## Prerequisites
- Backtested performance metrics: Sharpe ratio, max drawdown, win rate, avg slippage assumption.
- Live performance metrics over equivalent observation window.
+ - **One drawdown sign convention across both snapshots.** Either $-15.0$ or $15.0$ works; magnitudes are compared. Mixing them raises, because it means the two snapshots came from different sources.
+ - Win rate and fill rate as percentages in $[0, 100]$, not fractions.
+ - Optionally `observation_periods` on each snapshot, to enable the sample-adequacy flag.
## Workflow
1. **Capture Paired Metric Snapshots**:
- Record backtest baseline metrics $M_{\text{bt}}$ and live realized metrics $M_{\text{live}}$ at equivalent time horizons.
- 2. **Compute Divergence Scores**:
- - Sharpe Divergence: $\Delta_{\text{sharpe}} = \frac{S_{\text{bt}} - S_{\text{live}}}{S_{\text{bt}}} \times 100\%$
- - Drawdown Divergence: $\Delta_{\text{dd}} = \frac{DD_{\text{live}} - DD_{\text{bt}}}{DD_{\text{bt}}} \times 100\%$
- - Fill Rate Divergence: $\Delta_{\text{fill}} = \text{FillRate}_{\text{bt}} - \text{FillRate}_{\text{live}}$
+ 2. **Compute Divergence Scores** — five metrics on three different comparison bases. `comparison_value` carries the quantity actually classified and shares the scale of the thresholds; `divergence_pct` is for display only.
+ - Sharpe Decay, relative %: $\Delta_{\text{sharpe}} = \frac{S_{\text{bt}} - S_{\text{live}}}{S_{\text{bt}}} \times 100\%$
+ - Win Rate Decay, relative %: same form on win rate
+ - Drawdown Blow-Up, ratio: $|DD_{\text{live}}| / |DD_{\text{bt}}|$
+ - Slippage Amplification, ratio: $SL_{\text{live}} / SL_{\text{bt}}$
+ - Fill Rate Gap, percentage points: $\Delta_{\text{fill}} = \text{FillRate}_{\text{bt}} - \text{FillRate}_{\text{live}}$
- 3. **Classify Divergence Severity**:
+ 3. **Handle Comparisons That Cannot Be Formed**: A zero or non-positive baseline — a backtest that assumed no slippage, recorded no drawdown, or produced a non-positive Sharpe — admits no ratio or relative decay. Such a metric is escalated to `WARNING` with an explanatory `notes` string, **never** reported as `ACCEPTABLE`. An unassessed dimension is not a benign one.
+
+ 4. **Classify Divergence Severity** — thresholds are **inclusive**, and the classified value is rounded first so the number in the report can never contradict the verdict printed beside it.
- `ACCEPTABLE`: All divergence metrics within tolerance thresholds.
- - `WARNING`: One or more metrics exceed soft threshold (e.g., Sharpe decay $>20\%$).
- - `CRITICAL`: Sharpe decay $>50\%$ or live drawdown $>2\times$ backtest drawdown — triggers strategy suspension review.
+ - `WARNING`: One or more metrics at or beyond the soft threshold (e.g., Sharpe decay $\ge 20\%$), or any comparison that could not be formed.
+ - `CRITICAL`: Sharpe decay $\ge 50\%$, live drawdown $\ge 2\times$ backtest, win rate decay $\ge 25\%$, fill rate gap $\ge 15$ points, or slippage $\ge 4\times$ backtest — triggers strategy suspension review.
+ - Overall severity is the worst individual metric; `driving_metrics` names which ones sit at that level.
- 4. **Generate Divergence Report & Alerts**:
- - Emit structured divergence audit report with per-metric breakdown.
+ 5. **Check Sample Adequacy Before Acting**: If `is_sample_adequate` is False the verdict is noise-dominated. Investigate rather than act — the severity is deliberately not downgraded, because a short sample is not evidence of safety.
+ 6. **Generate Divergence Report & Alerts**:
+ - Emit structured divergence audit report with per-metric breakdown. `DivergenceSeverity` is a string enum, so the report serialises directly to JSON for an alerting pipeline.
+
> Full procedure: see `references/workflows.md`.
> Standards reference: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- **Comparing Mismatched Time Windows**: Comparing 3-year backtest Sharpe against 2-week live Sharpe, inflating noise-driven divergence.
- **Ignoring Survivorship Bias in Backtest**: Backtest includes delisted winners; live portfolio never held them.
- **Attributing All Divergence to Execution**: Assuming all Sharpe decay is slippage when it may be regime-driven alpha decay.
+ - **Mixed Drawdown Sign Conventions**: A tracker guarding on `backtest_drawdown > 0` silently skips the comparison when fed the negative convention that `backtest-reporting-standardized-tearsheet` emits. A live drawdown five times the backtest then reports `ACCEPTABLE`.
+ - **A Zero-Slippage Backtest**: The most common backtest omission is also the one that defeats a ratio-based slippage check. Against a zero baseline there is no amplification factor to compute, and defaulting it to $1\times$ blesses unlimited live execution cost.
+ - **NaN Passing Every Threshold**: `max(0.0, nan)` is `0.0` and `nan >= threshold` is `False`, so an unguarded NaN in any live metric reports no divergence and no suspension. Reject non-finite inputs at the boundary.
+ - **Thresholds That Invert the Ladder**: A warning threshold above its critical counterpart makes mild divergence classify `CRITICAL` and severe divergence `WARNING`. Validate on construction.
+ - **Floating-Point Deciding a Threshold Case**: A Sharpe of $2.0$ decaying to $1.6$ is exactly $20\%$, but computes as $19.999999999999996$. Classifying the unrounded value while displaying the rounded one produces an audit record reading "divergence 20.0, warning threshold 20.0, severity ACCEPTABLE".
+ - **Reading `divergence_pct` Against a Threshold**: For the two ratio metrics the displayed percentage and the threshold are on different scales — 80.0 versus 1.5. Compare `comparison_value`.
## Verification
- Submit paired metrics with 25% Sharpe decay, verify `WARNING` classification.
- Submit paired metrics with 60% Sharpe decay, verify `CRITICAL` classification.
- - Run `python scripts/test_divergence_tracker.py` and confirm 100% pass rate.
+ - Submit the same drawdown pair under both sign conventions and assert identical verdicts; submit mixed conventions and assert it raises.
+ - Submit a zero backtest slippage against 50 bps live and assert the metric is not `ACCEPTABLE`.
+ - Submit a NaN live metric and assert it raises rather than classifying.
+ - Submit exactly 20.0% Sharpe decay and assert `WARNING`, with `divergence_pct` and `comparison_value` both reading 20.0.
+ - Run `python -m unittest discover -s skills/backtest-vs-live-performance-divergence-tracking/scripts` and confirm 100% pass rate.
## Related Skills
- `transaction-cost-analysis-tca-integration`
- `paper-to-live-promotion-checklist`
- `multi-year-regime-coverage-requirement`
+ - `backtest-reporting-standardized-tearsheet`
+ - `kill-switch-and-drawdown-circuit-breakers`
+ - `strategy-performance-decay-detection-vs-market-wide-decay`
---