Immutable. This exact content is served forever at /api/v1/blob/eec561c65f9b0816.
---
name: backtest-vs-live-performance-divergence-tracking
description: Use when monitoring promoted strategies to systematically measure, decompose,
and alert on divergence between backtested (hypothetical) performance and realized
live trading performance across Sharpe ratio, drawdown, fill rate, and slippage
dimensions. Periodic snapshot comparison; not real-time monitoring and not an automatic
kill switch.
domain: algorithmic-trading
subdomain: backtesting-methodology
tags:
- backtesting-methodology
- backtest-live-divergence
- performance-tracking
- strategy-monitoring
- slippage-drift
- sharpe-decay
brokers_frameworks:
- Divergence Tracking Engine
- Python Statistics
version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
Invoke this skill after promoting a strategy from backtesting to live trading. Every strategy experiences some divergence between its backtested equity curve and realized live performance. Small divergence ($<20\%$ Sharpe decay) is expected due to execution friction. Large unexplained divergence ($>30\%$ Sharpe decay, or max drawdown $2\times$ backtest worst case) signals model overfitting, regime shift, or execution infrastructure failure. This skill provides a structured framework for tracking, decomposing, and alerting on backtest-vs-live divergence.
The decomposition is the point. A strategy whose Sharpe halved because fill rate collapsed and slippage tripled has an execution problem; one whose Sharpe halved with execution metrics intact has an alpha problem. Those need different responses, and the per-metric breakdown separates them.
## When NOT to Use
- **Not real-time monitoring, and not a kill switch.** This compares two periodic snapshots. It cannot detect a runaway algorithm within a session. Firms subject to a real-time monitoring obligation — EU investment firms engaged in algorithmic trading fall under Article 16 of RTS 6 (Commission Delegated Regulation (EU) 2017/589) — do not discharge it with a divergence report. See `mifid-ii-algo-trading-compliance-eu`, `sec-rule-15c3-5-risk-controls-us`, and `kill-switch-and-drawdown-circuit-breakers`. *(Sourcing note: EUR-Lex was not retrievable during review; the article number and title are corroborated from secondary reproductions of RTS 6, not read from the primary text.)*
- **Not a cause attribution.** It reports which metrics moved, not why. Slippage amplification and Sharpe decay moving together is consistent with an execution problem *and* with a volatility regime change that widened spreads and hurt the signal simultaneously.
- **Not a validated threshold set.** No authoritative source prescribes backtest-vs-live divergence limits. Every default here is an implementation default. Calibrate against your own strategy population before wiring the output to a suspension workflow.
- **Not usable on a short live sample.** Comparing a multi-year backtest Sharpe against two weeks of live results measures noise. Supply `observation_periods` and set `min_live_observations` so the report flags it; the flag warns in *both* directions, since a short sample can hide a real problem as easily as invent one.
- **Not a substitute for a paired like-for-like backtest.** The backtest snapshot must cover a comparable regime and instrument universe, otherwise the divergence is an artefact of the comparison, not of the strategy.
## Prerequisites
- Backtested performance metrics: Sharpe ratio, max drawdown, win rate, avg slippage assumption.
- Live performance metrics over equivalent observation window.
- **One drawdown sign convention across both snapshots.** Either $-15.0$ or $15.0$ works; magnitudes are compared. Mixing them raises, because it means the two snapshots came from different sources.
- Win rate and fill rate as percentages in $[0, 100]$, not fractions.
- Optionally `observation_periods` on each snapshot, to enable the sample-adequacy flag.
## Workflow
1. **Capture Paired Metric Snapshots**:
- Record backtest baseline metrics $M_{\text{bt}}$ and live realized metrics $M_{\text{live}}$ at equivalent time horizons.
2. **Compute Divergence Scores** — five metrics on three different comparison bases. `comparison_value` carries the quantity actually classified and shares the scale of the thresholds; `divergence_pct` is for display only.
- Sharpe Decay, relative %: $\Delta_{\text{sharpe}} = \frac{S_{\text{bt}} - S_{\text{live}}}{S_{\text{bt}}} \times 100\%$
- Win Rate Decay, relative %: same form on win rate
- Drawdown Blow-Up, ratio: $|DD_{\text{live}}| / |DD_{\text{bt}}|$
- Slippage Amplification, ratio: $SL_{\text{live}} / SL_{\text{bt}}$
- Fill Rate Gap, percentage points: $\Delta_{\text{fill}} = \text{FillRate}_{\text{bt}} - \text{FillRate}_{\text{live}}$
3. **Handle Comparisons That Cannot Be Formed**: A zero or non-positive baseline — a backtest that assumed no slippage, recorded no drawdown, or produced a non-positive Sharpe — admits no ratio or relative decay. Such a metric is escalated to `WARNING` with an explanatory `notes` string, **never** reported as `ACCEPTABLE`. An unassessed dimension is not a benign one.
4. **Classify Divergence Severity** — thresholds are **inclusive**, and the classified value is rounded first so the number in the report can never contradict the verdict printed beside it.
- `ACCEPTABLE`: All divergence metrics within tolerance thresholds.
- `WARNING`: One or more metrics at or beyond the soft threshold (e.g., Sharpe decay $\ge 20\%$), or any comparison that could not be formed.
- `CRITICAL`: Sharpe decay $\ge 50\%$, live drawdown $\ge 2\times$ backtest, win rate decay $\ge 25\%$, fill rate gap $\ge 15$ points, or slippage $\ge 4\times$ backtest — triggers strategy suspension review.
- Overall severity is the worst individual metric; `driving_metrics` names which ones sit at that level.
5. **Check Sample Adequacy Before Acting**: If `is_sample_adequate` is False the verdict is noise-dominated. Investigate rather than act — the severity is deliberately not downgraded, because a short sample is not evidence of safety.
6. **Generate Divergence Report & Alerts**:
- Emit structured divergence audit report with per-metric breakdown. `DivergenceSeverity` is a string enum, so the report serialises directly to JSON for an alerting pipeline.
> Full procedure: see `references/workflows.md`.
> Standards reference: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- **Comparing Mismatched Time Windows**: Comparing 3-year backtest Sharpe against 2-week live Sharpe, inflating noise-driven divergence.
- **Ignoring Survivorship Bias in Backtest**: Backtest includes delisted winners; live portfolio never held them.
- **Attributing All Divergence to Execution**: Assuming all Sharpe decay is slippage when it may be regime-driven alpha decay.
- **Mixed Drawdown Sign Conventions**: A tracker guarding on `backtest_drawdown > 0` silently skips the comparison when fed the negative convention that `backtest-reporting-standardized-tearsheet` emits. A live drawdown five times the backtest then reports `ACCEPTABLE`.
- **A Zero-Slippage Backtest**: The most common backtest omission is also the one that defeats a ratio-based slippage check. Against a zero baseline there is no amplification factor to compute, and defaulting it to $1\times$ blesses unlimited live execution cost.
- **NaN Passing Every Threshold**: `max(0.0, nan)` is `0.0` and `nan >= threshold` is `False`, so an unguarded NaN in any live metric reports no divergence and no suspension. Reject non-finite inputs at the boundary.
- **Thresholds That Invert the Ladder**: A warning threshold above its critical counterpart makes mild divergence classify `CRITICAL` and severe divergence `WARNING`. Validate on construction.
- **Floating-Point Deciding a Threshold Case**: A Sharpe of $2.0$ decaying to $1.6$ is exactly $20\%$, but computes as $19.999999999999996$. Classifying the unrounded value while displaying the rounded one produces an audit record reading "divergence 20.0, warning threshold 20.0, severity ACCEPTABLE".
- **Reading `divergence_pct` Against a Threshold**: For the two ratio metrics the displayed percentage and the threshold are on different scales — 80.0 versus 1.5. Compare `comparison_value`.
## Verification
- Submit paired metrics with 25% Sharpe decay, verify `WARNING` classification.
- Submit paired metrics with 60% Sharpe decay, verify `CRITICAL` classification.
- Submit the same drawdown pair under both sign conventions and assert identical verdicts; submit mixed conventions and assert it raises.
- Submit a zero backtest slippage against 50 bps live and assert the metric is not `ACCEPTABLE`.
- Submit a NaN live metric and assert it raises rather than classifying.
- Submit exactly 20.0% Sharpe decay and assert `WARNING`, with `divergence_pct` and `comparison_value` both reading 20.0.
- Run `python -m unittest discover -s skills/backtest-vs-live-performance-divergence-tracking/scripts` and confirm 100% pass rate.
## Related Skills
- `transaction-cost-analysis-tca-integration`
- `paper-to-live-promotion-checklist`
- `multi-year-regime-coverage-requirement`
- `backtest-reporting-standardized-tearsheet`
- `kill-switch-and-drawdown-circuit-breakers`
- `strategy-performance-decay-detection-vs-market-wide-decay`
---