195 added, 19 removed. Audit A to A.
---
name: adversarial-robustness-of-trading-signals
- description: Tests financial machine learning models for adversarial vulnerability
- by injecting epsilon-bounded perturbations (simulating flash-crash noise or FGSM
- attacks) and measuring signal flip rates.
+ description: Pre-deployment governance gate that measures how often an ML trading
+ signal can be flipped by epsilon-bounded feature perturbations (FGSM-lite / market
+ microstructure noise). Deterministic, domain-clipped, multi-noise-model robustness
+ tester producing a vulnerability score and flipped-sample indices. Rejects models
+ whose signal flips faster than the configured tolerance before they go live.
domain: algorithmic-trading
subdomain: financial-ml-robustness
tags:
- ml
- trading
- adversarial-robustness
- fgsm
- signal-processing
+ - model-governance
+ - pre-deployment-gate
brokers_frameworks:
- scikit-learn
- numpy
- version: "1.1.0"
- author: System
- license: MIT
+ jurisdictions: [global] # technique is jurisdiction-agnostic
+ version: "1.2.0"
+ author: algo-trading-skills-contributors
+ license: Apache-2.0
---
## When to Use
- Invoke this skill before deploying any machine learning model (e.g., Random Forests, Gradient Boosted Trees, or Deep Neural Networks) to production trading. Financial markets are highly noisy, and an attacker (or a market flash crash) can introduce imperceptible noise to order book features that forces a model to flip from a "BUY" to a "SELL" signal. This skill measures the model's Adversarial Vulnerability Score.
+ Invoke this skill **before promoting any ML trading-signal model to production**
+ — Random Forests, Gradient Boosted Trees, deep nets, or any model exposing a
+ `predict(X)` callable. Financial microstructure is adversarial in the technical
+ sense: a flash crash, a spoofing layer, or genuine quote noise can perturb order-
+ book features by less than one bid-ask spread and flip a model's signal from BUY
+ to SELL. A model that flips under such noise is unsafe to trade.
+ The skill produces a `SignalAdversarialTester` that injects epsilon-bounded
+ perturbations into the validation feature matrix and reports:
+
+ 1. The **vulnerability score** — `% of samples whose signal flips`.
+ 2. The **flipped-sample indices** — so quant research can attribute *which*
+ inputs sit on the decision boundary.
+ 3. A **deployment verdict** — robust / vulnerable against a tolerance threshold.
+
+ ## When NOT to Use
+
+ - **Differentiable models requiring a *true* worst-case bound.** This engine is a
+ black-box, gradient-free tester. For a differentiable net, real FGSM/PGD gives a
+ tighter (and actually worst-case) attack: `adv_x = x + ε·sign(∇ₓ L)`. Use a
+ dedicated adversarial-robustness library (CleverHans, Foolbox, ART) instead, then
+ feed that attack's flip rate into this governance gate if you want a uniform
+ scoring contract.
+ - **Provable-robustness requirements.** This is *empirical* robustness — it can
+ only show a model is *vulnerable*, never prove it is *robust*. For a certified
+ L2 radius use Cohen-style randomized smoothing (see `references/standards.md` §3).
+ - **Non-ML signals** (pure rule-based / threshold strategies with no learned
+ component). Adversarial perturbation of a fixed threshold is just sensitivity
+ analysis; use `backtest-parameter-sensitivity-analysis` instead.
+ - **Production-time / online monitoring.** The perturbation sweep is O(n_trials)
+ expensive and assumes a static validation set. For live drift monitoring use
+ `concept-drift-vs-staleness-differentiation` / `model-staleness-detection`.
+ - **Insufficient validation data.** With < ~1000 out-of-sample samples the flip
+ rate confidence interval is wide; a "pass" near the tolerance is statistically
+ meaningless. Bootstrap or use a larger holdout.
+
## Prerequisites
- - Python 3.9+
- - `numpy`
- - A trained model with a `predict(X)` method.
+ - Python 3.9+, `numpy`.
+ - A **trained model** exposing `predict(X)` returning one of:
+ - integer/boolean 1D labels (used directly as the signal),
+ - a 1D float score (split at `decision_threshold`, default 0.5), or
+ - a 2D probability matrix (signal = `argmax` over columns).
+ - A **representative out-of-sample validation set** `X_clean` (2D array). Never
+ test on training data — the model has seen it and will appear artificially robust.
+ - A **calibrated epsilon** — the perturbation budget. Default 0.01 (1% of
+ per-feature scale); for equities, set this to one average bid-ask spread as a
+ fraction of price (see `references/standards.md` §1).
+ - (Recommended) **Training-set feature scales** — pass `feature_scales` so the
+ test is independent of the validation sample drawn (the ptp fallback couples ε
+ to the sample).
## Workflow
- 1. **Baseline Evaluation**: Compute trading signals on clean historical feature data $X$.
- 2. **Adversarial Perturbation**: Inject bounded adversarial noise $\epsilon$ (epsilon) to the features. This simulates FGSM (Fast Gradient Sign Method) style evasion attacks or extreme market microstructure noise.
- 3. **Signal Flip Measurement**: Count the percentage of predictions that changed ($y_{\text{clean}} \neq y_{\text{adversarial}}$).
- 4. **Vulnerability Scoring**: If the flip rate exceeds the tolerance threshold (e.g., $> 5\%$), flag the model as non-robust and reject deployment.
+ 1. **Assemble the validation set.** Pull a frozen, out-of-sample `X_clean` aligned
+ with the model's training feature schema. Confirm it is 2D and finite.
+ 2. **Configure the perturbation budget.** Construct an `AdversarialRobustnessConfig`:
+
+ ```python
+ from signal_adversarial_tester import (
+ SignalAdversarialTester, AdversarialRobustnessConfig, NOISE_RANDOM_SIGN,
+ )
+
+ config = AdversarialRobustnessConfig(
+ epsilon=0.01, # 1% of per-feature scale
+ flip_tolerance_pct=5.0, # reject if > 5% of signals flip
+ noise_type=NOISE_RANDOM_SIGN, # full-magnitude random-sign stress
+ n_trials=25, # for montecarlo_worst only
+ seed=42, # reproducible (reproducible-ml-pipelines)
+ feature_scales=train_scales, # from training set, not validation
+ feature_bounds=train_bounds, # [min, max] per feature for clipping
+ clip_to_clean_domain=True,
+ decision_threshold=0.5,
+ )
+ ```
+
+ 3. **Run the evaluation:**
+
+ ```python
+ tester = SignalAdversarialTester(config)
+ report = tester.evaluate_model(model.predict, X_clean)
+ ```
+
+ 4. **Interpret the verdict:**
+ - `report.is_robust == True` → signal is stable under the configured noise; the
+ model may proceed to deployment *subject to the other gates in your pipeline*.
+ - `report.is_robust == False` → reject. Route the model back to quant research
+ for **adversarial training** (retrain on perturbed data to smooth the decision
+ boundary), feature re-engineering, or a higher decision threshold.
+
+ 5. **Attribute the fragility.** `report.flipped_indices` identifies exactly which
+ validation samples sit on a flippable boundary — the actionable set for a
+ targeted adversarial-training augmentation.
+
+ ## Decision Points
+
+ | Situation | Action |
+ |-----------|--------|
+ | `vulnerability_score_pct` just under tolerance (e.g. 4.7% vs 5%) | Statistically marginal. Re-run with a larger validation set or bootstrap the CI; do not treat as a clean pass. |
+ | `random_sign` passes but `montecarlo_worst` (n_trials=50) fails | The model is fragile to *some* perturbation direction that a single draw missed. Trust the worst-of-N bound; reject. |
+ | Flips concentrate in a feature subset (inspect `flipped_indices` → rows) | The model is over-reliant on a few fragile features. Re-engineer or drop them rather than globally retrain. |
+ | Flip rate is ~50% on a boundary-clustered set | Expected for a sharp threshold at the boundary — not a bug. Re-test on a *realistic* distribution, not a degenerate constant. |
+ | Model returns probabilities, not labels | Handled automatically (argmax over a 2D array). Verify the class ordering matches your BUY/SELL convention. |
+ | Need a *true* worst-case bound (not empirical) | This engine cannot provide it. Graduate to FGSM/PGD (differentiable models) or certified randomized smoothing. |
+ | `clip_to_clean_domain=True` collapses all noise | The validation feature domain is degenerate (constant column). Either widen the domain via `feature_bounds` or set `clip_to_clean_domain=False` for stress tests that deliberately leave the manifold. |
+ | Reproducibility required across CI runs | Always set `seed`. `None` makes the test non-deterministic and unfit for governance gating. |
+
## Common Pitfalls
- - **Testing with only Gaussian Noise**: Simple Gaussian noise doesn't reflect true adversarial perturbations. Worst-case directional noise (bounding to epsilon limits) is required.
- - **Ignoring Feature Scaling**: Injecting raw noise without respecting the scale of the features (e.g., adding 0.01 noise to a feature bounded between 0 and 1, vs a feature bounded between 1000 and 2000).
+ - **Treating `random_sign` as worst-case.** The legacy `worst_case_sign` name was
+ a misnomer — the signs are *random*, not gradient-derived. Worst-case directional
+ noise needs the loss gradient (FGSM). `montecarlo_worst` gives a *lower bound*
+ on the worst case by taking the max over N random trials; it is tighter than a
+ single draw but still not a true worst case. Read the verdict accordingly.
+ - **Testing on training data.** The model has memorised its training set; flip
+ rates will be artificially low and the gate is meaningless. Always use a frozen
+ out-of-sample holdout.
+ - **Epsilon not calibrated to microstructure.** A blanket `epsilon=0.01` is wrong
+ when features have wildly different scales (price in $ vs a normalized ratio in
+ [0,1]). Either pass per-feature `feature_scales` (training-set) or calibrate ε
+ to one bid-ask spread per instrument.
+ - **No domain clipping.** Perturbing a price feature to a negative value, or a
+ normalized ratio above 1, produces *infeasible* inputs that overstate
+ vulnerability (the model never sees them in production). Use
+ `clip_to_clean_domain=True` or explicit `feature_bounds`.
+ - **Non-deterministic governance.** A deployment gate that flips verdicts between
+ CI runs is worse than no gate. Always set `seed`; pin it in the model card.
+ - **Single noise model.** `uniform` is average-case; it can pass a model that
+ `random_sign` or `montecarlo_worst` fails. Run at least `random_sign` for
+ governance; prefer `montecarlo_worst` for the tighter bound.
+ - **Ignoring the probability path.** A model returning 2D probabilities is decoded
+ via `argmax`; a 1D float score via `decision_threshold`. Verify the decode
+ matches your signal convention before trusting the flip count.
+ - **Using Gaussian noise only.** Gaussian samples rarely reach the ε boundary, so
+ they *under*-state vulnerability versus `random_sign` (which always hits the
+ boundary). Do not use Gaussian as the sole stress.
## Verification
- Run `python scripts/test_signal_adversarial_tester.py` to confirm the injection logic and vulnerability scoring triggers correctly when models flip predictions under noise.
+ Run the unit tests:
+ ```bash
+ python -m unittest discover -s skills/adversarial-robustness-of-trading-signals/scripts -v
+ ```
+
+ What they assert:
+
+ - A robust model passes under `uniform` noise.
+ - A fragile model fails under `random_sign` and under `montecarlo_worst`.
+ - `montecarlo_worst` reports the max flip rate across `n_trials` (≥ any single trial).
+ - Same `seed` → identical report and `flipped_indices` (determinism).
+ - 2D probability matrices decode via `argmax`; 1D float scores via `decision_threshold`.
+ - Domain clipping suppresses infeasible perturbations (both implicit and explicit bounds).
+ - Explicit `feature_scales` collapse a huge ε; zero-variance features fall back to scale 1.0.
+ - Config validation rejects negative ε/tolerance, invalid `noise_type`, `n_trials < 1`,
+ non-positive `feature_scales`, mis-shaped `feature_bounds`, inverted bounds, bad `batch_size`.
+ - Legacy `worst_case_sign` alias routes to `random_sign` (deprecation logged).
+ - `report.as_dict()` round-trips through JSON.
+ - `batch_size` chunking is byte-identical to whole-array evaluation.
+ - Non-2D input raises `ValueError`.
+
+ Confirm with the operational checklist in `assets/checklist.md` before promoting.
+
+ ## Success Criteria
+
+ An adversarial-robustness gate is **healthy in production** when:
+
+ 1. Every model promotion produces a `RobustnessReport` persisted to the model
+ card with `seed`, `epsilon`, `noise_type`, `n_trials`, and `vulnerability_score_pct`.
+ 2. `vulnerability_score_pct` is reported with a confidence interval (or the
+ validation set is large enough that the binomial CI is narrow — ≥ ~2000 samples
+ for a 5% tolerance at the 95% level).
+ 3. `montecarlo_worst` with `n_trials ≥ 25` is the default governance noise model;
+ `uniform`/`random_sign` single-draw results are recorded but not gating.
+ 4. `epsilon` is calibrated per instrument to one average bid-ask spread and the
+ calibration is documented (not a blanket 0.01).
+ 5. Rejected models are routed to adversarial training with the `flipped_indices`
+ attached as the augmentation target set.
+ 6. The gate is deterministic across CI runs (fixed `seed`); a re-run yields the
+ same verdict.
+
## Related Skills
- - `feature-engineering-without-leakage`
- - `backtest-outlier-and-bad-tick-filtering`
+ - `feature-engineering-without-leakage` — the validation set fed to this gate must
+ be leak-free or the robustness verdict is invalid.
+ - `backtest-outlier-and-bad-tick-filtering` — adversarial perturbations and bad-tick
+ outliers are two faces of the same noise; filter the latter, stress the former.
+ - `reproducible-ml-training-pipelines` — the seed discipline this gate depends on.
+ - `model-card-documentation-for-trading-models` — persist the RobustnessReport
+ (seed, ε, noise model, vulnerability score) into the model card.
+ - `model-versioning-and-rollback` — a regression in vulnerability score between
+ versions should block the promotion.
+ - `backtest-parameter-sensitivity-analysis` — for non-ML / rule-based strategies,
+ where adversarial perturbation reduces to parameter sensitivity.
+ - `concept-drift-vs-staleness-differentiation` — live complement: this gate is
+ pre-deployment; drift monitoring is post-deployment.
+ - `class-imbalance-handling-for-rare-signal-events` — rare-signal models have few
+ positive samples to flip; the flip-rate CI is correspondingly wide.