v1.0.0 to v1.0.0
45 added, 23 removed. Audit A to A.
---
name: feature-selection-stability-across-folds
- description: Skill for feature selection stability across folds
- domain: Financial ML
- subdomain: Feature Selection
- tags:
- - ml
- - trading
- brokers_frameworks:
- - scikit-learn
- version: 1.0.0
- author: System
- license: MIT
+ description: >-
+ Quantitative ML engine for measuring feature selection stability across cross-validation folds using Nogueira's Index, computing inclusion frequencies, and filtering unstable overfitted features.
+ domain: Financial ML & Validation
+ subdomain: Cross-Validation & Feature Selection Robustness
+ tags: ["feature-selection", "nogueira-index", "kuncheva-index", "cross-validation", "fold-stability", "p-hacking", "overfitting-control"]
+ brokers_frameworks: ["Nogueira 2017 Measure", "Kuncheva Index", "Python Dataclasses"]
+ version: "1.0.0"
+ author: algo-trading-skills-contributors
+ license: Apache-2.0
---
- # feature-selection-stability-across-folds
-
## When to Use
- Use when implementing feature selection stability across folds in financial ML workflows.
+ Use this skill in quantitative feature selection pipelines, walk-forward cross-validation setups, and model robustness audits. When feature selection algorithms (Lasso, Boruta, RFE) select different subsets of features across cross-validation folds, the resulting model is fragile and overfitted. This module calculates **Nogueira's Stability Index ($\Phi$)** and feature inclusion frequencies ($p_i$), retaining only consensus features selected in $\ge 80\%$ of folds.
+
## Prerequisites
- - Python 3.9+
- - scikit-learn
+ - Total number of candidate features $M$.
+ - List of selected feature sets across $K$ cross-validation folds: $[S_1, S_2, \dots, S_K]$.
+ - Minimum inclusion threshold $p_{\text{min}} = 0.80$ and minimum Nogueira stability threshold $\Phi_{\text{min}} = 0.70$.
+
## Workflow
- 1. Initialize the components
- 2. Process the input data
- 3. Return outputs
+ 1. **Feature Inclusion Frequency Calculation**:
+ - For each candidate feature $i \in \{1 \dots M\}$:
+ - Compute selection probability: $p_i = \frac{1}{K} \sum_{k=1}^K \mathbf{1}_{i \in S_k}$.
+ 2. **Nogueira Stability Index ($\Phi$) Evaluation**:
+ - Compute average selected subset size $\bar{k} = \frac{1}{K} \sum_{k=1}^K |S_k|$.
+ - Calculate sample variance $s_i^2 = \frac{K}{K-1} p_i (1 - p_i)$.
+ - Calculate Nogueira Index: $\Phi = 1 - \frac{\frac{1}{M} \sum s_i^2}{\bar{k} \left(1 - \frac{\bar{k}}{M}\right)}$.
+ 3. **Consensus Feature Set Extraction**:
+ - Retain Consensus Features: $S_{\text{consensus}} = \{f_i \mid p_i \ge 0.80\}$.
+ - Prune Unstable Features: $S_{\text{pruned}} = \{f_i \mid p_i < 0.80\}$.
+ 4. **Stability Audit & Status Determination**:
+ - If $\Phi \ge 0.70 \implies$ Flag `STABLE_FEATURE_SET`.
+ - Else $\implies$ Flag `UNSTABLE_OVERFITTED_FEATURE_SET`.
+ 5. **Audit Report Generation**: Output structured `FeatureStabilityAuditReport`.
+
+ > Full procedure: see `references/workflows.md`.
+ > Standards reference: see `references/standards.md`.
+ > Printable pre-flight checklist: see `assets/checklist.md`.
+
## Common Pitfalls
- - Overfitting
- - Lookahead bias
+ - **Ignoring Cross-Fold Feature Variance**: Training models on union feature sets containing unstable features selected in only 1 of 10 folds.
+ - **Using Jaccard Index Without Chance Correction**: Evaluating raw subset overlap without adjusting for random chance agreements when $\bar{k} \approx M / 2$.
+ - **Failing to Re-Train on Consensus Features**: Retaining all features despite low Nogueira stability scores.
+
## Verification
- - Run tests in the scripts folder
+ - Instantiate `FeatureStabilityAnalyzerEngine`. Input 10 candidate features across 5 walk-forward CV folds. Scenario 1: Identical feature selection across all 5 folds $\implies$ verify engine computes Nogueira Index $\Phi = 1.0$, flags `STABLE_FEATURE_SET`, and retains 100% consensus. Scenario 2: Random erratic feature selections across folds $\implies$ verify engine computes $\Phi < 0.30$, prunes unstable features, and flags `UNSTABLE_OVERFITTED_FEATURE_SET`.
+ - Run `python scripts/test_feature_stability_analyzer.py`.
+
## Related Skills
- - feature-engineering
+
+ - `walk-forward-validation-setup`
+ - `feature-importance-drift-monitoring`
+ ---