v1.0.0 to v1.1.0
126 added, 26 removed. Audit A to A.
---
name: factor-research-multiple-testing-correction
description: >-
- Quantitative research engine for applying multiple hypothesis testing corrections (Bonferroni, Holm-Bonferroni, Benjamini-Hochberg FDR, and Harvey-Liu-Zhu t>=3.0 haircut) to eliminate p-hacking and false alpha factor discoveries.
+ Quantitative research engine applying multiple hypothesis testing corrections (Bonferroni, Holm, Benjamini-Hochberg FDR, Benjamini-Hochberg-Yekutieli FDR under arbitrary dependence) and the Harvey-Liu-Zhu (2016) t>=3.0 hurdle, so a factor zoo screened at p<=0.05 does not promote noise to production.
domain: Quantitative Research
subdomain: Factor Research & Overfitting Control
- tags: ["factor-research", "multiple-testing-correction", "p-hacking", "fdr", "benjamini-hochberg", "bonferroni", "harvey-liu-zhu"]
- brokers_frameworks: ["Harvey-Liu-Zhu 2016 Standard", "SciPy Stats", "Python Dataclasses"]
- version: "1.0.0"
+ tags: ["factor-research", "multiple-testing-correction", "p-hacking", "fdr", "benjamini-hochberg", "benjamini-yekutieli", "harvey-liu-zhu"]
+ brokers_frameworks: ["Harvey-Liu-Zhu (2016) RFS", "Benjamini-Hochberg (1995)", "Benjamini-Yekutieli (2001)", "Holm (1979)", "Python Standard Library"]
+ version: "1.1.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Use this skill in quantitative alpha research, factor discovery pipelines, and backtest overfitting control. When a quant team tests hundreds or thousands of candidate alpha factors ($M \ge 100$), traditional single-hypothesis significance testing ($t \ge 1.96$, $p \le 0.05$) suffers from severe p-hacking, declaring dozens of useless factors significant purely by chance. This module applies **Benjamini-Hochberg (BH) False Discovery Rate (FDR)**, **Bonferroni / Holm-Bonferroni FWER**, and **Harvey-Liu-Zhu ($t \ge 3.0$) haircut benchmarks**.
+ Use this skill when a research pipeline screens more than one candidate factor and the
+ selection decision rests on t-statistics or p-values: factor discovery sweeps, signal
+ libraries, parameter grids evaluated as separate hypotheses, and any promotion gate
+ where "it was significant in the backtest" is the argument for going live.
+ Screening $M$ candidates at the single-test cutoff ($p \le 0.05$, $|t| \ge 1.96$) is
+ expected to produce $0.05 \times M$ significant results from pure noise — 50 of them
+ out of 1,000. Harvey, Liu and Zhu (2016) argue this is exactly how the published
+ factor zoo was assembled, and recommend that a newly discovered factor clear
+ $|t| \ge 3.0$.
+
+ ## When NOT to Use
+
+ - **A single pre-registered hypothesis.** With $M = 1$ every procedure here collapses
+ to the raw p-value; the correction adds nothing.
+ - **As a fix for a broken backtest.** These are statistical corrections. They cannot
+ see look-ahead leakage, a survivorship-biased universe, or overlapping-return
+ t-statistic inflation. A corrected t-statistic on a leaked signal is still leaked —
+ see `lookahead-bias-elimination`.
+ - **As a per-factor probability.** FDR is an expectation over the batch. Controlling
+ FDR at 5% says nothing about whether any *particular* survivor is real.
+ - **Ranking or sizing.** These procedures answer accept/reject, not "how much alpha"
+ or "how much capital".
+
## Prerequisites
- - List of candidate factor test results (factor ID, name, sample size $N$, raw t-statistic, raw p-value).
- - Target significance level $\alpha = 0.05$ and FDR target $q^* = 0.05$.
+ - Per-candidate test results: factor ID, name, sample size, raw t-statistic, and the
+ **two-sided** p-value corresponding to that t-statistic. `two_sided_p_value_from_t`
+ is provided when only t-statistics are on hand.
+ - $\alpha$ (family-wise level, default $0.05$) and $q^*$ (FDR level, default $0.05$).
+ - An honest count — or defensible upper bound — of the tests actually conducted,
+ including the ones that were discarded and never written down.
## Workflow
- 1. **Raw Significance Audit**:
- - Evaluate raw unadjusted significance ($p_i \le 0.05$, $t_i \ge 1.96$).
- 2. **Bonferroni & Holm-Bonferroni FWER Correction**:
- - Compute Bonferroni threshold $\alpha_{\text{Bonferroni}} = \alpha / M$.
- - Execute Holm step-down sequence: $p_{(i)} \le \frac{\alpha}{M - i + 1}$.
- 3. **Benjamini-Hochberg (BH) False Discovery Rate (FDR) Control**:
- - Sort p-values $p_{(1)} \le p_{(2)} \le \dots \le p_{(M)}$.
- - Find max $k$ where $p_{(k)} \le \frac{k}{M} \times q^*$. Accept factors $1 \dots k$ as true discoveries.
- 4. **Harvey-Liu-Zhu (HLZ 2016) Haircut Benchmark**:
- - Enforce mandatory institutional t-statistic hurdle $t_i \ge 3.0$ ($p_i \le 0.0027$).
- 5. **Audit Report Generation**: Output structured `FactorMultipleTestingAuditReport`.
+ 1. **Establish $M$ before looking at any result.** $M$ is the number of tests
+ *conducted*, not the number *kept*. If 40 specifications were run and 6 survived
+ into the results file, $M = 40$, not 6. Pass the true count as
+ `total_tests_conducted`; leaving it to default to `len(factors)` does not make the
+ audit conservative, it makes it optimistic, and it is the specific bias HLZ
+ document. If the true count is genuinely unknown, use a defensible upper bound and
+ record how it was derived.
+ 2. **Check that each p-value matches its t-statistic and is two-sided.** The HLZ
+ hurdle is stated on $|t|$; every other rule here is stated on $p$. If they
+ disagree, the report contradicts itself. The engine raises on p-values outside
+ $[0, 1]$ or non-finite t-statistics, and logs a warning when a p-value falls below
+ the normal two-sided p-value implied by its t-statistic — impossible for any
+ t-test, since Student-t tails are fatter than normal tails at every finite df.
+ 3. **Decide FWER or FDR before seeing the counts, not after.** FWER (Bonferroni,
+ Holm) bounds the probability of *even one* false discovery — appropriate when a
+ single bad factor going live is costly. FDR (BH, BHY) bounds the *expected
+ proportion* of false discoveries among those accepted — appropriate for a research
+ funnel feeding further validation. Picking whichever gives more survivors is
+ itself a form of p-hacking.
+ 4. **On a correlated factor zoo, read BHY, not BH.** BH controls FDR under
+ independence and under positive regression dependency (Benjamini and Yekutieli
+ 2001, Thm 1.2). Value, quality and low-volatility variants built from overlapping
+ signals satisfy neither. BHY divides the threshold by
+ $c(M) = \sum_{j=1}^{M} 1/j$, restoring FDR control under arbitrary dependence
+ (ibid., Thm 1.3) — this is the FDR procedure HLZ themselves apply. At $M = 100$,
+ $c(M) \approx 5.19$: BHY is roughly five times stricter than BH, and that gap is
+ the price of the guarantee. The engine reports both so the cost is visible.
+ 5. **Apply the HLZ hurdle as an independent cross-check, not a substitute.** The
+ $|t| \ge 3.0$ hurdle is a fixed benchmark calibrated on the published factor zoo;
+ it does not depend on this batch's $M$. Treat disagreement between it and the FDR
+ verdict as a signal to look harder, not as a tie to break by preference.
+ 6. **Read the adjusted p-values, not only the flags.** `adjusted_p_value_bh <= q` is
+ exactly equivalent to `is_bh_fdr_significant`, for every procedure. The adjusted
+ value is the smallest level at which the factor would be accepted, so it shows how
+ much slack a survivor has left.
+ 7. **Record the audit with the batch.** Persist $M$, $\alpha$, $q^*$, $c(M)$ and the
+ per-factor adjusted p-values alongside the promotion decision. Recomputing an
+ audit later with a different $M$ silently changes every verdict.
> Full procedure: see `references/workflows.md`.
- > Standards reference: see `references/standards.md`.
+ > Sourced thresholds and their provenance: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Relying on Raw t-Stats ($t = 2.0$) for Large Factor Zoos**: Selecting top factors out of 1,000 candidates using $t \ge 2.0$ without multiple testing corrections, deploying random noise factors to production.
- - **Over-Conservative Bonferroni Rejection**: Using rigid Bonferroni corrections on highly correlated factor families, mistakenly throwing away genuine alpha signals (Type II error).
- - **Ignoring the Harvey-Liu-Zhu Haircut Benchmark**: Neglecting to apply $t \ge 3.0$ hurdles when testing historical anomaly factors documented in academic literature.
+ - **Setting $M$ to the number of factors you kept.** The 6 survivors written to the
+ results file were selected out of 40 runs; correcting for $M = 6$ leaves the
+ selection bias entirely intact and produces a report that looks rigorous while
+ correcting for almost nothing.
+ - **Reporting BH adjusted p-values without the monotonicity step.** Computing
+ $M p_{(i)} / i$ per rank in isolation yields adjusted p-values that are non-monotone
+ in rank and that can exceed $q^*$ for factors the step-up rule *accepts*. A
+ downstream filter on `adjusted_p <= q` then silently drops factors the audit
+ declared significant. The correct value is the running minimum
+ $\min_{j \ge i} \min[M p_{(j)} / j,\ 1]$.
+ - **Using plain BH on a correlated factor zoo and calling it "the HLZ standard".** BH
+ is not what HLZ apply; BHY is. On HLZ's own ten-test example BH accepts all ten
+ while BHY accepts six.
+ - **Letting a NaN p-value through.** It compares `False` against every threshold, so
+ it is never flagged significant — but it also breaks the sort the rank-based
+ procedures depend on, corrupting the verdicts of every *other* factor in the batch.
+ The engine rejects it rather than degrading quietly.
+ - **Reading `false_discoveries_filtered_count` as a count of proven false factors.**
+ It counts raw-significant candidates that BH did not confirm. The procedure never
+ identifies *which* discoveries are false.
+ - **Rejecting genuine alpha with Bonferroni on a correlated family.** Bonferroni
+ assumes the worst about dependence in the direction of strictness and is uniformly
+ dominated by Holm. If FWER is the requirement, use Holm.
+ - **Re-running the audit with a friendlier $M$ or $q^*$ after seeing the result.**
+ That is the p-hacking this skill exists to stop, moved up one level.
## Verification
- - Instantiate `FactorMultipleTestingCorrectionEngine`. Ingest 100 candidate factor test results (where 10 are true alphas $t > 3.5$, 10 are marginal $t \approx 2.1$, and 80 are noise $t < 1.5$). Verify unadjusted testing falsely accepts 20 factors, Bonferroni accepts 10, and Benjamini-Hochberg FDR controls false discoveries to $\le 5\%$, correctly identifying true alpha factors.
- - Run `python scripts/test_factor_research_multiple_testing_correction.py`.
+ `scripts/test_factor_research_multiple_testing_correction.py` (37 tests) anchors
+ correctness on values published independently of this implementation:
+ - **HLZ Table 4 reproduction.** The paper's worked ten-test example, whose discovery
+ counts are published: single tests 10, Bonferroni 3, Holm 4, BHY 6 — including
+ which test IDs are in each rejection set. `harmonic_sum(10) = 2.928968` reproduces
+ the paper's rank-1 BHY threshold of 0.17%.
+ - **HLZ hurdle p-values.** `two_sided_p_value_from_t` reproduces the paper's
+ published mappings: $t = 3.00 \to 0.27\%$, $t = 2.78 \to 0.54\%$,
+ $t = 3.39 \to 0.07\%$.
+ - **Monotonicity regression.** With $p = (0.010, 0.012)$, $M = 2$, $q^* = 0.015$, the
+ step-up rule accepts both; the un-monotonised formula reports an adjusted p-value of
+ 0.020 for rank 1, contradicting that acceptance. The test asserts the flag and the
+ adjusted p-value agree, and fails against the un-monotonised formula.
+ - **Pure-noise zoo.** 100 tests on the deterministic uniform grid
+ $p = 0.005, 0.015, \dots, 0.995$ — the spread expected when every null is true. Raw
+ screening accepts 5; Bonferroni, Holm, BH, BHY and the HLZ hurdle each accept 0.
+ - **Tie handling, input-order independence, boundary p-values, NaN/out-of-range/
+ duplicate-ID rejection, the `total_tests_conducted` override, and determinism.**
+
+ Run: `python -m unittest discover -s skills/factor-research-multiple-testing-correction/scripts`
+
## Related Skills
- - `research-idea-pipeline-tracking-and-prioritization`
+ - `research-idea-pipeline-tracking-and-prioritization` — tracks how many hypotheses
+ were actually tried, which is where a defensible $M$ comes from.
+ - `lookahead-bias-elimination` — the failure mode no multiple-testing correction can
+ detect.
+ - `walk-forward-validation-setup` — out-of-sample confirmation for factors that
+ survive this audit.
+ - `backtest-parameter-sensitivity-analysis` — a parameter sweep is a family of
+ hypotheses and should be counted into $M$.
- `backtesting-alt-data-strategies-with-realistic-availability-lag`
- ---