foundations-causal-inference · v1.4 · 2026-10-05 · sha256 1fe43b7de117a48c
foundations-causal-inference v1.4A
Immutable. This exact content never changes and is served at /api/v1/blob/1fe43b7de117a48c.
--- name: foundations-causal-inference description: "Tests whether changes caused a metric move without a clean A/B test: DiD, synthetic control, IV, RDD. Use when asking did a launch cause lift." compatibility: Portable core only. version: "1.4" last_validated: 2026-08-14 --- # Causal Inference Foundations ## When to Apply **Apply causal-inference when:** - "Did the change cause the outcome, or just correlate?" question - A/B test is impossible (rollout already happened, ethics, ramping risk) — observational methods needed - Confounding suspected — non-random treatment assignment - Heterogeneous treatment effects matter (CATE, uplift) - Mediation question — "is the effect through path X or path Y?" - Units interfere — marketplace, social graph, shared inventory, ranking model, or agents sharing a backend resource; randomization alone does not identify the launch effect - LLM evaluation pipeline uses logged data — prompt distribution, judge bias, or user self-selection confound the quality signal (Pearl's Ladder applies: estimating P(Y|do(prompt)) is different from P(Y|prompt)) - A new router, ranker or bandit policy must be judged from logs of the old one, a short test must decide on a long-term metric, or outcomes are missing unequally by arm — see [Beyond the canonical 12](references/beyond-canonical-designs.md) **Skip and use simpler alternatives when:** - Clean RCT / A/B test is already running *and* units do not interfere — read the result, don't re-derive it observationally. If units share a marketplace, graph, or backend resource, the test is not clean: see [Interference and SUTVA](#interference-and-sutva-when-randomization-is-not-enough) - Question requests an associational summary only — descriptive analytics is enough. The magnitude of a causal effect still requires identification - No plausible causal mechanism — correlation is just measurement, not insight - Effective sample size or treatment overlap is inadequate for the target estimand — narrow the population, change the design, or collect more data - Sensitivity analysis shows that a substantively plausible omitted confounder could reverse the decision — qualify the claim or use a stronger design - Question is about choosing an action under uncertainty rather than estimating an effect — use foundations-decision-theory; for test sizing and intervals, foundations-statistical-inference - Question is about strategic interaction (multi-actor) — use foundations-game-theory ## Quick Reference: Primitives Each primitive has a playbook under [assets/templates/causal-inference/](assets/templates/causal-inference/README.md). Use [references/formal-theory-map.md](references/formal-theory-map.md) for the theory area (SCM, potential outcomes, identification, quasi-experiments) that grounds each one. | # | Primitive | Use When | Failure Mode It Addresses | Core Output | |---|-----------|----------|---------------------------|-------------| | 1 | [DAGs and Structural Causal Models](assets/templates/causal-inference/01-dag-scm.md) | Mapping assumed data-generating process | Implicit untested causal assumptions | Causal graph; confounders, mediators, colliders | | 2 | [Do-Calculus](assets/templates/causal-inference/02-do-calculus.md) | Identifying causal effects from observational data | Treating P(Y\|X) as causal without identification | Identifiability check; expression for P(Y\|do(X)) | | 3 | [Backdoor / Frontdoor Criterion](assets/templates/causal-inference/03-backdoor-frontdoor.md) | Choosing a valid adjustment set | Conditioning on the wrong variables; collider bias | Minimal sufficient adjustment set | | 4 | [Instrumental Variables](assets/templates/causal-inference/04-instrumental-variables.md) | Unobserved confounders; a valid instrument exists | Omitted-variable bias | LATE for compliers | | 5 | [Regression Discontinuity](assets/templates/causal-inference/05-rdd.md) | Treatment assigned by a threshold rule | Selection bias in threshold assignment | Local effect at the cutoff | | 6 | [Difference-in-Differences](assets/templates/causal-inference/06-diff-in-diff.md) | Pre/post data with treated and comparison groups | Pre-existing trends misattributed as effects | ATT under parallel trends | | 7 | [Synthetic Control (and SDiD)](assets/templates/causal-inference/07-synthetic-control.md) | One or few treated units, a donor pool | No valid single control unit | Counterfactual trajectory | | 8 | [Propensity Score Methods](assets/templates/causal-inference/08-propensity-score.md) | Measured confounders only | Covariate imbalance | ATE/ATT via matching, IPW or DR; dose-response via DML for continuous treatment | | 9 | [CATE / Uplift Modeling](assets/templates/causal-inference/09-cate-uplift.md) | Heterogeneous effects matter | ATE masking opposing subgroup effects | Subgroup CATE; uplift scores | | 10 | [Simpson's Paradox and Confounding Traps](assets/templates/causal-inference/10-simpsons-paradox.md) | Aggregate trend contradicts subgroups | Aggregation reversals; collider conditioning | Correct stratification | | 11 | [Mediation Analysis](assets/templates/causal-inference/11-mediation-analysis.md) | Decomposing direct and indirect paths | Total effect treated as direct | NDE, NIE, proportion mediated | | 12 | [Sensitivity Analysis](assets/templates/causal-inference/12-sensitivity-analysis.md) | Robustness to unobserved confounding | Conclusions that collapse under modest hidden bias | Scale-compatible sensitivity (partial-R²/OVB, E-value for risk ratios, Rosenbaum Γ, HonestDiD) | Designs outside these 12 — interrupted time series, off-policy evaluation, surrogate index, Manski/Lee bounds, negative controls and proximal inference — are in [references/beyond-canonical-designs.md](references/beyond-canonical-designs.md). --- ## Anti-Patterns and Misuse | Anti-Pattern | Causal Diagnosis | Fix | |-------------|-----------------|-----| | Treating correlation, prediction or P(Y\|X) as a causal effect | Confounders invalidate effect direction, let alone magnitude | State the estimand; check identifiability (#2) before any regression | | Drawing a DAG after seeing results, or adjusting for every available variable | Post-hoc graphs encode the desired conclusion; colliders and mediators add bias | Draw the DAG (#1) before modeling; use the backdoor adjustment set (#3) | | Conditioning on a post-treatment variable | Blocks the causal path; collider bias on mediator proxies | Identify mediators in the DAG; use mediation analysis (#11) if the path is the target | | Parallel-trends violation in DiD, or a passed pretest read as proof | Pre-trend tests have low power; the control group may not be a valid counterfactual | Report pretrends with uncertainty and bounded-trend sensitivity (HonestDiD), not a pass/fail gate; synthetic control (#7) is a different design with its own assumptions, not a drop-in repair | | Weak instrument accepted on a first-stage F > 10 | With one instrument, a true 5% t-test needs F > 104.7; at F = 10 the valid critical value is 3.43, not 1.96 (Lee, McCrary, Moreira & Porter, *AER* 2022) | One instrument: report Anderson–Rubin intervals or tF-adjusted inference. Several instruments: report the effective F (Montiel Olea & Pflueger 2013) with weak-IV-robust (AR-type) inference; do not extend the 104.7 threshold beyond one instrument (#4) | | Propensity-score overlap failure | Extreme propensities give unstable weights; effective sample collapses | Distinguish numerical tails from structural non-overlap; restrict and disclose the target estimand. Clipping, DR and matching do not identify absent counterfactuals (#8) | | Averaging heterogeneous effects, or publishing CATE without overlap checks | Opposing subgroups cancel; CATE extrapolates outside support | Run CATE/uplift (#9) on an identified ATE; check positivity and subgroup N | | Calling an observational estimate "proven impact" | Unmeasured confounding remains possible | Choose sensitivity for the effect scale: E-values for risk ratios (or explicitly justified conversions), Rosenbaum bounds for matched designs, partial-R²/OVB for continuous outcomes; for IV, IV robustness values (Cinelli & Hazlett 2025) (#12) | | Reporting a unit-randomized marketplace or shared-backend test as the launch effect | Interference changes the identified contrast even under perfect randomization | Name the exposure mapping, assignment support and target contrast; cluster/switchback designs are options, not automatic identification of a global effect | --- ## Decision Checklist Use this to pick the right method before modeling: - [ ] **Can you draw the assumed DAG?** If not, stop — assumptions are implicit and untestable. Draw DAG (#1) first. - [ ] **Is the effect you want interventional (do(X)) or conditional?** If interventional, check identifiability with do-calculus (#2). - [ ] **Can one unit's treatment change another unit's outcome?** (marketplace supply/demand, social graph, shared inventory, ranking model, geographic proximity) If yes, specify direct, indirect, total or global estimand, exposure mapping and assignment support before choosing an estimator; redesign only when current support/assumptions do not identify the target. See [Interference and SUTVA](#interference-and-sutva-when-randomization-is-not-enough). - [ ] **Do you have an RCT or clean natural experiment?** If yes, use the design directly. If outcomes are missing unequally by arm (attrition, timeouts), use Lee bounds ([beyond-canonical-designs](references/beyond-canonical-designs.md#partial-identification-bounds-instead-of-a-point)). If the decision metric arrives long after the test, consider a validated surrogate index. If no, continue. - [ ] **Is there a threshold that determines treatment?** → RDD (#5). Report the robust bias-corrected interval, not the conventional one. - [ ] **Is there pre/post data with a comparable untreated group?** → DiD (#6). Check parallel trends first. - [ ] **Is treatment staggered (units adopt at different times)?** → Use Callaway–Sant'Anna, Sun–Abraham, BJS imputation, or Gardner 2-stage. Use a heterogeneity-robust estimator aligned with the target ATT; dynamic effects can contaminate TWFE comparisons. - [ ] **Is the treatment a continuous dose?** → Continuous-treatment DiD (Callaway, Goodman-Bacon & Sant'Anna); parallel trends alone does not make comparisons across doses causal. - [ ] **Is parallel trends uncertain?** → Apply HonestDiD (Rambachan & Roth 2023) for honest CIs under bounded violations. - [ ] **Pre/post data with donor pool but parallel trends uncertain?** → Synthetic DiD (Arkhangelsky et al. 2021, #7 extension). - [ ] **Single treated unit with no clean control?** → Synthetic control (#7). **No donor units at all, only one series?** → Interrupted time series / BSTS, the weakest design here ([beyond-canonical-designs](references/beyond-canonical-designs.md#interrupted-time-series-its-and-bsts)). - [ ] **Are there unobserved confounders and a valid instrument?** → IV (#4). Validate exclusion; use weak-IV-robust inference (AR or tF for one instrument), not an F > 10 gate. - [ ] **Observational data with measured confounders only?** → Propensity score matching / IPW / DR (#8). Check overlap. If a known confounder is unmeasured, test with negative controls first. - [ ] **Is the "treatment" a new decision policy evaluated on logs of the old policy?** → Off-policy evaluation; propensities must be logged ([beyond-canonical-designs](references/beyond-canonical-designs.md#off-policy-evaluation-from-logged-data)). - [ ] **Do you need conditional average effects for covariate-defined groups?** → CATE / uplift (#9). Choose meta-learner by sample size. - [ ] **Does the aggregate trend contradict subgroup evidence?** → Check for Simpson's paradox via DAG stratification (#10). - [ ] **Is the total effect mediated by an intermediate variable?** → Mediation analysis (#11). Requires treatment and mediator exchangeability, consistency and positivity; natural effects also require no exposure-induced mediator–outcome confounding. - [ ] **Is the conclusion actionable under unobserved confounding?** → Select a design- and scale-compatible sensitivity method (#12), name plausible confounders, and report it. If point identification fails, report bounds instead. - **Before writing any number.** Verify every citation against a primary source before writing it; `data/sources.json` is the canonical citation list — do not restate it here. Never invent a coefficient, p-value, or weight; re-derive worked examples and hedge or omit what you cannot verify. --- ## Composition Recipes Multi-method stacks for common scenarios; full stacks, worked examples and caveats are in [references/composition-recipes.md](references/composition-recipes.md). | Recipe | Use When | Key Stack | |---|---|---| | Uplift from Observational Data | Conditional average effects, no RCT | DAG (#1) → propensity/DR, DML for continuous treatment (#8) → CATE/X-learner (#9) → sensitivity (#12); add IV (#4) if a valid instrument exists | | Policy Evaluation with No Control Group | Single market/cohort, no comparison group | DAG (#1) → synthetic control with donors and donor/pre-fit checks; if no donors, ITS/BSTS with time-series sensitivity (#7, #12) | | Mechanism Attribution | Decompose direct vs indirect (mediated) effect | DAG (#1) → backdoor adjustment (#3) → propensity/DR (#8) → mediation analysis (#11) → sensitivity (#12) | | LLM Evaluation Pipeline | Deconfound a logged, judge-scored quality signal for a prompt/model/RLHF change | DAG (#1) → do-calculus/backdoor (#2, #3) → DR (#8), or off-policy evaluation if the change is a per-request policy → CATE (#9) → sensitivity (#12). LLM-proposed DAG edges are priors to validate, not identified structure. | --- ## Interference and SUTVA: When Randomization Is Not Enough No-interference is part of SUTVA, which also concerns treatment versions. Many conventional unit-level formulas assume it, but DAGs/SCMs and potential-outcome models can explicitly include cross-unit causes. Interference does not make every estimand unrecoverable. Specify the exposure mapping, the direct/indirect/total/global intervention contrast, assignment probabilities and support, consistency of exposures, and remaining identification assumptions. Randomization alone does not turn a direct effect into the all-treated-versus-all-control launch effect. **Counterexample:** independent Bernoulli(.5) assignments Z1,Z2 with Y1=3Z1+7Z2 and Y2=3Z2+7Z1 have interference. The Horvitz–Thompson direct-effect estimator averaging 2ZiYi−2(1−Zi)Yi over the two units has expectation 3 across the four equally likely assignments. The global all-treated-versus-all-control contrast is 10. Both contrasts exist; they answer different questions. Exposure-probability weighting requires positive probability for the exposures being contrasted. [Aronow & Samii (2017), design/exposure/estimand framework](https://arxiv.org/abs/1305.6156). Identify the interference structure first, then pick the design: | Interference structure | Design | Estimation note | |---|---|---| | Spatial or graph neighbors (social, geo, ride-hailing) | Cluster randomization on the graph's dense components | Difference-in-neighbors (Peng, Ye & Zheng 2025) attains second-order bias in interference magnitude with far lower variance than Horvitz–Thompson | | Temporal carryover on a single shared system (pricing, matching, ranking) | Switchback: randomize treatment over time blocks | Align schedule and estimator with carryover order *m*. Under its horizon assumptions, [Bojinov et al., Theorem 3.7](https://arxiv.org/html/2009.00148v1) permits interior epochs of length *m* | | Both spatial and temporal (delivery, marketplace supply) | Clustered switchback (Jia, Kallus & Yu 2025) | Truncated Horvitz–Thompson; MSE matches the lower bound up to log terms on sparse graphs | | Market-level equilibrium effects (budget, inventory, auction) | Geo or market-level randomization; unit-level tests cannot see it | Few treated units — use randomization inference, not asymptotic SEs | **The reporting distinction that matters**: under interference, the unit-level "treatment effect" and the effect of switching *everyone* (the global/total treatment effect) are different quantities. Cluster or switchback designs target contrasts defined by their allocation, carryover and exposure assumptions; neither automatically identifies the global rollout effect. Say which contrast was identified and justify any extrapolation. Agent and LLM products hit this directly: agents sharing a rate limit, a retrieval index, a cache, or a tool backend interfere through the shared resource, so per-session randomization can identify a different contrast and may understate or invert the launch effect; establish this for the actual resource/exposure model. --- ## Expert Judgment ### Picking an Identification Strategy From Data Shape - **One treated unit, a time series, and a pool of comparable untreated units** → synthetic control or synthetic DiD, not a hand-picked comparison unit. If pre-treatment fit is poor, say so and stop rather than force it. - **A rule with a hard numeric cutoff and enough density of units near it** → RDD, not a linear control for the running variable. If the running variable is coarse (rounded scores, integer ages), check for heaping before trusting continuity. - **Treatment rolled out at different times across units** → check whether never-treated or not-yet-treated units exist, then use a heterogeneity-robust staggered-DiD estimator (Callaway–Sant'Anna, Sun–Abraham, BJS, or Gardner). Dynamic effects can contaminate TWFE comparisons; cohort heterogeneity alone does not imply negative weights or bias for every target. - **Confounders you can name and measure completely, with common support across treated/control** → propensity/DR. If you cannot name the confounders, no amount of covariate adjustment substitutes for a design — look for a natural experiment (IV, RDD) instead. - **An exogenous shock or rule that shifts treatment for some units and not others, for a reason unrelated to the outcome** → IV, but only if the exclusion story survives being explained to a skeptical colleague in one sentence. If the one-sentence version needs three caveats, the instrument is probably not clean. - **The real question is "who benefits," not "what's the average effect"** → CATE/uplift layered on top of an already-validated ATE/ATT, never as a substitute for identification. A confounded CATE just reports which subgroup has the most confounding. ### The Assumption That Actually Fails in Practice The textbook assumption is rarely violated the way the textbook describes it. What experts actually watch for: | Method | Textbook assumption | What breaks in real data | |---|---|---| | DiD | Parallel trends | Treated units were selected *because* they were already diverging (mean reversion, selection on trend) — pre-trend tests have low statistical power, so a "flat" pre-trend plot is weak evidence, not proof (Roth 2022) | | IV | Exclusion restriction | The instrument is excludable in theory but leaks through an unmodeled common shock (e.g., a policy or cohort effect correlated with both the instrument and unobserved confounders) | | RDD | Continuity / no manipulation | The running variable is granular (rounded, integer, self-reported) — heaping at the cutoff looks like a density blip, not manipulation, and a density test at one bandwidth can miss it | | Synthetic control | Good pre-treatment fit | Low aggregate RMSPE is achieved by 2–3 donors carrying nearly all the weight (interpolation bias) — inspect the weight vector itself, not just RMSPE | | Propensity / DR | Strong ignorability (all confounders measured) | Treatment was assigned by a human or algorithm using private information not in X (a manager's judgment, a salesperson's read on the customer) — balance tables on measured covariates cannot detect this, and it is the single most common real-world failure | | CATE / uplift | Same ignorability as ATE, per subgroup | Overlap can fail in exactly the subgroup with the highest estimated CATE; the "best segment" is often the one with the least support and the most confounding, not the most persuadable one | | Mediation | Sequential ignorability | Randomizing exposure alone does not establish mediator–outcome exchangeability. Defend mediation identification assumptions and report mediation-specific sensitivity; see [Imai et al., pp. 309–313](https://imai.fas.harvard.edu/research/files/BaronKenny.pdf) | ### Placebo and Robustness Checks an Expert Always Runs - **Placebo-in-time**: rerun the design as if treatment happened one period earlier; expect a null effect. - **Placebo-in-space / placebo-outcome**: rerun on units or outcomes the treatment should not affect. - **Leave-one-out**: drop the highest-weight synthetic-control donor, or the strongest component of a composite instrument, and confirm the estimate does not collapse. - **Specification / bandwidth curve**: show the estimate across a range of RDD bandwidths or DiD control sets, not just the one preferred specification. - **Randomization inference**: use permutation p-values instead of asymptotic SEs when clusters or treated units are few (a handful of treated states or markets). - **Sensitivity analysis as a routine output, not an appendix**: a scale-compatible method (partial-R²/OVB, E-value for risk ratios, Rosenbaum bounds, or HonestDiD) accompanies every observational or DiD point estimate, not just the ones that look fragile. ### When Causal ML Adds Nothing Over a Good Quasi-Experiment - If a credible design already exists (valid IV, sharp RDD, staggered DiD with a heterogeneity-robust estimator) and the target is a single ATE/ATT/LATE, doubly-robust ML nuisance estimation buys efficiency, not identification. The design is doing the causal work; DML is just a better nuisance-function fitter. - Causal ML (causal forests, DML, meta-learners) earns its complexity when: (a) covariates are high-dimensional with an unknown confounding functional form, (b) the question is heterogeneity (CATE/uplift) that a single quasi-experiment cannot answer without infeasible sample size, or (c) treatment is continuous/high-cardinality with no closed-form estimator. - It does not repair a broken identification strategy. Running `econml` on top of a DiD with violated parallel trends, or a `dowhy` refutation suite on top of an IV with a leaky exclusion restriction, produces a precise, doubly-robust, wrong answer. Fix identification before reaching for machine learning. - **Where the literature is genuinely unsettled** (state this plainly rather than picking a side): (1) which staggered-DiD estimator (Callaway–Sant'Anna, Sun–Abraham, BJS, Gardner) to prefer is setting-dependent, not resolved — the 2026 JEL practitioner's guide (Baker, Callaway, Cunningham, Goodman-Bacon & Sant'Anna) frames the choice by design and target estimand rather than naming a winner, and the estimators can disagree meaningfully on the same panel; (2) the best-practice sensitivity-analysis default for ML-based ATEs (Chernozhukov, Cinelli et al. 2026 vs. simpler partial-R² benchmarks) is still settling in applied practice; (3) using LLMs to propose or accelerate causal discovery had no consensus validation protocol in the sources reviewed for this skill — treat LLM-proposed edges as priors to test, not conclusions to report. --- ## Workflow 1. State the intervention, outcome, unit and estimand: intervention effect, mechanism, heterogeneous effect, or policy value? 2. Draw the DAG (#1). Identify confounders, mediators, and colliders. 3. Use the [Decision Checklist](#decision-checklist) to select the identification strategy. 4. Open [references/primitives-overview.md](references/primitives-overview.md) for the decision map, estimand taxonomy, assumption inventory and [Tooling Landscape](references/primitives-overview.md#tooling-landscape); open the matching template for inputs, failure modes and a worked example. 5. For multi-method stacks, use the [Composition Recipes](#composition-recipes) above. 6. Check [references/patterns-scenarios-traps.md](references/patterns-scenarios-traps.md) and always close with sensitivity analysis (#12) when reporting observational estimates. For risk ratios, `python3 scripts/evalue.py --rr RR --ci-low LOW --ci-high HIGH` validates the interval and uses the confidence bound closest to the null. 7. Report assumptions, effect, uncertainty and fragility; on unidentified effects report an association, bounds, or "inconclusive". --- ## Related Skills Consumer skills keep domain adapters in `references/causal-inference-applied.md` (or a domain-named equivalent). Each gates on this skill's [When to Apply](#when-to-apply) and links back here for theory: - `../marketing-cro/references/causal-inference-applied.md` and `causal-inference-experimentation.md` — CUPED, switchback and geo-experiments, CATE for personalization - `../marketing-product-analytics/references/causal-inference-applied.md` and `causal-inference-analytics.md` — funnel and retention attribution - `../marketing-paid-advertising/references/causal-inference-applied.md` — media-mix and geo-lift attribution - `../marketing-email-automation/references/causal-inference-applied.md` — send-time and lifecycle-campaign lift - [`../product-management/references/causal-inference-applied.md`](../product-management/references/causal-inference-applied.md) — feature-rollout impact - `../startup-business-models/references/causal-inference-applied.md` — pricing and monetization lift - [`../qa-debugging/references/causal-inference-applied.md`](../qa-debugging/references/causal-inference-applied.md) — regression root-cause attribution - [`../foundations-decision-theory/references/thresholds-stopping-and-ope.md`](../foundations-decision-theory/references/thresholds-stopping-and-ope.md) — OPE estimators and acting on a policy-value estimate --- ## Navigation - Designs beyond the 12 (ITS/BSTS, off-policy evaluation, surrogate index, bounds, negative controls/proximal) — read when the checklist lands outside the canonical primitives: [references/beyond-canonical-designs.md](references/beyond-canonical-designs.md) - Practical completion contract and known-answer controls: [references/practical-contract.md](references/practical-contract.md) - Formal theory map: [references/formal-theory-map.md](references/formal-theory-map.md) - Patterns, scenarios, and traps: [references/patterns-scenarios-traps.md](references/patterns-scenarios-traps.md) - Decision map, estimands, assumptions, tooling: [references/primitives-overview.md](references/primitives-overview.md) - Per-primitive playbooks: [assets/templates/causal-inference/README.md](assets/templates/causal-inference/README.md) - Sources: [`data/sources.json`](data/sources.json) is the citation register; verify a claim against the primary source it lists before quoting a number. ## Learnings Loop When prior decisions or pitfalls are relevant, consult `learnings.consolidated.md` if present; use `learnings.md` only for needed history or as the available fallback. Otherwise skip both. After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.