measurement-experimentation-ops git:20261014.8613433A

Immutable. This exact content never changes and is served at /api/v1/blob/8ef84c866d6b9b5f.

---
name: measurement-experimentation-ops
description: "Gatekeeper for 'is this result real before scaling' on Meta/Facebook buys: SRM, peeking, contamination, lag and multiple-testing checks; picks causal, screening or infrastructure test mode plus Meta lift tools (A/B Test, Conversion Lift, GeoLift, Robyn). When volume, dying accounts or no rep rule out a lift study, prescribes the honest weaker method instead of a fake lift."
---

# Measurement & Experimentation Ops

The other skills act on measured differences; this decides whether a difference is real or
noise before they do.

## Pick the testing mode by the decision at stake

Three modes, different evidence bars; match to the cost of being wrong:

1. **Causal**: estimates incrementality (no design *proves* causality without assumptions).
   A randomized experiment (treatment vs non-overlapping control/holdout, pre-sized) is
   strongest; quasi-experimental estimation (GeoLift synthetic control, pre/post) is the
   fallback when you can't randomize. For expensive, hard-to-reverse bets: offer, funnel,
   landing page, "does this channel even lift sales". Cost: volume + discipline + often a
   Meta rep.
2. **Screening** (directional): many concepts in one ad set / parallel ABO cells; delivery is
   UNEQUAL by design, so a "winner" is a hypothesis, not a proof. For creative hunting where
   fast beats certain. Never present a screen result as validated.
3. **Infrastructure**: hold the CREATIVE fixed, vary one infra axis (domain / proxy cluster /
   account batch) across a balanced set, to attribute delivery/ban/CPM differences to infra,
   not creative (`../meta-grey-ops/references/06-portfolio-forensics.md`).

## Feasibility gate (check BEFORE promising a clean test)

Causal measurement is often unavailable on grey/small-account buys: too little volume to
power a holdout, accounts die mid-test, no clean pixel signal, no rep for a sandboxed
Conversion Lift. Then SAY SO and drop to the best affordable proxy (geo holdout, pre/post
with tracker truth, screening), labeled directional. The honest weaker method beats a
"causal" test that's silently contaminated.

## Validity traps (each silently flips a conclusion)

- **SRM:** check the RANDOMIZED-UNIT split (50/50 arriving 55/45 = broken
  randomization/logging) on assignment counts, NOT on spend/impressions/conversions (those
  diverging is a delivery effect).
- **Peeking:** leave Meta's A/B "end test early if a winner is found" off and run the pre-set
  window (02).
- **Contamination:** overlapping audiences between cells (Advantage+ broad bleeding into
  manual cells; duplicated winners cannibalizing in the auction). Use the A/B tool's
  non-overlapping split, or geo separation.
- **Conversion lag:** judging before the payout event matures counts spend against unripe
  conversions, so fresh cohorts look like losers. Window ≥ lag; nowcast if you must decide
  early (`../tracker-ops/references/03-metrics-and-math.md`).
- **Day of week:** cost per payout event swings weekday vs weekend/holiday more than with
  volume; compare same-kind full days (01).
- **Multiple testing:** screening tolerates chance winners (you re-test anyway); a causal
  decision needs the bar corrected for the number of comparisons.
- **Underpowered:** "no significant difference" ≠ "no effect"; size first (01).

## Route references

| Need | Reference |
|---|---|
| Sizing (MDE/power as decision rules), SRM, peeking, contamination, lag, inconclusive handling | `references/01-experiment-design.md` |
| Meta tools: A/B Test, `ad_study` API, Conversion Lift, Brand Lift, GeoLift, Robyn/MMM, Andromeda implication | `references/02-meta-measurement-tools.md` |

Buy mechanics: `../meta-ads` (`references/09-diagnostics-unit-economics.md` owns
single-account diagnosis and test-design intake); this skill owns the validity layer above
it. Counting and cohort truth: `../tracker-ops`. Portfolio decisions on the result:
`../senior-buyer-ops`.