git:20260729.5b86b60 to git:20260729.4a3a4d0

---

name: experiment-design

description: "Playbook for trustworthy experiments — hypothesis framing, test types, sample size and power, statistical vs practical significance, stopping rules, and the traps (peeking, multiple comparisons, novelty effects). Used by experiment-designer."

---

## Codex host binding

- Treat references to Claude slash workflows as the equivalently named Codex skill.

- Resolve bundled paths from the installed GreatPM plugin root.

- Ignore Claude-only model aliases, colors, turn limits, and tool allowlists.

- Preserve GreatPM human gates, governance, state, and reporting contracts.

# Experiment Design — playbook

Experiments produce useful answers only when they are honestly designed.

Underpowered tests produce noise; peeking produces false positives;

guardrail-blind tests produce shipped regressions.

## 1. Hypothesis — one sentence with a mechanism

`If we do <change>, <primary metric> will move by <effect size> because <mechanism>.`

- The metric is real and pre-defined (it is in the metrics plan).

- The effect size is the MDE — the minimum effect that would matter for the

business. Smaller MDE = bigger sample required.

- The mechanism is why this should work. If you cannot articulate the

mechanism, the result will not generalise.

## 2. Test types — pick by what you can randomize

- **A/B test** — randomize users into control vs treatment. Default for

feature changes.

- **Multivariate** — multiple variants at once (A/B/C/D). Beware: inflates

multiple-comparison risk; correct alpha or use one decision metric.

- **Holdout** — keep a slice of users in pure control long-term, even after

launch. Useful for measuring cumulative effect.

- **Switchback** — for marketplace / two-sided contexts where user-level

randomization leaks. Alternate the treatment in time windows by region or

segment.

If randomization is impossible (a regulator change, an architecture change

that touches everyone), do not pretend — declare it observational and label

the result co-movement, not effect.

## 3. Sample size & statistical power

A test that cannot detect the effect you care about is a noise generator.

- **Power** = probability of detecting a real effect. Default: 80%.

- **Alpha** = false-positive rate. Default: 5%.

- **MDE** (minimum detectable effect) = the smallest effect that would matter.

- **N** depends on baseline rate, MDE, power, alpha.

A simple rule of thumb (for binary metrics, two-arm test, 80% power, 5%

alpha): `N per arm ≈ 16 × p × (1−p) / MDE²` where p is the baseline rate.

Use a proper power calculator for non-trivial cases.

If the sample is unreachable in a reasonable window: shrink the MDE

honestly, segment more carefully, or accept that this question cannot be

answered by an experiment.

## 4. Statistical significance vs practical significance

- **Statistical significance** — p < alpha. Says the effect is unlikely to be

random.

- **Practical significance** — the effect is big enough to matter.

A statistically significant 0.2% lift on a flagship metric is real but maybe

not worth shipping. State both.

## 5. Stopping rules — pre-register them, no peeking

Before you start, write down:

- Planned duration (calendar days).

- Planned sample size (per arm).

- Primary decision metric.

- Guardrail metrics (what must NOT regress).

- The rule for early stop (often: don't, unless sequential testing is

pre-planned with corrected alpha).

**Peeking at unplanned interim results inflates the false-positive rate.**

If you must look early, use sequential testing methods (mSPRT, Bayesian

group-sequential) with explicit corrections.

## 6. Guardrails — what must NOT regress

For every experiment, name the metrics that must NOT get worse:

- Latency / performance.

- Error rate.

- Long-term retention (if the test is short).

- A core engagement signal in an adjacent area.

A "winning" primary metric with a broken guardrail is not a win — it is a

regression in disguise.

## 7. Common traps

- **Peeking** — looking at results before the planned end. Inflates false

positives.

- **Multiple comparisons** — running 10 sub-metric tests, finding one

"winner". Correct alpha (Bonferroni) or pre-register one decision metric.

- **Novelty effects** — early uplift fades as the new wears off. Run long

enough; consider a long-hold cohort.

- **Survivorship** — looking only at users who completed the funnel.

- **Simpson's paradox** — aggregate result flips when segmented. Pre-declare

the primary cut.

- **HARKing** — Hypothesising After the Results are Known. Frame the

hypothesis BEFORE the test, in writing.

## 8. Output shape

An experiment plan: hypothesis (one sentence with mechanism), primary +

guardrail metrics, test type with justification, sample size from a power

calc, duration, pre-registered stopping rules, risks (novelty / seasonality /

network effects), and a read-out template to be filled at the end.