experiment · diff
git:20260816.a197bdc to git:20260820.35b5e3e
20 added, 17 removed. Audit A to A.
---
name: experiment
description: Measure performance or tuning changes with a baseline, controlled comparison, correctness check, and recorded verdict. Use for speed, memory, latency, size, "ทดลอง", or any claim that one approach is better. Not for correctness bugs.
---
# Experiment — no claim without a measurement
- An optimization without a baseline is a guess with confidence. The protocol is the same loop as always — but *verify* here means **measure**, and the doctrine's "never re-derive a dead end twice" means null results get recorded with the same weight as wins.
+ Optimization needs numbers; null results get recorded too.
## 1. Define the metric before touching code
- - One command that prints the number: runtime, RSS, p95 latency, binary size, query count. If no such command exists, building it is step zero (the measurement twin of "no verify command is the first bug").
- - Decide **now** what improvement would count — "worth it if ≥10% faster" — so the verdict is not negotiated after the numbers exist.
+ Choose one command that prints the number: runtime, RSS, latency, size, or count.
+ Decide **now** what improvement would count, before seeing results. If no metric
+ command exists, build it first.
## 2. Baseline
- - Run the measurement **at least 3 times**; record all values, not just the mean — the spread is what separates signal from noise.
- - Pin what you can: fixed seed, same input data, warm/cold state chosen deliberately, machine as quiet as you can get it. Note what you could not pin.
- - Correctness verify must be green before and after — a fast wrong answer is not an optimization.
+ Run at least 3 times and record all values, not only the mean. Pin what you can:
+ seed, input, cache state, and environment; state what remains uncontrolled.
+ Correctness verify must be green before and after.
## 3. One variable per experiment
- Change one thing. Two changes in one measurement produce a number that explains neither. (Same discipline as `/debug`; same reason.)
+ Change one thing. Multiple changes make the result uninterpretable.
## 4. Measure again
- - Same command, same repetitions, same conditions.
- - The difference counts only if it clearly beats the baseline spread. Inside the noise = **null result**, not "slightly faster".
+ Use the Same command, same repetitions, same conditions. A result must beat the
+ baseline spread; Inside the noise = **null result**.
## 5. Verdict and record
- Append to `docs/experiments.md` (create it if absent; follow the repo's existing log if one exists):
+ Follow the repository's existing experiment log; otherwise create and append to
+ `docs/experiments.md`:
```
- ## <date> — <hypothesis, one line>
- change: <what was changed, file/approach>
- baseline: <values> | result: <values>
+ ## <date> — <hypothesis>
+ change: <one variable>
+ baseline: <all values> | result: <all values>
verdict: WIN <n%> | NULL (inside noise) | LOSS
- decision: <kept / reverted> — <one-line reason>
+ decision: <kept or reverted + reason>
```
- - **Losers and nulls are reverted immediately** — the log keeps the knowledge, the tree keeps only wins.
- - A null result is a finding: "tried X, no measurable gain — do not retry without new evidence" saves the next session the same hour. If it is load-bearing, surface it via `/retro` into the project notes too.
- - Never delete a previous entry; if a new experiment overturns an old one, add the new entry and cross-reference.
+ Losers and nulls are reverted immediately; the log preserves the finding, not
+ the bad diff. A null result is a finding—record it so it is not retried without
+ new evidence. Never delete a previous entry; append a correction when later
+ evidence overturns it. Route load-bearing nulls through `/retro`.