reading-measurements ยท diff
git:20261005.1b046dd to git:20261010.419d0bb
47 added, 32 removed. Audit A to A.
---
name: reading-measurements
description: "Read a number without being fooled by it: an exit code, a passing result file, a test count, a list size, a control run, a red or a green. Use before reporting any figure, before turning a test run into a verdict, before acting on 'N items are broken', and whenever two outputs or two counts of the same thing disagree."
---
# Reading measurements
Wrong reports rarely lack a measurement. They read a real number as answering a
question it never asked. Before you report, decide or spend anything on a
figure, ask:
> What would this same number look like if my claim were false?
- If the answer is "exactly the same", you hold agreement, not evidence. Build the
- control.
+ If the answer is "exactly the same", the number alone does not distinguish the
+ claim from its alternative. Seek a discriminating observation or control.
## Reading any figure
- Any item that fails invalidates the figure until it is fixed.
+ Use the relevant checks below to decide what the figure supports. A limitation
+ may invalidate an interpretation without invalidating the observation itself.
- 1. **An exit code is not a verdict.** pytest exits 4 (usage error, such as a
- path that does not exist) or 5 (nothing collected) when no test ran. Read
- the count line. `no tests ran` is a broken test command, not a pass.
+ 1. **An exit code is not a verdict.** pytest exit 4 indicates a usage error,
+ such as a missing path; exit 5 indicates nothing was collected. Read the
+ collection and result summaries. `no tests ran` does not establish a pass;
+ investigate selection, setup and the change before assigning cause.
2. **Capture the command's own exit code.** In `cmd | tail; echo $?` the
- status is `tail`'s. Use `set -o pipefail`, or write the output to a file
- and read the status before reading the file.
- 3. **Read every artefact.** A result file, a CI summary, a log and a progress
- counter are separate claims about one run. When they disagree, the
+ status is `tail`'s in ordinary shell pipeline behavior. Capture the tested
+ command's status before reading a saved log, or use the shell's pipeline
+ status facility. `set -o pipefail` detects a failing pipeline in shells
+ that support it; its status is not necessarily the tested command's code.
+ 3. **Read the relevant artefacts.** A result file, a CI summary, a log and a
+ progress counter are separate claims about one run. When they disagree, the
disagreement is the finding.
- 4. **A green names its tree.** Quote the branch and commit with every figure
+ 4. **A green names its tree.** Quote the branch and commit with each code-run figure
and say whether the working tree was dirty. A run on main measured main,
not your change.
- 5. **A control is the same experiment**: the same paths, flags, environment
- and machine, a clean tree and a quoted commit. A control that runs a subset
- of the treatment is not a control.
- 6. **Results expire when main moves.** Re-run any red older than the last merge
- before acting on it.
- 7. **A detector that never says no is broken.** If every input is flagged, the
- instrument is at fault until a known-good case answers the other way.
+ 5. **Match the scope of a comparison**: paths, flags, environment and relevant
+ machine conditions, with clean trees and quoted commits. Rerun a narrowed
+ command on both trees before comparing; use the full suite if order or
+ shared state matters. Record unavoidable differences as limits. Matching
+ test names alone does not establish the same failure or exclude extra reds.
+ 6. **A result belongs to the revision measured.** It remains historical
+ evidence when main moves, but a claim about current main needs a fresh run.
+ 7. **An all-positive detector needs investigation.** The inputs may all be
+ affected, or the detector may be wrong. Check a known-good case and inspect
+ its classification before trusting the inferred failure rate.
8. **A count travels with its filter.** Print the query, and every exclusion,
- with the total. Two correct counts of the same thing usually differ by
- filter, not by error.
- 9. **An unreadable value is not a value.** An HTTP 4xx, an empty response, a
- column holding one value on every row and a 0-byte file all return
- something. Print size and modification time with every file you read.
+ with the total, including the unit counted: definitions, collected test
+ cases and executed cases are different quantities. Investigate differences
+ in filters, revisions, inputs and method before attributing a discrepancy.
+ 9. **A returned value needs interpretation.** An HTTP 4xx, an empty response,
+ a constant column or a 0-byte file may be valid for one question and useless
+ for another. Check contents and status; record size and modification time
+ when emptiness or freshness matters to the claim.
10. **Capture the whole output** to a file and read its tail from the file;
- `| tail` keeps the verdict and drops the cause.
+ a live `| tail` may omit the cause and hide progress.
11. **Agreement needs independence.** Two numbers copied from one source agree
with themselves.
- 12. **A hang is not a verdict.** Run the tests one per process under a wall
- clock and name the one that never returns.
+ 12. **A timeout is an observation, not a cause.** Record the deadline and last
+ progress. Isolate tests under a wall-clock limit when useful, but preserve
+ the original suite run: isolation can hide order or shared-state failures.
13. **A quiet log is not a stall.** A pipe ending in `| tail` prints nothing
- until the run exits. Check the process's CPU time before calling it stuck.
+ until the run exits. Check process state and progress before calling it
+ stuck; CPU time alone does not distinguish useful work, waiting or a loop.
## Acting on a count
- A list size is a filter, not a fact. Before acting on "N items are broken":
+ A list size measures the selected set, not necessarily the broken population.
+ Before acting on "N items are broken":
1. **Which filter produced this number?** Quote it with the figure.
2. **Is it one kind of thing?** Group the members by what they are about.
3. **Does one class dominate?** Classify failures by cause and count each
- class. The largest class usually has one mechanism and earns one fix.
+ class. A large class is a lead for a shared mechanism, not proof that one
+ fix covers every member.
4. **Would this detector ever say no?** Run it on a case known to be clean.
A self-selected sample does not predict the rate of the whole list, and a
larger denominator does not repair it; draw the sample differently.
## A reportable figure
- It carries its full command, its branch and commit, whether the tree was dirty,
- its count line, the control's commit for a verdict, the filter for a count, and
- the path of its log. A figure with no receipt is removed, not qualified.
+ For a code-run figure, record its command, branch and commit, whether the tree
+ was dirty, relevant summary and log path. Include scope and filters for counts,
+ both runs for a comparison, and any limits or untested scope. Other figures
+ need their source, date and method. If evidence is unavailable, omit the factual
+ claim or explicitly attribute a report; do not present it as your measurement.