sensemaking · diff
git:20260716.0e01ec7 to git:20260921.9b3de7c
44 added, 0 removed. Audit A to A.
---
name: sensemaking
description: "Check whether findings make sense before trusting or reporting them by comparing to the nearest reference, asking what mechanism could explain them, and naming a falsifier. Use when the agent is interpreting results, summarizing progress, assessing regressions, or deciding whether an observation should change the plan."
---
# Sensemaking
## Use It As A Default Habit
- Compose this skill with domain skills instead of replacing them.
- Use it whenever you are about to trust, summarize, or act on a finding.
- Default question: does this result actually make sense, and if so, what should change because of it?
## Use This Output Contract
When using this skill, structure the reasoning as:
1. Observation: what happened.
2. Expected: what you thought would happen instead.
3. Nearest reference: the baseline, sibling case, prior run, known-good path, or control.
4. Best explanation: the most plausible mechanism right now.
5. Falsifier: one observation that would seriously weaken that explanation.
6. Decision impact: what changes, or why nothing changes.
7. Baseline disposition: whether the nearest valid baseline remains active or
has been superseded, and by what evidence.
8. Falsification scope: the exact hypothesis or changed delta rejected, plus
the parent method or unchanged components that remain live.
## Apply These Rules
- Separate observation from interpretation.
- Compare against the nearest useful reference, not an abstract ideal.
- Say why the chosen reference is the right baseline or control; if no fair baseline exists, weaken the conclusion.
- Prefer mechanistic explanations over labels such as "noisy", "unstable", or "weird".
- When the mechanism is spatial, temporal, structural, or artifact-like, inspect or create the nearest useful visual comparison instead of relying on scalar metrics alone.
- Ask whether the finding is strong enough to change the next action.
- If the result does not change the decision, say so explicitly.
- If the result contradicts the current story, update the story.
## Preserve The Best Baseline
- Keep the best valid baseline active until a prospectively defined successor
beats it on the same decision readouts.
- Attribute a failure only to what changed relative to that baseline. Failure
of an additive rescue rejects the rescue or combination, not the retained
baseline.
- Before pivoting, name the baseline, changed delta, result, exact hypothesis
falsified, hypotheses not falsified, and retained active path.
- Scope stop rules narrowly. Do not retire a successful parent method because
a broader extension or challenger failed unless that parent was directly
retested and failed.
## Sanity-Check Findings
- Ask whether the effect size matches the claimed explanation.
- Ask whether the result is consistent across the closest comparable cases.
- Ask whether the evidence source is trustworthy enough for the claim being made.
- Ask whether a different explanation fits the evidence as well or better.
- Ask what would have to be true for this result to be misleading.
## Avoid Weak Sensemaking
- Do not report a metric or image difference without saying what it means.
- Do not prefer a summary metric over a decisive overlay, curve, slice, diff image, or side-by-side view when the mechanism is easier to see than describe.
- Do not narrate confidence beyond what the evidence supports.
- Do not treat one surprising result as a new truth without comparison.
- Do not stop at "better" or "worse"; say in what way and relative to what.
## End With A Decision
- If the finding is believable and consequential, state the changed plan.
- If the finding is believable but not consequential, state that no plan change is warranted.
- If the finding is not yet believable, state the next falsifier or sanity check.
+
+ ## Verify The Element Set Before Comparing Numbers
+
+ Verify a matched element set through a quantity that depends **only on the target**,
+ computed over the whole set by each side independently: the target's mean square, or
+ a rank-k approximation of it.
+
+ Item count, support, and scoring surface can all agree while the roster silently
+ differs, and every model-dependent number stays plausible when it does. A
+ target-only quantity cannot differ between two owners scoring the same elements, so
+ it fails loudly and immediately when they are not. Two owners once disagreed on a
+ reference by 0.024 with matched counts and matched surfaces, because their
+ selections overlapped on 12 of 48 rows; the rank-1 residual is what exposed it, and
+ on the shared 12 they agreed to 5e-16.
+
+ Join on physical identity rather than on a positional index. An index is renumbered
+ per artifact, and a wrong join returns a plausible number rather than an error.
+
+ ## Compare Matched Units, Not Aggregates
+
+ Per-unit losses and errors are usually heavy-tailed, so a mean over a window is
+ dominated by whichever few hard units it happens to contain. Pair by unit, report
+ medians and a paired ratio, and treat a difference that survives neither as absent.
+
+ ## Read The Units Before Converting
+
+ - Check whether a field is a unit fraction or already in percentage points before
+ converting — a `_frac` and a `_pct` suffix mean different things, and only the
+ metric's owner settles it. For decision-relevant values, report both the raw
+ fraction and the percentage explicitly.
+ - When merging CSV-backed metrics, select numeric cells value by value rather than
+ inferring a whole column from one row. Missing string identifiers deserialize as
+ `NaN` and must never enter a numeric aggregation.
+
+ ## Make QC Galleries Show The Edges
+
+ - A distribution or gate QC gallery must deterministically include the best example,
+ the nearest below threshold, the nearest above threshold, and the worst, within
+ each major stratum. The first few examples in lexicographic order are not enough.
+ - A renderer working in a native coordinate system must not assume a fixed array
+ axis is the one to slice along. Choose a readable plane from the actual geometry,
+ and verify explicitly that anisotropic inputs do not collapse into unreadable
+ narrow panels.
+
## References
- Read `references/checks.md` for reusable prompts and compact response templates.