diagnosing-bugs · diff
git:20260801.8234b5f to git:20261029.276ae81
94 added, 81 removed. Audit A to A.
---
name: diagnosing-bugs
- description: "Diagnose a defect whose root cause is unknown through reproducible investigation. Use for regressions, crashes, flaky failures, or performance problems. Not for an obvious local correction, feature work, or architecture review."
+ description: Diagnose a difficult reproducible bug or performance regression through a feedback loop, competing hypotheses, and regression proof.
---
- # Diagnose Bugs
+ Read [local host integration](../.shared/host-integration.md) before applying this imported workflow.
- Find and prove the root cause before changing production behavior. Drive the investigation with the fastest reliable signal that exercises the reported symptom.
+ # Diagnosing Bugs
- Read `references/evidence.md` for difficult reproduction, minimization, bisection, or causal isolation. If delegation is explicitly authorized, apply `delegate-work` only after establishing one shared reproduction signal; assign distinct hypotheses or boundaries, never parallel speculative fixes to the same code.
+ A discipline for hard bugs. Skip phases only when explicitly justified.
- ## Respect the Diagnostic Contract
+ When exploring the codebase, read `GLOSSARY.md` (if it exists) to get a clear mental model of the relevant modules, and check ADRs in the area you're touching.
- - For a diagnosis-only request, work read-only apart from temporary diagnostic artifacts and report the proven cause without implementing a fix.
- - For a fix request, continue through the smallest root-cause correction, regression coverage, and verification.
- - Preserve unrelated user changes and avoid broad refactoring during diagnosis.
- - Keep temporary instrumentation easy to identify and remove it before completion.
- - Do not confuse correlation, a silenced symptom, or a passing retry with a root cause.
+ ## Redact
- ## Establish the Exact Symptom
+ This skill has you show commands, outputs and captured artifacts. **Redact every secret first**: write `<REDACTED>` in its place. Build loops against env vars, so the credential stays in the environment rather than in what you show. Captured artifacts carry auth headers: quote only the lines that carry the signal.
- Determine:
+ If the redacted output is not enough to diagnose the bug, say so and ask the user.
- - the expected and observed behavior;
- - the affected environment, version, inputs, and frequency;
- - the last known-good state when available;
- - the precise user-visible failure or performance threshold;
- - the existing commands, tests, logs, traces, and recent changes relevant to the path.
+ ## Phase 1: Build a feedback loop
- Inspect the repository and runtime before asking the user. Ask only for information or access that cannot be discovered safely.
+ **This is the skill.** Everything else is mechanical. If you have a **tight** pass/fail signal for the bug (one that goes red on _this_ bug), you will find the cause; bisection, hypothesis-testing, and instrumentation all just consume it. If you don't have one, no amount of staring at code will save you.
- ## Build a Red-Capable Loop
+ Spend disproportionate effort here. **Be aggressive. Be creative. Refuse to give up.**
- Create the narrowest command that exercises the actual failing path and produces an unambiguous pass or fail signal. Prefer, in order of fit:
+ ### Ways to construct one, in roughly this order
- - an existing focused test;
- - a new regression test;
- - a CLI or API invocation;
- - a minimal browser or integration flow;
- - a replayed trace or fixture;
- - a benchmark with a defined threshold;
- - a temporary harness;
- - an automated version or commit bisection.
+ 1. **Failing test** at whatever seam reaches the bug: unit, integration, e2e.
+ 2. **Curl / HTTP script** against a running dev server.
+ 3. **CLI invocation** with a fixture input, diffing stdout against a known-good snapshot.
+ 4. **Headless browser script** (Playwright / Puppeteer) that drives the UI and asserts on DOM/console/network.
+ 5. **Replay a captured trace.** Save a real network request / payload / event log to disk; replay it through the code path in isolation.
+ 6. **Throwaway harness.** Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
+ 7. **Property / fuzz loop.** If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
+ 8. **Bisection harness.** If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can `git bisect run` it.
+ 9. **Differential loop.** Run the same input through old-version vs new-version (or two configs) and diff outputs.
+ 10. **HITL bash script.** Last resort. If a human must click, drive _them_ with `scripts/hitl-loop.template.sh` so the loop is still structured. Captured output feeds back to you.
- Run the loop before forming a detailed theory. Confirm that it fails for the reported symptom rather than a setup error or nearby defect.
+ Build the right feedback loop, and the bug is 90% fixed.
- Make the loop:
+ ### Tighten the loop
- - deterministic when possible;
- - fast enough to run repeatedly;
- - independent of unnecessary services and state;
- - explicit about inputs, output, exit status, and timing.
+ Treat the loop as a product. Once you have _a_ loop, **tighten** it:
- For intermittent failures, increase the reproduction rate through repetition, controlled concurrency, stress, or captured state. Record the observed rate and do not call a flaky run fixed after one pass.
+ - Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
+ - Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
+ - Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)
- If no executable reproduction is possible, document the exact evidence gap and use the strongest available observation. Do not overstate confidence.
+ A 30-second flaky loop is barely better than no loop; a 2-second deterministic one is tight, a debugging superpower.
- ## Minimize the Failure
+ ### Non-deterministic bugs
- Remove unrelated inputs, components, timing, and environmental variation while preserving the symptom. Trace the data and control flow across the smallest set of boundaries that can explain it.
+ The goal is not a clean repro but a **higher reproduction rate**. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not, so keep raising the rate until it's debuggable.
- Use characterization tests when legacy behavior must be understood before it can be changed. Compare against a known-good version, environment, or implementation when that produces a stronger signal.
+ ### When you genuinely cannot build a loop
- ## Test Ranked Hypotheses
+ Stop and say so explicitly. List what you tried. Ask the user for: (a) access to whatever environment reproduces it, (b) a redacted captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or (c) permission to add temporary production instrumentation. Do **not** proceed to hypothesise without a loop.
- Maintain a short ranked list of falsifiable hypotheses. For each hypothesis, state:
+ ### Completion criterion: a tight loop that goes red
- - the mechanism that would produce the exact symptom;
- - the evidence already supporting or contradicting it;
- - the smallest observation or experiment that can disprove it.
+ Phase 1 is done when the loop is **tight** and **red-capable**: you can name **one command** (a script path, a test invocation, a curl) that you have **already run at least once** (show the invocation and its output, redacted), and that is:
- Test one meaningful variable at a time. Prefer boundary observations, assertions, traces, counters, and targeted state inspection over broad logging.
+ - [ ] **Red-capable**: it drives the actual bug code path and asserts the **user's exact symptom**, so it can go red on this bug and green once fixed. Not "runs without erroring"; it must be able to _catch this specific bug_.
+ - [ ] **Deterministic**: same verdict every run (flaky bugs: a pinned, high reproduction rate, per above).
+ - [ ] **Fast**: seconds, not minutes.
+ - [ ] **Agent-runnable**: you can run it unattended; a human in the loop only via `scripts/hitl-loop.template.sh`.
- Tag temporary diagnostics distinctly so they can be found and removed. Do not retain sensitive values in logs or artifacts.
+ If you catch yourself reading code to build a theory before this command exists, **stop: jumping straight to a hypothesis is the exact failure this skill prevents.** No red-capable command, no Phase 2.
- Revise the ranking after every result. Reject disproven hypotheses explicitly and avoid returning to them without new evidence.
+ ## Phase 2: Reproduce + minimise
- ## Prove the Root Cause
+ Run the loop. Watch it go red as the bug appears.
- Require a causal explanation that connects:
+ Confirm:
- 1. the triggering condition;
- 2. the incorrect state transition or computation;
- 3. the observed symptom;
- 4. the evidence that isolates this mechanism from alternatives.
+ - [ ] The loop produces the failure mode the **user** described, not a different failure that happens to be nearby. Wrong bug = wrong fix.
+ - [ ] The failure is reproducible across multiple runs (or, for non-deterministic bugs, reproducible at a high enough rate to debug against).
+ - [ ] You have captured the exact symptom (error message, wrong output, slow timing) so later phases can verify the fix actually addresses it.
- When practical, demonstrate that controlling the causal condition predictably turns the failure off and back on. Do not start a production fix until the evidence supports the mechanism.
+ ### Minimise
- ## Implement the Smallest Complete Fix
+ Once it's red, shrink the repro to the **smallest scenario that still goes red**. Cut inputs, callers, config, data, and steps **one at a time**, re-running the loop after each cut, and keep only what's load-bearing for the failure.
- - Correct the causal mechanism rather than suppressing its output.
- - Preserve public behavior and compatibility outside the requested correction.
- - Avoid retries, sleeps, broad exception handling, weakened assertions, or raised limits unless evidence shows they are the correct design.
- - Add or update a regression test that fails before the fix and passes after it when the repository has an appropriate test surface.
- - Apply the active TDD workflow for the regression cycle when available.
- - Keep cleanup and unrelated architectural improvements out of the fix.
+ Why bother: a minimal repro shrinks the hypothesis space in Phase 3 (fewer moving parts left to suspect) and becomes the clean regression test in Phase 5.
- For a performance defect, compare equivalent workloads with repeated measurements, account for warm-up and variance, and report both baseline and corrected results.
+ Done when **every remaining element is load-bearing**: removing any one of them makes the loop go green.
- ## Verify and Clean Up
+ Do not proceed until you have reproduced **and** minimised.
- Run the focused reproduction first, then expand to relevant tests, type checks, linters, builds, and realistic runtime checks in proportion to risk.
+ ## Phase 3: Hypothesise
- Confirm:
+ Consider competing, falsifiable hypotheses before testing when the cause is genuinely uncertain. Choose as many as the evidence warrants rather than forcing a quota.
- - the original symptom no longer reproduces under the same conditions;
- - the regression test detects the unfixed behavior;
- - important neighboring behavior remains intact;
- - intermittent or performance results meet the defined confidence threshold;
- - all temporary instrumentation and artifacts are removed;
- - the final diff contains only the intended correction and coverage.
+ Each hypothesis must be **falsifiable**: state the prediction it makes.
- ## Present Diagnostic Evidence
+ > Format: "If <X> is the cause, then <changing Y> will make the bug disappear / <changing Z> will make it worse."
- For diagnosis-only work, report:
+ If you cannot state the prediction, the hypothesis is a vibe: discard or sharpen it.
- - the reproduced symptom and command;
- - the proven root cause and causal evidence;
- - rejected material hypotheses when useful;
- - the smallest recommended correction;
- - remaining uncertainty or unavailable verification.
+ Rank hypotheses using available deployment history, logs, and prior conversation before testing. Report a hypothesis only when it materially explains the next diagnostic step. Do not require an intermediate user checkpoint.
- For completed fixes, follow the active final-report instructions and include the reproduction and verification evidence.
+ ## Phase 4: Instrument
- ## Completion Standard
+ Each probe must map to a specific prediction from Phase 3. **Change one variable at a time.**
- Finish only when the reported symptom is tied to a demonstrated causal mechanism and the requested fix, if any, is covered by evidence that would fail on the original defect.
+ Tool preference:
+
+ 1. **Debugger / REPL inspection** if the env supports it. One breakpoint beats ten logs.
+ 2. **Targeted logs** at the boundaries that distinguish hypotheses.
+ 3. Never "log everything and grep".
+
+ **Tag every debug log** with a unique prefix, e.g. `[DEBUG-a4f2]`. Cleanup at the end becomes a single grep. Untagged logs survive; tagged logs die.
+
+ **Perf branch.** For performance regressions, logs are usually wrong. Instead: establish a baseline measurement (timing harness, `performance.now()`, profiler, query plan), then bisect. Measure first, fix second.
+
+ ## Phase 5: Fix + regression test
+
+ Write the regression test **before the fix**, but only if there is a **correct seam** for it.
+
+ A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call site. If the only available seam is too shallow (single-caller test when the bug needs multiple callers, unit test that can't replicate the chain that triggered the bug), a regression test there gives false confidence.
+
+ **If no correct seam exists, that itself is the finding.** Note it. The codebase architecture is preventing the bug from being locked down. Flag this for the next phase.
+
+ If a correct seam exists:
+
+ 1. Turn the minimised repro into a failing test at that seam.
+ 2. Watch it fail. If you forced the red by mutating code or a fixture, `diff` against a pristine copy to prove the mutation landed before you trust it.
+ 3. Apply the fix.
+ 4. Watch it pass.
+ 5. Re-run the Phase 1 feedback loop against the original (un-minimised) scenario.
+
+ ## Phase 6: Cleanup
+
+ Required before declaring done:
+
+ - [ ] Original repro no longer reproduces (re-run the Phase 1 loop)
+ - [ ] Regression test passes (or absence of seam is documented)
+ - [ ] All `[DEBUG-...]` instrumentation removed (`grep` the prefix)
+ - [ ] Task-owned throwaway prototypes and fixtures removed once no process needs them
+ - [ ] The established cause is reported in chat, or in a commit or PR description only when that operation is requested