--- name: analytical-troubleshooting description: >- Structured live troubleshooting for deviation problems: something worked (or should work) and now doesn't, and the cause is unknown. Staged method — quick likelihood-weighted tests first, escalating to a comparative IS/IS-NOT specification with distinctions-and-changes analysis when quick tests fail or the search space is large. Influenced by the problem-analysis tradition of Kepner & Tregoe. Works whether the user runs the tests (agent directs, user executes) or the agent can test directly. when_to_use: >- Use whenever the user wants to troubleshoot, diagnose, or narrow down an unexplained fault — "why does X crash / freeze / fail", "works on A but not B", "started failing after...", "never worked right", "intermittent", "flaky", "heisenbug", "we've tried everything", "help me find the cause" — in hardware, servers, networks, Kubernetes, storage, or software. Also use mid-conversation when unstructured guess-and-fix has stalled after a few failed attempts. NOT for post-incident reviews or postmortems of already-resolved incidents (use understanding-human-error), statistical process variation, choosing between design options, or ordinary code debugging where the failure reproduces cheaply on demand (a failing unit test after a commit — bisect and debug directly). --- # Analytical Troubleshooting A staged method for finding the cause of a **deviation**: performance that used to be acceptable (or should be) no longer is, and nobody knows why. The core discipline: track what the problem **IS** and, with equal care, what it plausibly **could be but IS NOT**. A cause that explains only the failures is a guess; a cause that explains the failures *and* the survivals is a diagnosis. Never anchor on the first plausible cause or collect only confirming evidence. Act as process leader and bookkeeper: keep the table, the hypothesis list and the test plan. When a human partner is involved, they are the sensors and hands — ask sharp questions rather than assume their system's state. Why each mechanism is here: `references/evidence.md`. ## Hard rules 1. **Never invent evidence.** Every fact in the analysis carries a provenance tag: `[observed]` (directly seen this session by agent or user), `[reported]` (someone said so), or `[assumed]`. Evidence not obtained from the world must not be recorded as observed. When a test hasn't run yet, its result is unknown — not "probably fine". Tag what the user did not say as `[assumed]`, never `[reported]`. Output from a shell that is not on the affected system (the agent's own sandbox or workstation) is not evidence about it: confirm the probe ran on the faulty object before recording anything, and never mix the two in the analysis. 2. **No fix before a surviving cause.** Do not propose remediation until at least one candidate cause has survived the paper test (Stage 4). Quick *diagnostic* actions are always fine; "let's just reinstall/replace/reboot and hope" is the failure mode this skill exists to prevent. (Exception: genuine safety/stop-the-bleeding containment — say explicitly that it is containment, not diagnosis.) 3. **Every planned test names its targets.** Before running or requesting any test, list each possible result and the candidates that result kills ("counter flat → H2 dead; counter climbs → H1, H4 dead"). A test no result of which kills a candidate is not worth running. 4. **The spec table is the analysis.** Maintain it continuously and re-emit the current table whenever it changes materially or the conversation grows long. Long sessions silently lose mid-context evidence; the re-emitted table is the antidote. For multi-session work, persist it to a file. 5. **If nothing survives, tighten the spec — not the story.** When every candidate fails testing, the specification is missing data. Go get more facts. Do not relax the standard until a pet theory passes. 6. **Never spend the recovery point or the evidence on a theory.** While a cause is untested, refuse any step that destroys the only good copy or the fault's evidence (deleting snapshots, logs or data; wiping or rebuilding over the original), and say so in one sentence. Then deliver the nearest safe version of what was asked, in the form asked for (commands when they asked for commands), before any table or argument. A backup or replica is `[assumed]` until a restore from it has been tested; making it good is data protection — label it that way, not as the fix. ## Stage 0 — Triage Before analyzing anything, make sure there is *one* analyzable problem. - **Separate.** "The cluster is broken" is usually several deviations. Different symptoms, different objects, different timelines → separate analyses. Bundled deviations contaminate each other's evidence; splitting them is often the single move that unsticks a stalled investigation. - **Prioritize** multiple deviations by current impact, urgency, and growth potential. Fix the bleeding first, analyze the important. - **Entry gate.** Full analysis is warranted only if all three hold: 1. There is a real deviation (a stateable SHOULD vs ACTUAL gap). 2. The cause is unknown. 3. Knowing the cause matters for acting well (if a rebuild is cheap and acceptable, do that — say so honestly). Severity does not decide this: a minor fault that keeps eating debugging sessions qualifies; a severe one with an obvious verified cause does not. - **Route non-deviations elsewhere.** Choosing between options is a decision, not a problem. Learning from a resolved incident is a retrospective (hand off to `understanding-human-error`). Process variation ("yield wobbles 2–4%") needs statistical methods. See `references/edge-cases.md` for the boundary map. ## Stage 1 — Fast path (earn the right to skip the full method) Most problems don't deserve the full apparatus. Try to win cheaply first — but under discipline, with a strict exit. - **Known-fault check:** does the symptom match a known issue, recent advisory, or an error message with a well-documented cause? Search first; a recognized problem needs recognition, not analysis. - **Cheap discriminating tests, ordered by probability ÷ cost.** Prefer the test that is most likely to hit, cheapest to run, and — best of all — splits the possibility space (a half-split beats ten one-at-a-time checks in a chain of components; see `references/test-toolkit.md`). - **Fixed windows get a cut line.** When tests must fit a bounded outage, put the restore deadline in the plan and state which steps are dropped or deferred, in order, if time runs short. - **One change per test.** Vary one thing, observe, record. Shotgun changes destroy the evidence Stage 2 will need. Cleaning, reseating, reconnecting and rebooting are changes too: inspect and record first, then alter one element per step. - **Exit condition (count honestly):** after **~3 failed hypotheses**, or when the candidate space is clearly large/opaque, or when any single test is expensive, slow, risky, or the fault won't reproduce on demand — stop guessing and escalate to Stage 2. Log the failed attempts; they become IS-NOT data ("we swapped the PSU and nothing changed" is evidence). ## Stage 2 — Specify Build the comparative specification. This is the heart of the method. **Problem statement first:** one object + one deviation, stated factually ("`backup job nightly-pg` exits 137 on 3 of 12 VMs since 07-19" — not "backups are broken"). Test it: if the statement is already explainable, back up to the thing that *isn't*. A brief why-chain on the symptom helps strip explainable layers before specifying. **The table.** Four dimensions × IS / IS-NOT, plus what distinguishes them: | | IS | COULD BE, but IS NOT | Distinctions / Changes | |---|---|---|---| | **WHAT** — which object, which deviation | | | | | **WHERE** — where observed; where on the object | | | | | **WHEN** — first seen; since then (pattern?); when in lifecycle | | | | | **EXTENT** — how many objects; how bad; how many per object; trend | | | | Rules that make the table work: - **IS-NOT is the closest logical comparison**, not "everything else". The sibling VM that *didn't* fail, the identical node that stayed up, the time window when it *didn't* happen. The closer the comparison, the sharper the boundary around the cause. - **Ask every cell.** First fill every IS-NOT the facts in hand already imply — "the job gets OOM-killed" IS NOT a crash in the job's own code; "only on weekdays" IS NOT weekends; a change already reverted with the fault still present IS NOT that change. Mark the rest "unknown — check: ", never a bare `?`. A skipped cell is a hole the favorite theory will hide in. Emit the unknown cells as one explicit "facts to collect" list beside the table, each with who or what fetches it; a test plan does not replace that list. - **Tag provenance** per hard rule 1. - **Then mine it:** for each IS/IS-NOT pair ask *what is distinctive about the IS side?* — and for each distinction, *what changed in, on, or around it, and when?* Dated changes that line up with the WHEN row are prime cause material. - **Read the WHEN row for the edge case** (`references/edge-cases.md`): first seen ≈ commissioned is a Day One deviation — no change to find, hunt distinctions against a working sibling only; a gradual decline is drift — do not seize the most visible recent change; an intermittent fault is specified as occurrences vs. non-occurrences over time. Full question set, worked example, and table-maintenance guidance: `references/specification.md`. When facts are in logs, compress them first (the `lessence` skill pairs well) and transcribe findings into cells with provenance. **With a human partner:** request facts one dimension at a time — a focused question set they can actually answer — never a 20-question dump. ## Stage 3 — Hypothesize Generate **3–5 candidate causes in parallel** — "candidate", "hypothesis", and H1..Hn all name the same thing here — each stated as a **mechanism**, not a component: not "the switch", but "switch port MTU dropped below tunnel overhead, so large packets silently fragment and the session stalls". A mechanism can be tested; a pointed finger cannot. Sources, in order: 1. **Knowledge and experience** — cast a wide net from the agent's and the user's knowledge of systems like this. Fastest when it works. 2. **Distinctions and changes** — derive causes mechanically from the table when brainstorming yields nothing plausible, too much, or only candidates that fail Stage 4. Parallel matters: a single cherished hypothesis is the documented anchor-trap. Keep a live numbered list (H1..Hn) with status (alive / refuted / verified) and tag every piece of evidence with the hypotheses it bears on. **State each mechanism at the weakest level the table forces.** The spec sets a floor on specificity (the candidate must explain every IS and spare every IS-NOT) and testability keeps it refutable — but do not exceed that floor. Every detail not purchased by a table cell is an unforced commitment that makes the hypothesis strictly less probable while making the story *feel* more explanatory (the conjunction fallacy). "Something in the upgraded DB version raises peak memory" beats "the v16 parallel-dump default doubles work_mem" until a cell forces the narrower claim. ## Stage 4 — Paper test (kill candidates before spending on tests) For each candidate, walk the full table: *"If H3 is the cause, how does it explain each IS — **and** each IS-NOT?"* - A candidate that requires the IS-NOT side to be false is refuted by evidence already in hand. Kill it. This is free — no lab time spent. - A change dated after onset cannot be the originating cause, and a change applied everywhere cannot alone explain a fault seen only somewhere: mark it refuted, not "ranked last". If the refuting date or location is doubtful, verifying it goes on the data-collection list; the candidate stays dead until that check reverses it. Only an `[assumed]` cell fails to refute. - A candidate that fits only with extra assumptions survives *provisionally*; write the assumptions down. Rank survivors **weakest first**: the candidate that commits to the least beyond what the table forces, counting each written-down assumption as one more unforced commitment. - The top survivor is the **most probable cause** — a rank, not a verdict. Paper never proves; it only prunes. If *every* candidate dies: hard rule 5 — the spec is missing a distinction or a change. Collect more facts; consider whether two problems are bundled under one name (back to Stage 0). ## Stage 5 — Verify in the world Confirm the most probable cause with the **safest, surest, cheapest, fastest** real-world check available: - **Observe** the mechanism in action (capture the packet, watch the counter, catch the OOM kill in the log). - **Experiment**: swap the suspect component, or better, **reverse the suspected change** and watch the problem stop — then, ideally, re-apply it and watch the problem return (the strongest evidence there is). - **Fix and monitor** when direct observation is impossible — apply the fix that follows from the mechanism and watch the *specific* symptom, over a window long enough to be meaningful (for intermittent faults: several times the longest observed gap between occurrences). - When the evidence is destroyed or unreachable, verify the *assumptions* the candidate depends on instead. Physical/production actions belong to the human partner when one is present; the agent specifies exactly what to do and what result each outcome would imply; the human executes and reports. A fix that merely coincides with recovery is unverified, not done. Close the loop: state the verified cause, the fix, and update the spec table one last time showing the cause explaining every row. If the analysis revealed contributing conditions worth systemic attention, note the handoff to a proper retrospective — don't do it here. ## Escalation and red flags - **Escalate to the human** after 3 failed *verification* attempts (not paper kills — those are progress), or when the surviving cause implies a design problem rather than a fault. - Red flags that mean *return to the method*, spoken by either partner: "it's probably just X", "let's try reinstalling everything", "must be a bug" (without a mechanism), proposing a second fix while the first is unverified, an IS-NOT cell contradicting a hypothesis everyone still likes, three swaps in a row with no new table entries, a hypothesis more specific than any cell forces (the vivid detailed story is the anchor-trap wearing its best clothes). ## References - `references/specification.md` — full question set per dimension, worked example, provenance and table-maintenance discipline. Read when entering Stage 2. - `references/edge-cases.md` — Day One deviations, gradual drift, intermittent faults, multi-causal/systemic boundaries, handoff map. Read when the standard change-hunt feels wrong. - `references/test-toolkit.md` — test-selection math (probability ÷ cost), half-split, families-of-variation pruning, one-variable discipline, delta debugging, best-vs-worst comparison. Read when choosing what to test next. - `references/evidence.md` — why each mechanism is in the skill, with the research behind it. Read when curious or when adapting the method. - `references/sources.md` — per-URL index behind evidence.md, with its `Freshened:` stamp. No need to read during troubleshooting.