systematic-debugging git:20261101.bb8b61dA

Immutable. This exact content never changes and is served at /api/v1/blob/d0beb2a64f177307.

---
name: systematic-debugging
description: Debugs by evidence instead of guessing. Use whenever a test fails, behavior regressed, a bug is intermittent or environment-specific, or the user needs to find which commit introduced a defect, even if they only say "this is broken" or "why does this fail". Reproduces, isolates and proves the root cause, then locks the fix with a regression test. Uses git bisect, blame and reflog.
---

# systematic-debugging

Debug by evidence, not by guessing. Each step narrows the search space until the root cause is proven and a regression test locks the fix.

## When to use

- A test fails or behavior regressed
- A bug is intermittent or environment-specific
- You need the exact commit that introduced a defect

## Method

1. **Reproduce**: build the smallest reliable failing case
2. **Observe**: collect logs, stack traces, inputs and versions
3. **Hypothesize**: write 2 to 3 candidate causes, ranked by likelihood
4. **Isolate**: bisect code, inputs or history to test each hypothesis
5. **Prove**: confirm the root cause with a failing test or a printed fact
6. **Fix**: change the minimum necessary
7. **Guard**: add a regression test, then run the full suite
8. **Record**: put cause and fix in the commit message

## Git-powered tools

```bash
# Find the commit that introduced the bug
git bisect start
git bisect bad HEAD
git bisect good v1.2.0
git bisect run ./test.sh      # automated bisect
git bisect reset

# Who changed this line, and why
git blame -L 40,60 path/to/file
git log -S "functionName" --oneline     # when a string was added or removed
git log -p -- path/to/file

# Recover lost work or inspect past states
git reflog
git stash push -m "wip debugging"
git worktree add ../repro <commit>      # reproduce in a clean checkout
```

## Instrumentation tips

```bash
# Shell scripts
set -euxo pipefail

# Compare working vs broken environments
diff <(env | sort) <(ssh host env | sort)
```

```python
import logging
logging.basicConfig(level=logging.DEBUG)
# breakpoint()  # drop into pdb at the suspicious line
```

## Rules of thumb

- Change one variable at a time
- If you cannot reproduce it, you cannot claim to have fixed it
- Verify that the code you edit is the code that actually runs
- After 30 minutes without progress, explain the problem in writing (rubber duck)
- Never commit a fix without a test or a documented manual proof

## Bug report template

```markdown
**Symptom:** what was observed
**Expected:** what should happen
**Repro steps:** numbered, minimal
**Root cause:** one paragraph with evidence
**Fix:** what changed and why
**Regression test:** file and test name
```

## Related skills

- `commitsmith`: write the commit message for the fix, naming the root cause
- `pdf-processing`: an example pipeline that benefits from verification-first debugging