cf-sys-debug · diff
git:20260827.debd979 to git:20260827.45d78ea
95 added, 116 removed. Audit A to A.
---
name: cf-sys-debug
description: >
Systematic 4-phase debugging — root cause, hypothesis testing, regression-guarded fix,
mandatory bug doc. Auto-invoke for non-trivial or recurring bugs — signals: "still
broken", "same error again", "came back", flaky/intermittent/race, "used to work, now
broken", hard-to-reproduce, works locally fails in CI, "find the root cause",
"investigate", "diagnose", "why is this happening". Prefer over cf-fix for hard bugs. Do
NOT auto-invoke for trivial typos, one-line fixes, or obvious config errors.
created: 2026-02-17
updated: 2026-08-27
disable-slash-command: true
---
# Systematic Debugging
> **CLI Requirement:** OPTIONAL — Uses the memory MCP from `coding-friend-cli` for fast indexed search and storage. Without the CLI: falls back to grep over `docs/memory/` and direct file writes. Full functionality preserved, slower memory recall. See [CLI requirements](../../../docs/cli-requirements.md).
## Custom Guide
- Custom guide — auto-loaded below (if the raw command shows instead of its output, run it yourself):
-
```!
bash "<plugin-root>/lib/load-custom-guide.sh" cf-sys-debug
```
- If output is not empty, integrate returned sections: `## Before` → before first step, `## Rules` → apply throughout, `## After` → after final step.
+ If output is not empty: `## Before` → before first step, `## Rules` → throughout, `## After` → after final step.
## Core Constraint
**Do not touch code until you can state the root cause in one sentence:**
> "I believe the root cause is [X] because [evidence]."
- Name a specific file, function, and line. "A state management issue" is not testable. "Stale cache in `useUser` at `src/hooks/user.ts:42` because the dependency array is missing `userId`" is testable. If you cannot be that specific, you do not have a hypothesis yet.
+ Name a specific file, function, and line. Vague labels are not testable. If you cannot be that specific, you do not have a hypothesis yet.
- **Same symptom after a fix = hard stop.** Both a recurrence and "let me just try this" mean the hypothesis is unfinished. Re-read the execution path from scratch before touching code again.
+ **Same symptom after a fix = hard stop.** Recurrence or "let me just try this" means the hypothesis is unfinished. Re-read the execution path from scratch before touching code again.
- **After 3 failed hypotheses, stop.** Use the Handoff Format below to surface what was checked, ruled out, and unknown. Ask how to proceed.
+ **After 3 failed hypotheses, stop.** Use the Handoff Format below. Ask how to proceed.
## Rationalization Watch
- When these surface, stop and re-examine:
+ Stop and re-examine when these surface:
- | Thought | What it means | Rule |
- | ----------------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------- |
- | "I'll just try this one thing" | No hypothesis, random-walking | Stop. Write the hypothesis first. |
- | "I'm confident it's X" | Confidence is not evidence | Run an instrument that proves it. |
- | "Probably the same issue as before" | Treating a new symptom as a known pattern | Re-read the execution path from scratch. |
- | "It works on my machine" | Environment difference IS the bug | Enumerate every env difference before dismissing. |
- | "One more restart should fix it" | Avoiding the error message | Read the last error verbatim. Never restart more than twice without new evidence. |
+ | Thought | Rule |
+ | ----------------------------------- | ------------------------------------------------------------------- |
+ | "I'll just try this one thing" | Write the hypothesis first |
+ | "I'm confident it's X" | Confidence is not evidence — instrument it |
+ | "Probably the same issue as before" | Re-read the execution path from scratch |
+ | "It works on my machine" | Environment difference IS the bug — enumerate every env difference |
+ | "One more restart should fix it" | Read the last error verbatim. Max two restarts without new evidence |
## Progress Signals
- When these appear, the diagnosis is moving in the right direction:
+ Diagnosis is moving when:
- | Thought | What it means | Next step |
- | ---------------------------------------------------- | ----------------------------------- | ------------------------------------------------------------- |
- | "This log line matches the hypothesis" | Positive evidence found | Find one more independent piece of evidence to cross-validate |
- | "I can predict what the next error will be" | Mental model is forming | Run the prediction; if it matches, the model is correct |
- | "Root cause is in A but symptoms appear in B" | Propagation path understood | Trace the call chain from A to B and confirm each link |
- | "I can write a test that would fail on the old code" | Hypothesis is specific and testable | Write the test before applying the fix |
+ - A log line matches the hypothesis → find one more independent piece of evidence
+ - You can predict the next error → run the prediction
+ - Cause is in A, symptoms in B → confirm each link in the A→B chain
+ - You can write a test that would fail on the old code → write it before the fix
- Do not claim progress without observable evidence matching at least one of these signals.
+ Do not claim progress without observable evidence matching at least one signal.
## 4-Phase Process + Documentation
### Phase 1: Root Cause Investigation
**1a. Check existing bug docs** (memory recall):
- Before investigating, search for related past bugs. Extract 2-3 keywords from the bug description.
+ Extract 2–3 keywords.
- **Primary — Memory MCP** (if `memory_search` tool is available):
- Call `memory_search` with: `{ "query": "<bug keywords>", "type": "episode", "limit": 3 }`
+ **Primary — Memory MCP** (if `memory_search` is available):
+ `{ "query": "<bug keywords>", "type": "episode", "limit": 3 }`
- **Fallback — grep** (if memory MCP unavailable):
- Check `{docsDir}` from `.coding-friend/config.json` (default: `docs`).
+ **Fallback — grep** (`{docsDir}` from `.coding-friend/config.json`, default `docs`):
- 1. Grep `^description:` lines across `{docsDir}/memory/bugs/**/*.md` — match against bug keywords
- 2. If no match, grep `^tags:` lines across `{docsDir}/memory/bugs/**/*.md`
+ 1. Grep `^description:` in `{docsDir}/memory/bugs/**/*.md`
+ 2. Else grep `^tags:`
- If matches found, read the top 1-2 matched files — they may reveal known root causes or patterns that save investigation time.
+ Read the top 1–2 matches.
**1b. Investigate:**
- 1. **Read the actual error.** Do not guess. Read the full stack trace, error message, and logs.
- 2. **Reproduce the bug.** Write a test or command that triggers the failure reliably.
- 3. **Trace backward.** Follow the call chain from the error to its origin. The bug is usually NOT where the error appears.
+ 1. **Read the actual error** — full stack, message, logs. Do not guess.
+ 2. **Reproduce** with a reliable test or command.
+ 3. **Trace backward** from the error to its origin.
### Phase 2: Pattern Analysis
- 1. **When did it start?** Check recent changes: `git log --oneline -20`, `git diff HEAD~5`
- 2. **Is it consistent?** Does it fail every time, or intermittently? Intermittent = timing/state issue.
- 3. **What's the minimal reproduction?** Strip away everything unrelated until you have the smallest case.
- 4. **Pay attention to deflection.** When someone says "that part doesn't matter," treat it as a signal. The area someone avoids examining is often where the problem lives.
+ 1. **When did it start?** `git log --oneline -20`, `git diff HEAD~5`
+ 2. **Consistent or intermittent?** Intermittent = timing/state.
+ 3. **Minimal reproduction** — strip everything unrelated.
+ 4. **Deflection** — an area someone dismisses is often where the problem lives.
### Bisect Mode
- Activate when the symptom is "used to work, now broken" or "broke after an update". Random-walking forward from the current state wastes context and produces random fixes.
-
- **Flow:**
+ Activate for "used to work, now broken" or "broke after an update".
- 1. Find `last-known-good` using the most recent tag where the behavior was correct: `git tag --sort=-version:refname | head -5`. Do not use a date or raw SHA as the anchor.
- 2. Define a pass/fail test command before starting. It must be runnable non-interactively and produce an unambiguous exit code. Write it once; reuse it at every step.
- 3. Run `git bisect start`, `git bisect bad` (current), `git bisect good <tag>`. Let bisect drive; do not jump ahead.
- 4. Context conservation: do not re-read large files at each step. Read once, note the key function or line, reference from notes.
- 5. When bisect names the culprit commit: read only that commit's diff, not surrounding history. Identify the specific line that introduced the regression.
+ 1. Anchor `last-known-good` on the most recent good tag: `git tag --sort=-version:refname | head -5`. Do not use a date or raw SHA.
+ 2. Define a non-interactive pass/fail command with a clear exit code. Reuse it at every step.
+ 3. `git bisect start`, `git bisect bad` (current), `git bisect good <tag>`. Let bisect drive.
+ 4. Do not re-read large files each step — note the key function/line once.
+ 5. When bisect names the culprit: read only that commit's diff; identify the introducing line.
### Phase 3: Hypothesis Testing
- 1. **Form one hypothesis** using the template: "The bug is caused by [X] because [evidence]." Name file and line.
- 2. **Add one targeted instrument:** a log line, a failing assertion, or the smallest test that would fail if the hypothesis is correct. Run it.
- 3. **If the evidence contradicts the hypothesis, discard it completely.** Re-orient with what was just learned. Do not preserve a hypothesis the evidence disproves.
- 4. **External tool failure: diagnose before switching.** When an MCP tool or API fails, determine why first (server running? API key valid? config correct?) before trying an alternative.
- 5. **Stack trace points into a library?** Walk back 3 frames into your own code. The bug is almost always there, not in the dependency.
+ 1. **One hypothesis:** "The bug is caused by [X] because [evidence]." Name file and line.
+ 2. **One instrument:** log, assertion, or smallest test that would fail if correct. Run it.
+ 3. **Evidence contradicts → discard completely.** Re-orient. Do not keep a disproved hypothesis.
+ 4. **External tool/API failure:** diagnose first (server, key, config) before switching tools.
+ 5. **Stack trace in a library?** Walk back 3 frames into your code. The bug is almost always there.
### Phase 4: Implementation
- 1. **Fix the root cause**, not the symptom. If the fix touches more than 5 files, pause and confirm scope with the user.
- 2. **Regression Guard** — for any bug that recurred or was previously "fixed", the fix is not done until:
- - A regression test exists that fails on the unfixed code and passes on the fixed code
- - The test lives in the project's test suite, not a temporary file
- - The commit message states why the bug recurred and why this fix prevents it
- 3. **Write a regression test** that would have caught this bug
- 4. **Run the full test suite** — your fix must not break anything else
- 5. **Verify the original error is gone** — reproduce the original failure and confirm it's fixed
+ 1. **Fix the root cause**, not the symptom. >5 files → pause and confirm scope.
+ 2. **Regression Guard** — if the bug recurred or was previously "fixed":
+ - A regression test that fails unfixed and passes fixed
+ - Lives in the project suite (not a temp file)
+ - Commit message states why it recurred and why this fix prevents it
+ 3. **Write a regression test** that would have caught this
+ 4. **Full test suite** — no collateral breakage
+ 5. **Verify the original error is gone**
#### Capturing out-of-scope side-effects
- While investigating or fixing, if you uncover a problem **unrelated to the root cause under investigation** that is non-trivial (addressing it inline would derail this fix), do NOT fix it now. Record it for later, then continue:
+ Non-trivial problem **unrelated to the root cause** → do not fix inline. Record it, then continue:
```bash
bash "<plugin-root>/lib/capture-later.sh" \
--name "<short title>" --description "<what & where — enough to act on cold>" \
--source cf-sys-debug [--slug <bug-doc/task slug, if one exists>] [--problem "<the bug under investigation>"]
```
- This writes `<docsDir>/later/YYYY-MM-DD-<name>.md` with frontmatter (slug, problem, conversation_id). This is an in-repo audit trail, independent of the `spawn_task` tool.
+ Writes `<docsDir>/later/YYYY-MM-DD-<name>.md` (frontmatter: slug, problem, conversation_id).
### Phase 5: Document the Bug
- cf-sys-debug is only invoked for hard bugs — always document the findings.
+ Hard bugs always get a doc.
- 1. Read `language` config (local `.coding-friend/config.json` overrides global, default: `en`)
- 2. Use `MAIN_REPO_ROOT` from the SessionStart bootstrap context (injected via session-init.sh). If absent, fall back to running `pwd` for `$CWD` and use `$CWD` as `MAIN_REPO_ROOT`. Read config from `CF_CONFIG_FILE` (= `$MAIN_REPO_ROOT/.coding-friend/config.json`) for `docsDir` (default: `docs`) — do NOT search sub-folders. Use `CF_DOCS_ROOT` as the docs base dir.
- 3. Construct a write spec and delegate to **cf-writer agent** by calling `invoke_subagent` with agent `cf-writer` (use absolute `file_path`):
+ 1. Read `language` (local `.coding-friend/config.json` overrides global, default `en`)
+ 2. `MAIN_REPO_ROOT` from SessionStart bootstrap (`session-init.sh`); else `pwd`. Config from `CF_CONFIG_FILE` (`$MAIN_REPO_ROOT/.coding-friend/config.json`) for `docsDir` (default `docs`) — do not search sub-folders. Docs base: `CF_DOCS_ROOT`.
+ 3. Delegate to **cf-writer** by calling `invoke_subagent` with agent `cf-writer` (absolute `file_path`):
```
WRITE SPEC
----------
task: create
file_path: {CF_DOCS_ROOT}/memory/bugs/YYYY-MM-DD-{name}.md
language: {language from config}
content: |
---
title: "<Short bug title>"
description: "<One-line summary of the bug and fix, under 100 chars>"
tags: [tag1, tag2, tag3]
created: YYYY-MM-DD
updated: YYYY-MM-DD
type: episode
importance: 4
source: conversation
---
# <Bug Title>
## Overview
- <What went wrong — symptom and why it was hard to diagnose>
+ <Symptom and why it was hard to diagnose>
## Investigation
- <What was tried and ruled out — hypotheses that didn't pan out>
+ <Tried and ruled out>
## Root Cause
- <What was actually wrong — the real cause>
+ <Real cause>
## Fix
- <What was changed to fix it>
+ <What changed>
## Prevention
- <How to avoid this bug in the future>
+ <How to avoid it>
## Related Files
- - `path/to/file1`
- - `path/to/file2`
+ - `path/to/file`
readme_update: false
auto_commit: false
existing_file_action: skip
```
- > **Backward compat:** Existing bug memory files without a date prefix are still readable — do not rename them.
-
- **Frontmatter rules:**
+ Existing bug files without a date prefix stay as-is — do not rename.
- - `description`: factual summary for grep recall. Good: `"Circular dependency in plugin loader causing silent init failure"`. Bad: `"Hard bug fixed"`.
- - `tags`: include error type, affected module, root cause category (e.g., `[circular-dependency, plugin-loader, initialization]`)
+ **Frontmatter:** `description` is a factual grep summary (not "Hard bug fixed"). `tags`: error type, module, root-cause category.
### Index in CF Memory (MANDATORY)
- **This step is REQUIRED — do NOT skip it.**
-
- After the cf-writer saves the bug doc, you MUST call the `memory_store` MCP tool to index it in the database. This is a separate action from writing the file — the cf-writer agent does NOT do this.
-
- Call `memory_store` with:
+ **Required — do not skip.** After cf-writer saves the file, call `memory_store` yourself (the writer does not):
- - `title`: from the frontmatter title
- - `description`: from the frontmatter description
+ - `title` / `description` / `tags` from frontmatter
- `type`: `episode`
- - `tags`: from the frontmatter tags
- - `content`: the full markdown content (including frontmatter)
+ - `content`: full markdown including frontmatter
- `importance`: 4
- `source`: "auto-capture"
- `index_only`: true
- If the MCP tool is unavailable, log a warning to the user but do NOT fail silently.
+ If MCP is unavailable, warn the user — do not fail silently.
- Show the user a 2-line summary:
+ 2-line summary:
- **Markdown file:** `{docsDir}/memory/bugs/...md` (created or updated)
- **Memory DB:** indexed ✓ — or: MCP unavailable, file only
## Gotchas
- | What happened | Rule |
- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
- | Patched symptom file instead of origin | Trace the execution path backward before touching any file |
- | MCP not loading, switched tools instead of diagnosing | Check server status, API key, config before switching |
- | Orchestrator said RUNNING but a downstream stage was misconfigured | In multi-stage pipelines, test each stage in isolation |
- | Race condition diagnosed as stale-state bug | For timing-sensitive issues, inspect event timestamps and ordering before state |
- | Reproduced locally but failed in CI | Align the environment first (runtime version, env vars, timezone), then chase the code |
- | Stack trace points deep into a library | Walk back 3 frames into your own code; the bug is almost always there |
- | Adding a `try/catch` to suppress the error | Hides the bug, doesn't fix it — find and fix the root cause |
- | Multiple changes at once | Can't tell which one fixed it — one change at a time, test after each |
+ | What happened | Rule |
+ | ------------------------------------- | ------------------------------------------------------------- |
+ | Patched symptom file, not origin | Trace the path backward first |
+ | Switched tools after MCP/API failure | Diagnose server, key, config first |
+ | Pipeline "RUNNING" but a stage is bad | Test each stage in isolation |
+ | Race diagnosed as stale state | Inspect timestamps and ordering before state |
+ | Local repro, CI fail | Align env (runtime, vars, timezone) before chasing code |
+ | Stack deep in a library | Walk back 3 frames into your code |
+ | `try/catch` to hide the error | Find the root cause |
+ | Multiple changes at once | One change at a time; test after each |
## Debugging Tools
- - `git bisect` — Find the exact commit that introduced a bug
- - `git stash` — Isolate your changes to test clean state
- - Print/log tracing — Add strategic logging at decision points
- - Minimal reproduction — Smallest possible code that triggers the bug
+ `git bisect`, `git stash`, targeted logs, smallest reproduction.
## Outcome
### Success Format
```
Root cause: [what was wrong, file:line]
Fix: [what changed, file:line]
Confirmed: [evidence or test that proves the fix]
Tests: [pass/fail count, regression test location]
Regression guard: [test file:line] or [none, reason]
```
- Status: **resolved**, **resolved with caveats** (state them), or **blocked** (state what is unknown).
+ Status: **resolved**, **resolved with caveats** (state them), or **blocked** (what is unknown).
### Handoff Format (after 3 failed hypotheses)
```
Symptom:
- [Original error description, one sentence]
+ [Original error, one sentence]
Hypotheses Tested:
- 1. [Hypothesis 1] → [Test method] → [Result: ruled out because...]
- 2. [Hypothesis 2] → [Test method] → [Result: ruled out because...]
- 3. [Hypothesis 3] → [Test method] → [Result: ruled out because...]
+ 1. [Hypothesis] → [Test] → [Ruled out because...]
+ 2. ...
+ 3. ...
Evidence Collected:
- - [Log snippets / stack traces / file content]
- - [Reproduction steps]
- - [Environment info: versions, config, runtime]
+ - [Logs / stack / files]
+ - [Repro steps]
+ - [Env: versions, config, runtime]
Ruled Out:
- - [Root causes that have been eliminated]
+ - [Eliminated causes]
Unknowns:
- - [What is still unclear]
- - [What information is missing]
+ - [Still unclear / missing info]
Suggested Next Steps:
- 1. [Next investigation direction]
- 2. [External tools or permissions that may be needed]
- 3. [Additional context the user should provide]
+ 1. [Next direction]
+ 2. [Tools or permissions needed]
+ 3. [Context the user should provide]
```
Status: **blocked**
## Review Reminder
- After the fix is verified, ask the user if they want to run `/cf-review` or `/cf-commit`. Do NOT auto-run — wait for their choice.
+ After the fix is verified, ask whether to run `/cf-review` or `/cf-commit`. Do not auto-run.