git:20260806.7ecf589 to git:20260922.c6db1f2

- description: Triage bugs reported in chat/issues, search for duplicates, file or update GitHub issues with full context, and push fix PRs.

- trigger: User reports a bug, or asks to triage/file an issue for a reported problem.

- ---

-

- # Bug Triage Skill

-

- Triage bugs into well-structured GitHub issues on the correct upstream repo.

-

- ## 1. Pre-flight

-

- - **Pull latest code:** `git pull origin main`. Stale code = bad triage.

- - **Target repo:** Always file on the **current repo** (`Untrivial-ai/agent-orchestrator`), not forks.

- - **Record source:** chat URL, reporter name, attachments.

-

- ## 2. Gather Context

-

- ### 2a. Extract the report

-

- | Source | How to gather |

- |--------|---------------|

- | **Discord/Slack thread** | Read full thread. Extract: reporter name, original description (the thread starter, not whoever tagged you), screenshots, follow-ups |

- | **GitHub issue** | `gh issue view <number> --repo <repo> --json body,comments` |

- | **Live observation** | Pull live state via observability tools |

-

- ### 2b. Minimum viable report gate

-

- Before tracing code, verify the report has enough substance:

-

- **Required (ALL):** what happened, where (page/command/feature), when (after upgrade? first time?)

-

- **Required (2 of 4):** OS/shell/runtime, AO version (`ao --version`), reproducibility (consistent vs intermittent), reproduction steps

-

- If insufficient, ask:

- > "I'd like to triage this but need more info: (1) **What happened?** (error/behavior), (2) **Where?** (page/command), (3) **When did it start?**, (4) **How to reproduce?**"

-

- ### 2c. Local diagnostics (if bug is on same machine)

-

- Gather everything yourself before asking the reporter:

-

- ```bash

- # Environment

- ao --version && node --version && echo $SHELL && uname -a

- cat agent-orchestrator.yaml

- cat ~/.agent-orchestrator/running.json

-

- # Process health

- pm2 status

- tmux list-sessions

- lsof -i :3000

-

- # AO event log — structured timeline

- ao events list --limit 50 # recent events

- ao events list --session ao-5 --limit 100 # filter by session

- ao events list --log-level error --since 1h # errors only

- ao events search "spawn failed" # full-text search

- ao events stats # counts by kind/source

-

- # Session state files

- cat ~/.agent-orchestrator/projects/*/sessions/*.json | python3 -m json.tool

- ```

-

- Event kinds: `session.spawned`, `session.spawn_failed`, `session.killed`, `lifecycle.transition`, `ci.failing`, `review.pending`, `runtime.probe_failed`, `agent.process_probe_failed`, `reaction.escalated`, `lifecycle.poll_failed`. Sources: `lifecycle`, `session-manager`, `api`, `runtime`, `agent`, `reaction`.

-

- **Try the reproduction steps.** Running the actual command is worth 100 lines of code tracing.

-

- ## 3. Investigate

-

- ### 3a. Trace the code path

-

- **Always trace the actual code** — don't surface-level diagnose. [#1129](https://github.com/Untrivial-ai/agent-orchestrator/issues/1129) looked like a simple `ao stop` issue but was actually a session lineage/cascade problem.

-

- ```bash

- git fetch origin main && git log --oneline origin/main -5 # current HEAD

- # Record the commit hash you're analyzing against

- ```

-

- **Git archaeology** — find which commits introduced/removed specific code:

- ```bash

- git log --oneline -S 'exact-string' -- <file>

- git show <sha> -- <file> | grep -B 5 -A 10 'pattern'

- ```

- Example: [#1391](https://github.com/Untrivial-ai/agent-orchestrator/issues/1391) traced a mobile layout break to a `display: flex` → `display: grid` change.

-

- **Research upstream dependencies** (xterm, node-pty, React, etc.) — check installed vs latest version, search their GitHub issues, check changelogs. Root cause is often upstream.

-

- ### 3b. Cross-platform check

-

- AO runs on **Windows, macOS, Linux** as first-class targets. If env info indicates Windows (or is unknown), check for these patterns:

-

- - **Path separators** — hardcoded `/` or `\` breaks cross-platform

- - **Shell syntax** — PowerShell lacks `&&`, `$VAR`, `$(cat ...)`, `/dev/null`, here-docs

- - **`process.platform === "win32"` inline** — must use `isWindows()` from `@aoagents/ao-core`

- - **`process.kill(-pid)`** — POSIX-only; use `killProcessTree()`

- - **Named pipes vs Unix sockets** — Windows uses `\\.\pipe\ao-pty-<id>`

- - **`localhost`** — Windows resolves to `::1` first, causing ~21s stalls on IPv4-only servers

- - **NTFS case-insensitivity** — use `pathsEqual()`, not `===`

- - **ConPTY orphans** — can trigger WER dialogs if pty-host not shut down cooperatively

- - **`.cmd` shim resolution** — needs `shell: true` for `PATHEXT` lookup

-

- Key files: `packages/core/src/platform.ts`, `docs/CROSS_PLATFORM.md`, `packages/plugins/runtime-process/`, `packages/cli/src/lib/path-equality.ts`

-

- ### 3c. Stop-and-ask triggers

-

- Stop and ask for more info if:

- - **3 failed hypotheses** — traced 3 code paths, none explain it

- - **Root cause is upstream** — file with upstream reference, don't guess a local fix

- - **UI-only bug** and you can't screenshot — ask reporter to describe

- - **Can't reproduce** — ask for different config/sequence

-

- ## 4. Search for Duplicates

-

- Search with multiple strategies, always using `--state all` (closed bugs regress):

-

- ```bash

- gh issue list --repo <repo> --state all --search "<symptom>"

- gh issue list --repo <repo> --state all --search "<component-name>"

- gh issue list --repo <repo> --state all --search "<error-message>"

- gh pr list --repo <repo> --state all --search "<keywords>"

- ```

-

- ### Duplicate found → comment on existing issue

-

- ```bash

- gh issue comment <number> --repo <repo> --body "$(cat <<'EOF'

- ## New Report

- **Reported by:** @<reporter> in [chat](<url>)

- **Date:** <YYYY-MM-DD> | **Checkout:** `<commit-hash>`

- <context, differences from original, screenshots>

- EOF

- )"

- ```

-

- ### No duplicate → file new issue (next section)

-

- ## 5. File New Issue

-

- ### 5a. Pre-submission checklist

-

- - [ ] Reporter attribution correct (original reporter, not who tagged you)

- - [ ] Commit hash recorded

- - [ ] AO version recorded

- - [ ] Root cause confidence scored (see 5c)

- - [ ] Related issues cross-linked

- - [ ] Reproduction steps are concrete

- - [ ] Screenshots uploaded with real URLs (see 5b)

-

- ### 5b. Upload screenshots

-

- **⛔ NEVER use placeholder URLs.** Upload BEFORE creating the issue. ([#1151](https://github.com/Untrivial-ai/agent-orchestrator/issues/1151) RCA on this pattern.)

-

- ```bash

- SLUG="descriptive-slug"

- # Create asset branch

- gh api -X POST repos/<repo>/git/refs \

- -f ref="refs/heads/issue-assets-${SLUG}" \

- -f sha=$(git rev-parse origin/main)

-

- # Upload (portable base64)

- IMG_B64=$(base64 < /path/to/screenshot.png | tr -d '\n')

- gh api -X PUT "repos/<repo>/contents/.issue-assets/${SLUG}/name.png" \

- -f message="chore: upload screenshot" \

- -f content="$IMG_B64" \

- -f branch="issue-assets-${SLUG}"

- # Use: 

- ```

-

- ### 5c. Create the issue

-

- ```bash

- gh issue create --repo <repo> --title "<title>" --body "$(cat <<'EOF'

- ## Bug

- <summary>

-

- **Source:** <url> | **Reported by:** @<reporter> | **Analyzed against:** `<hash>`

- **Confidence:** High/Medium/Low

-

- ## Reproduction

- 1. <step>

-

- ## Root Cause

- <file paths, line numbers, explanation>

-

- ## Fix

- <suggested approach>

-

- ## Impact

- - <effects>

- EOF

- )"

- ```

-

- ### 5d. Label and prioritize

-

- ```bash

- gh issue edit <number> --repo <repo> --add-label "bug"

- ```

-

- | Priority label | Criteria |

- |----------------|----------|

- | `priority: critical` | Data loss, security, system down |

- | `priority: high` | Core feature broken, no workaround |

- | `priority: medium` | Feature degraded, workaround exists |

- | `priority: low` | Cosmetic, edge case |

-

- **Confidence scoring** (include in issue body):

-

- | Level | Meaning | Extra labels |

- |-------|---------|-------------|

- | **High** | Traced exact code path, specific lines, mechanism explained | `bug` only |

- | **Medium** | Strong hypothesis but unconfirmed | `bug`, `to-explore` |

- | **Low** | Can't trace, multiple conflicting theories | `bug`, `to-reproduce` |

-

- Example: [PR #1608](https://github.com/Untrivial-ai/agent-orchestrator/pull/1608) was diagnosed High as xterm v6 issue — real cause was a `=` prefix on tmux `set-option`. Should have been Medium.

-

- **All available labels:** `priority: critical/high/medium/low`, `bug`, `enhancement`, `good-first-issue`, `to-reproduce`, `to-explore`. No others (no `p0`, `p1`, etc.).

-

- ### 5e. Cross-link related issues

-

- Search by subsystem and add a `## Related` section to the issue body:

- ```

- ## Related

- - [#1020](url) — stale session blocking ao start (same subsystem)

- - [#1035](url) — same race condition

- ```

-

- ### 5f. Push a fix PR (always attempt)

-

- - **Trivial fix:** Push immediately.

- - **Complex fix:** Note in issue, suggest spawning an agent.

- - **Unclear fix:** Don't push a guess. Document and flag.

-

- ```bash

- OLD_STRING='<old>' NEW_STRING='<new>' \

- python3 skills/bug-triage/scripts/push_fix_to_github.py \

- <owner/repo> fix/slug path/to/file.tsx \

- "fix(scope): commit msg" "fix(scope): PR title" \

- "Fixes #<n>

-

- ## Summary

- <what changed>

-

- ## Test

- <how to verify>"

- ```

-

- The script reads from GitHub API, applies one replacement, pushes, opens PR — no local checkout needed. **Verify `OLD_STRING` matches GitHub first:** `gh api repos/<repo>/contents/<path>?ref=main -q '.content' | python3 -c "import base64,sys; sys.stdout.buffer.write(base64.b64decode(sys.stdin.read()))"`

-

- **Multiple edits to same file:** The push script does one replacement per run. For multiple changes, use a Python script to read from the branch (`gh api` + `base64.b64decode`), apply all replacements, then push via `gh api -X PUT` with the updated SHA.

-

- ### 5g. Report back

-

- Issue URL, PR URL (if created), labels, root cause summary, whether fix agent was suggested.

-

- ## Appendix

-

- ### A. Subsystem Quick Reference

-

- | Subsystem | Collect | Key files |

- |-----------|---------|-----------|

- | **CLI** (`ao start/stop/spawn`) | Config YAML, install method, version, OS | `packages/cli/src/commands/` |

- | **Web UI** | Screenshot, browser, viewport | `packages/web/src/components/`, `globals.css` |

- | **Terminal** | Runtime type, tmux version, shell | `DirectTerminal.tsx`, `useXtermTerminal.ts` |

- | **Lifecycle** | State transitions, session IDs | `core/src/lifecycle-manager.ts`, `core/src/lifecycle-state.ts` |

- | **Sessions** | Session ID, spawn config, runtime | `core/src/session-manager.ts` |

- | **Plugins** | Plugin name, agent version | `packages/plugins/<agent>/` |

- | **Config** | YAML contents, project path | `packages/core/src/config.ts` |

-

- **Misrouting patterns:**

- - Terminal bugs → tmux (runtime-tmux) vs xterm (web) vs PTY (runtime-process/Windows). Trace where bytes flow.

- - "Session stuck" → lifecycle state machine vs agent process vs runtime connection.

- - "Config not saving" → config loading (c12) vs project registration (running-state.ts) vs YAML write (permissions).

-

- ### B. Remote Code Inspection (no local clone)

-

- ```bash

- gh api repos/{owner}/{repo}/git/trees/main?recursive=1 --jq '.tree[].path' # list files

- gh api repos/{owner}/{repo}/contents/{path} --jq '.content' | python3 -c "import base64,sys; sys.stdout.buffer.write(base64.b64decode(sys.stdin.read()))" # read file

- gh search code "term" --repo {owner}/{repo} --json path --jq '.[].path' # search code

- gh api "repos/{owner}/{repo}/commits?path={path}&per_page=10" --jq '.[] | "\(.sha[0:8]) \(.commit.message | split("\n")[0])"' # file history

- ```

-

- ### C. NPM Package Regression Diffing

-

- Diff **published** packages (not local builds) when regression follows an upgrade:

-

- ```bash

- mkdir -p /tmp/ao-diff/{v1,v2}

- curl -sL https://registry.npmjs.org/@scope/pkg/-/pkg-OLD.tgz | tar xz -C /tmp/ao-diff/v1

- curl -sL https://registry.npmjs.org/@scope/pkg/-/pkg-NEW.tgz | tar xz -C /tmp/ao-diff/v2

- diff -rq /tmp/ao-diff/v1/package/ /tmp/ao-diff/v2/package/

- ```

-

- Example: [PR #1608](https://github.com/Untrivial-ai/agent-orchestrator/pull/1608) — source analysis led to wrong theories, npm diff showed the only change was a `=` prefix on tmux `set-option`.

-

- ## Formatting Rules

-

- - **Linkify all issue/PR refs:** `[#123](https://github.com/Untrivial-ai/agent-orchestrator/issues/123)`, `[PR #456](url)`. Never bare `#123`.

-

- ## Pitfalls

- - **Reporter ≠ person who tagged you.** Always attribute to the original reporter.

- - **Record the commit hash** you analyzed — code changes fast.

- - **GitHub issue is mandatory** — every triaged bug gets one, even if fix is trivial.

- - **`gh api --jq .content` truncates large files** (>~100KB). Use local git instead.

- - **Push script arg limits** — long commit messages hit `OSError: Argument list too long`. Use a Python script with JSON payloads instead.

- - **`OLD_STRING` must match GitHub byte-for-byte** — local code may differ from `origin/main`.

- - **New fields on shared TS interfaces MUST be optional** (`field?: Type`). Downstream `Partial<X>` spreads break on required fields. Example: [PR #1523](https://github.com/Untrivial-ai/agent-orchestrator/pull/1523).