running-tests git:20260711.5972d3eA

Immutable. This exact content is served forever at /api/v1/blob/c029f766b9b0e128.

---
name: running-tests
audience: swarm-plugin
description: >
  Safe test execution patterns for opencode-swarm. Covers when to use the test_runner
  tool vs shell bun commands, scope safety rules, per-file isolation loops (bash and
  PowerShell), pre-existing failure verification, CI log reading, and failure
  classification. Load this skill when you need to run tests — not when you need to
  write them (see writing-tests for authoring guidance).
---

# Running Tests for opencode-swarm

This skill is about **executing** tests safely. For **writing** tests, see `writing-tests`.

---

## ⛔ The One Rule That Prevents Session Kills

**Never use `test_runner` with more than one source file for any discovery scope.**

`graph` and `impact` each fan out per file through the import tree; `convention` maps
each source file to a test file by name convention. The union quickly exceeds
`MAX_SAFE_TEST_FILES = 50`, triggering `scope_exceeded`, which causes LLMs to
cascade to `scope: 'all'` and kill the session. All three scopes now reject with
`scope_exceeded` before fan-out when `sourceFiles.length > MAX_SAFE_SOURCE_FILES = 1`.

---

## Three-Layer Defense Against Session Blocking

test_runner has three pre-resolution guards that prevent unbounded fan-out from blocking the session:

### Layer 1 — Source-file count guard (synchronous, fires before any I/O)
`sourceFiles.length > MAX_SAFE_SOURCE_FILES (1)` → returns `scope_exceeded` immediately. Catches the common case of multi-file calls before any filesystem access.

### Layer 2 — Pre-resolution fan-out estimate (fast, ~100ms)
`estimateFanOut(sourceFiles, workingDir)` reads the cached impact map and counts unique test files without spawning subprocesses. If the estimate exceeds `MAX_SAFE_TEST_FILES = 50`, the call returns `scope_exceeded` immediately — before any graph traversal begins. Only fires when `sourceFiles.length === 1` (Layer 1 has already passed).

### Layer 3 — Budget-limited traversal + post-resolution length check
`analyzeImpact` accepts a `budget` parameter (`MAX_SAFE_TEST_FILES = 50`). The traversal stops as soon as it has visited 50 test files and sets `budgetExceeded: true`. The call site checks this flag and returns `scope_exceeded` before processing results.
After graph resolution, the final `testFiles.length` is additionally compared to `MAX_SAFE_TEST_FILES`. If exceeded, `scope_exceeded` is returned.

**Result:** When fan-out exceeds the safe threshold, the session gets `outcome: 'scope_exceeded'` instead of hanging.

---

## Decision Tree: test_runner tool vs bun shell command

```
Do you need to run tests?
│
├─ Single test file, targeted validation
│   └─ Either works. Prefer shell: bun --smol test <file> --timeout 30000
│
├─ Multiple files in the same directory (e.g. all agents tests)
│   └─ Shell only — per-file loop. Never test_runner with multiple files.
│
├─ Find tests related to ONE changed source file
│   └─ test_runner is fine: { scope: 'graph', files: ['src/agents/coder.ts'] }
│      (single file → bounded fan-out)
│
├─ Find tests related to MULTIPLE changed source files
│   └─ Shell only — per-file loop over the changed files, or run the whole directory.
│      test_runner with any discovery scope + multiple source files = scope_exceeded
│      (guard fires before fan-out for convention, graph, and impact scopes).
│
└─ Validate the entire repo (pre-push)
    └─ Shell only — 5-tier suite from commit-pr skill. Never test_runner scope:'all'.
```

---

## Scope Safety Reference

| Scope | With `files: [one]` | With `files: [many]` | Notes |
|-------|--------------------|--------------------|-------|
| `'convention'` | ✅ Safe | ❌ Rejected (`scope_exceeded`) | Guard fires before fan-out; direct test file paths exempt |
| `'graph'` | ✅ Safe (capped at 50 via budget) | ❌ Rejected (`scope_exceeded`) | Two-layer guard: source-file count + fan-out estimate |
| `'impact'` | ✅ Safe (capped at 50 via budget) | ❌ Rejected (`scope_exceeded`) | Two-layer guard: source-file count + fan-out estimate |
| `'all'` | ❌ Never | ❌ Never | Requires `allow_full_suite: true`; CI mirror only |
| `'all'` | ❌ Never | ❌ Never | Requires `allow_full_suite: true`; CI mirror only |

**Rule of thumb:** Pass exactly one source file to `test_runner`. For multiple files, use a shell loop.

---

## Per-File Isolation Loops

CI runs agents/tools/services in per-file isolation (one `bun --smol` process per file).
Reproduce this locally with the following loops.

### bash (Linux / macOS)

```bash
# Single directory — per-file isolation
for f in tests/unit/agents/*.test.ts; do
  bun --smol test "$f" --timeout 30000
done

# Multiple directories
for dir in tests/unit/tools tests/unit/services tests/unit/agents; do
  for f in "$dir"/*.test.ts; do
    bun --smol test "$f" --timeout 30000
  done
done

# Stop on first failure (useful for debugging)
for f in tests/unit/agents/*.test.ts; do
  bun --smol test "$f" --timeout 30000 || { echo "FAILED: $f"; break; }
done
```

### PowerShell (Windows)

```powershell
# Single directory — per-file isolation
Get-ChildItem tests/unit/agents/*.test.ts | ForEach-Object {
  bun --smol test $_.FullName --timeout 30000
}

# Multiple directories
@('tests/unit/tools', 'tests/unit/services', 'tests/unit/agents') | ForEach-Object {
  Get-ChildItem "$_/*.test.ts" | ForEach-Object {
    bun --smol test $_.FullName --timeout 30000
  }
}

# Capture output (avoids truncation on large output)
Get-ChildItem tests/unit/agents/*.test.ts | ForEach-Object {
  bun --smol test $_.FullName --timeout 30000
} | Out-File "$env:TEMP\test_out.txt"
Get-Content "$env:TEMP\test_out.txt" | Select-Object -Last 50
```

**Common PowerShell pitfalls:**
- `for f in ...; do` — invalid, use `Get-ChildItem | ForEach-Object`
- `Select-String -Last N` — invalid parameter, use `Select-Object -Last N`
- `2>&1 2>&1` — duplicate redirection, causes parse error; use `2>&1` once
- `&&` — not supported in PowerShell 5.1; use `; if ($?) { cmd2 }` instead
- `bun test --exec bash` — fails on Windows hosts with ENOENT (bash is not available in standard PowerShell). Use `bun test` directly or a PowerShell-based loop instead.
- After `bun install --frozen-lockfile --force`, non-elevated Windows shells can hit `EPERM` while reading refreshed `node_modules` entries. Treat that as a host permission/access issue: rerun the same focused Bun command with approved/elevated access before diagnosing it as a code or test failure.

---

## Batch vs Per-File: Which Directories Need Isolation?

| Directory | Mode | Reason |
|-----------|------|--------|
| `tests/unit/tools/` | Per-file loop | Heavy `mock.module` usage; cache poisoning risk |
| `tests/unit/services/` | Per-file loop | Same |
| `tests/unit/agents/` | Per-file loop | Same |
| `tests/unit/hooks/` | Per-file loop | Same |
| `tests/unit/cli/` | Batch OK | Fewer mock conflicts |
| `tests/unit/commands/` | Batch OK | Fewer mock conflicts |
| `tests/unit/config/` | Batch OK | Fewer mock conflicts |
| `tests/integration/` | Batch OK | Integration fixtures, not mock-heavy |
| `tests/security/` | Batch OK | Adversarial inputs, no module mocks |
| `tests/smoke/` | Batch OK | Built-package tests |

---

## Truncated Output Recovery

When `bun test` output exceeds the bash tool's buffer, it is saved to a file with an ID
like `tool_dff778...`. This ID format is **not** accepted by `retrieve_summary` (which only
reads `S1`, `S2` etc. format IDs). The output is effectively lost.

**Prevention — pipe to a file explicitly:**

```powershell
# PowerShell
bun --smol test tests/unit/agents --timeout 60000 |
  Out-File "$env:TEMP\test_out.txt"
Get-Content "$env:TEMP\test_out.txt" | Select-Object -Last 50
```

```bash
# bash
bun --smol test tests/unit/agents --timeout 60000 2>&1 | tee /tmp/test_out.txt
tail -50 /tmp/test_out.txt
```

**To get a clean pass/fail summary only**, filter immediately:

```powershell
# PowerShell — show only summary lines
bun --smol test tests/unit/agents --timeout 60000 |
  Select-String "pass|fail|error" |
  Select-Object -Last 10
```

```bash
# bash
bun --smol test tests/unit/agents --timeout 60000 2>&1 | grep -E "pass|fail|error" | tail -10
```

---

## Verifying Pre-Existing Failures

Before documenting a failure as "pre-existing," prove it exists on `main` without affecting
your working tree. Use a Git worktree — safer than `git stash` (stash can drop untracked
files, fail on locked files on Windows, and leave you in an inconsistent state).

```bash
# bash — create a throwaway checkout of main
git worktree add /tmp/repro-check origin/main
bun --smol test /tmp/repro-check/tests/unit/agents/architect-workflow-security.test.ts --timeout 30000
git worktree remove /tmp/repro-check
```

```powershell
# PowerShell — same pattern (use Join-Path for robust separator handling)
git worktree add "$env:TEMP\repro-check" origin/main
$testPath = Join-Path "$env:TEMP\repro-check" "tests\unit\agents\architect-workflow-security.test.ts"
bun --smol test $testPath --timeout 30000
git worktree remove "$env:TEMP\repro-check"
```

**Decision after checking:**
- Fails on `main` too → pre-existing. Document under `## Pre-existing failures` in PR body. Continue.
- Fails only on your branch → you introduced it. Fix before pushing.

**⚠️ Check your own session history first.** Before documenting anything as pre-existing, confirm you did not fix or update this test earlier in the current session. A test you fixed 20 messages ago is not pre-existing — listing it as such in the table or PR body is incorrect and will be caught in review.

---

## Failure Classification

Not all failures are equal. Before deciding what to do, classify the failure:

| Class | Definition | Example | What to do |
|-------|-----------|---------|------------|
| **Stale assertion** | Test checks for text/value that was deliberately removed | `expect(prompt).toContain('CONSTRAINT: [what NOT to do]')` — template removed in refactor | Update the assertion to match current state |
| **Soft regression indicator** | Test checks a threshold the codebase has since exceeded | `expect(tokenCount).toBeLessThan(35000)` — prompt grew past limit | Fix the threshold or reduce the prompt; do not just document and ignore |
| **Genuine pre-existing** | Failure exists on `main` unrelated to any recent change | `full-auto-intercept.test.ts` logger gating issue | Document in PR body; do not fix unless scoped |
| **New regression** | Failure introduced by your changes | Tests for prompt text you removed without updating tests | Fix before pushing |

**Stale assertions and soft regression indicators are actionable** — they signal drift between
tests and code. Genuine pre-existing failures are not your responsibility to fix in this PR,
but they must be documented.

---

## Reading CI Failure Logs

When a CI job fails, the GitHub Actions log shows the exact `file:line` of the failure.
Do not guess — read the log.

```bash
# Get the failing job URL from the PR
gh pr view <number> --json statusCheckRollup --jq '.statusCheckRollup[] | select(.conclusion=="FAILURE") | .detailsUrl'

# Fetch and search the log (if gh CLI available)
gh run view --log <run-id> | grep -E "FAIL|error" | head -20
```

Or open the `detailsUrl` directly in a browser / via WebFetch and search for:
- `(fail)` — Bun test failure marker
- `error:` — parse or runtime error
- `at <anonymous>` — stack frame pointing to the test file and line

Once you have `tests/unit/agents/some-file.test.ts:354`, reproduce locally:
```bash
bun --smol test tests/unit/agents/some-file.test.ts --timeout 30000
```

---

## Quick Reference: Common Failures and Causes

| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| `scope_exceeded` returned from test_runner | Fan-out exceeded 50 test files during graph/impact resolution | Switch to per-file shell loop; reduce changed-files scope |
| Session killed during test_runner | Pre-fix: unbounded fan-out on multiple files | Now returns `scope_exceeded` instead — no more session kills |
| `mock.module` breaks unrelated tests | Missing spread of real module exports | Add `...realModule` spread |
| Windows tests fail with EBUSY | `mock.restore()` called while child process holds lock | Add `test.skipIf(process.platform === 'win32')` |
| Test output truncated, ID unreadable | Bash tool buffer exceeded | Pipe to `Out-File`/`tee` explicitly |
| `for f in ...; do` parse error | Bash syntax in PowerShell | Use `Get-ChildItem | ForEach-Object` |
| `Select-String -Last N` error | Invalid PowerShell parameter | Use `Select-Object -Last N` |
| Token budget test failure | Prompt grew past hardcoded threshold | Treat as soft regression; update threshold |
| CONSTRAINT assertion fails after refactor | Test checks for removed format template | Update assertion to match current prompt |
| `package-check` CI failure | `package-check` validates the npm tarball (`npm pack` + tarball contents) — a source/build/package-manifest problem, not generated-file drift. `dist/` is generated and NOT committed — do not stage it; run `bun run build` locally only when you need the bundle. There is no longer a committed-dist drift check. |

## Tree-sitter / WASM test timeouts

Tests that exercise tree-sitter (any test calling `extractFileSymbols` or loading a `web-tree-sitter` grammar) may take several seconds on **first WASM module load**. Depending on the code path, tree-sitter is reached via the dynamic symbol-graph import or the externalized runtime import; either way, the first `Parser.init` / grammar load in a process is slow.

- Use `--timeout 60000` (not 30000) for test files that load tree-sitter grammars.
- If the `test_engineer` agent gets stuck (no output for extended time), run the test file directly via bash with a longer timeout (`--timeout 120000`) to determine whether it's a WASM first-load delay or a genuine code failure.
- **Classify the timeout** before returning the test_engineer to the coder — a WASM-load timeout is infrastructure, not a code bug.
- Each test process loads WASM independently (no cross-process cache), so every file's first grammar load is slow.