running-tests · diff

git:20260901.d696ee9 to git:20260902.128748b

4 added, 0 removed. Audit A to A.

---

name: running-tests

audience: swarm-plugin

description: >

Safe test execution patterns for opencode-swarm. Covers when to use the test_runner

tool vs shell bun commands, scope safety rules, per-file isolation loops (bash and

PowerShell), pre-existing failure verification, CI log reading, and failure

classification. Load this skill when you need to run tests — not when you need to

write them (see writing-tests for authoring guidance).

---

# Running Tests for opencode-swarm

This skill is about **executing** tests safely. For **writing** tests, see `writing-tests`.

+ ## Graph-first evidence contract

+ Use `repo_map` `test_pack` only to discover focused candidate tests; Bun/shell output remains execution authority. Graph evidence is advisory only. If freshness is stale or inconclusive, confidence is low, source is missing, the language is unsupported/dynamic, the graph is absent, or the action fails, select tests from direct source, imports, and repository conventions.

---

## ⛔ The One Rule That Prevents Session Kills

**Never use `test_runner` with more than one source file for any discovery scope.**

`graph` and `impact` each fan out per file through the import tree; `convention` maps

each source file to a test file by name convention. The union quickly exceeds

`MAX_SAFE_TEST_FILES = 50`, triggering `scope_exceeded`, which causes LLMs to

cascade to `scope: 'all'` and kill the session. All three scopes now reject with

`scope_exceeded` before fan-out when `sourceFiles.length > MAX_SAFE_SOURCE_FILES = 1`.

---

## Three-Layer Defense Against Session Blocking

test_runner has three pre-resolution guards that prevent unbounded fan-out from blocking the session:

### Layer 1 — Source-file count guard (synchronous, fires before any I/O)

`sourceFiles.length > MAX_SAFE_SOURCE_FILES (1)` → returns `scope_exceeded` immediately. Catches the common case of multi-file calls before any filesystem access.

### Layer 2 — Pre-resolution fan-out estimate (fast, ~100ms)

`estimateFanOut(sourceFiles, workingDir)` reads the cached impact map and counts unique test files without spawning subprocesses. If the estimate exceeds `MAX_SAFE_TEST_FILES = 50`, the call returns `scope_exceeded` immediately — before any graph traversal begins. Only fires when `sourceFiles.length === 1` (Layer 1 has already passed).

### Layer 3 — Budget-limited traversal + post-resolution length check

`analyzeImpact` accepts a `budget` parameter (`MAX_SAFE_TEST_FILES = 50`). The traversal stops as soon as it has visited 50 test files and sets `budgetExceeded: true`. The call site checks this flag and returns `scope_exceeded` before processing results.

After graph resolution, the final `testFiles.length` is additionally compared to `MAX_SAFE_TEST_FILES`. If exceeded, `scope_exceeded` is returned.

**Result:** When fan-out exceeds the safe threshold, the session gets `outcome: 'scope_exceeded'` instead of hanging.

---

## Decision Tree: test_runner tool vs bun shell command

```

Do you need to run tests?

│

├─ Single test file, targeted validation

│ └─ Either works. Prefer shell: bun --smol test <file> --timeout 30000

│

├─ Multiple files in the same directory (e.g. all agents tests)

│ └─ Shell only — per-file loop. Never test_runner with multiple files.

│

├─ Find tests related to ONE changed source file

│ └─ test_runner is fine: { scope: 'graph', files: ['src/agents/coder.ts'] }

│ (single file → bounded fan-out)

│

├─ Find tests related to MULTIPLE changed source files

│ └─ Shell only — per-file loop over the changed files, or run the whole directory.

│ test_runner with any discovery scope + multiple source files = scope_exceeded

│ (guard fires before fan-out for convention, graph, and impact scopes).

│

└─ Validate the entire repo (pre-push)

└─ Shell only — 5-tier suite from commit-pr skill. Never test_runner scope:'all'.

```

---

## Scope Safety Reference

|-------|--------------------|--------------------|-------|

**Rule of thumb:** Pass exactly one source file to `test_runner`. For multiple files, use a shell loop.

For one named Go or CTest test, bypass file discovery with an exact native selector:

- Go: `{ scope: "target", native_target: { framework: "go-test", name: "TestName[/Subtest]", path: "relative/package" } }`

- CTest: `{ scope: "target", native_target: { framework: "ctest", name: "ExactTestName", path: "relative/build-dir" } }`

The target name is treated literally, the directory must stay within the project root, and the runner never falls back to a broader package or build-tree sweep.

---

## Per-File Isolation Loops

CI runs agents/tools/services in per-file isolation (one `bun --smol` process per file).

Reproduce this locally with the following loops.

### bash (Linux / macOS)

```bash

# Single directory — per-file isolation

for f in tests/unit/agents/*.test.ts; do

bun --smol test "$f" --timeout 30000

done

# Multiple directories

for dir in tests/unit/tools tests/unit/services tests/unit/agents; do

for f in "$dir"/*.test.ts; do

bun --smol test "$f" --timeout 30000

done

# Stop on first failure (useful for debugging)

for f in tests/unit/agents/*.test.ts; do

bun --smol test "$f" --timeout 30000 || { echo "FAILED: $f"; break; }

done

```

### PowerShell (Windows)

```powershell

# Single directory — per-file isolation

Get-ChildItem tests/unit/agents/*.test.ts | ForEach-Object {

bun --smol test $_.FullName --timeout 30000

}

# Multiple directories

@('tests/unit/tools', 'tests/unit/services', 'tests/unit/agents') | ForEach-Object {

Get-ChildItem "$_/*.test.ts" | ForEach-Object {

bun --smol test $_.FullName --timeout 30000

}

# Capture output (avoids truncation on large output)

Get-ChildItem tests/unit/agents/*.test.ts | ForEach-Object {

bun --smol test $_.FullName --timeout 30000

} | Out-File "$env:TEMP\test_out.txt"

Get-Content "$env:TEMP\test_out.txt" | Select-Object -Last 50

```

**Common PowerShell pitfalls:**

- `for f in ...; do` — invalid, use `Get-ChildItem | ForEach-Object`

- `Select-String -Last N` — invalid parameter, use `Select-Object -Last N`

- `2>&1 2>&1` — duplicate redirection, causes parse error; use `2>&1` once

- `&&` — not supported in PowerShell 5.1; use `; if ($?) { cmd2 }` instead

- `bun test --exec bash` — fails on Windows hosts with ENOENT (bash is not available in standard PowerShell). Use `bun test` directly or a PowerShell-based loop instead.

- After `bun install --frozen-lockfile --force`, non-elevated Windows shells can hit `EPERM` while reading refreshed `node_modules` entries. Treat that as a host permission/access issue: rerun the same focused Bun command with approved/elevated access before diagnosing it as a code or test failure.

---

## Batch vs Per-File: Which Directories Need Isolation?

| Directory | Mode | Reason |

|-----------|------|--------|

| `tests/unit/tools/` | Per-file loop | Heavy `mock.module` usage; cache poisoning risk |

| `tests/unit/services/` | Per-file loop | Same |

| `tests/unit/agents/` | Per-file loop | Same |

| `tests/unit/hooks/` | Per-file loop | Same |

| `tests/unit/cli/` | Batch OK | Fewer mock conflicts |

| `tests/unit/commands/` | Batch OK | Fewer mock conflicts |

| `tests/unit/config/` | Batch OK | Fewer mock conflicts |

| `tests/integration/` | Batch OK | Integration fixtures, not mock-heavy |

| `tests/security/` | Batch OK | Adversarial inputs, no module mocks |

| `tests/smoke/` | Batch OK | Built-package tests |

---

## Truncated Output Recovery

When `bun test` output exceeds the bash tool's buffer, it is saved to a file with an ID

like `tool_dff778...`. This ID format is **not** accepted by `retrieve_summary` (which only

reads `S1`, `S2` etc. format IDs). The output is effectively lost.

**Prevention — pipe to a file explicitly:**

```powershell

# PowerShell

bun --smol test tests/unit/agents --timeout 60000 |

Out-File "$env:TEMP\test_out.txt"

Get-Content "$env:TEMP\test_out.txt" | Select-Object -Last 50

```

```bash

# bash

bun --smol test tests/unit/agents --timeout 60000 2>&1 | tee /tmp/test_out.txt

tail -50 /tmp/test_out.txt

```

**To get a clean pass/fail summary only**, filter immediately:

```powershell

# PowerShell — show only summary lines

bun --smol test tests/unit/agents --timeout 60000 |

Select-String "pass|fail|error" |

Select-Object -Last 10

```

```bash

# bash

bun --smol test tests/unit/agents --timeout 60000 2>&1 | grep -E "pass|fail|error" | tail -10

```

---

## Verifying Pre-Existing Failures

Before documenting a failure as "pre-existing," prove it exists on `main` without affecting

your working tree. Use a Git worktree — safer than `git stash` (stash can drop untracked

files, fail on locked files on Windows, and leave you in an inconsistent state).

```bash

# bash — create a throwaway checkout of main

git worktree add /tmp/repro-check origin/main

bun --smol test /tmp/repro-check/tests/unit/agents/architect-workflow-security.test.ts --timeout 30000

git worktree remove /tmp/repro-check

```

```powershell

# PowerShell — same pattern (use Join-Path for robust separator handling)

git worktree add "$env:TEMP\repro-check" origin/main

$testPath = Join-Path "$env:TEMP\repro-check" "tests\unit\agents\architect-workflow-security.test.ts"

bun --smol test $testPath --timeout 30000

git worktree remove "$env:TEMP\repro-check"

```

**Decision after checking:**

- Fails on `main` too → pre-existing. Document under `## Pre-existing failures` in PR body. Continue.

- Fails only on your branch → you introduced it. Fix before pushing.

**⚠️ Check your own session history first.** Before documenting anything as pre-existing, confirm you did not fix or update this test earlier in the current session. A test you fixed 20 messages ago is not pre-existing — listing it as such in the table or PR body is incorrect and will be caught in review.

---

## Placeholder Scans Without Diff Line Numbers

`placeholder_scan` is diff-aware: when you can supply `added_lines` (a map of workspace-relative file path → added line numbers from the task/PR diff), its verdict covers only the added lines, so a pre-existing TODO/FIXME on an unchanged line inside a changed file no longer fails the gate. When you CANNOT map a file's added lines (new/untracked file, no diff access) — or the computed added-line set is EMPTY — omit that file from `added_lines` entirely: the tool scans it unfiltered, fail-closed, and you manually cross-check that file's findings against the changed lines before treating a finding as introduced by the change. NEVER pass an empty line array for a file (an empty array suppresses every finding in it) and NEVER hand-enumerate guessed line numbers: a wrong `added_lines` map silently suppresses findings.

---

## Failure Classification

Not all failures are equal. Before deciding what to do, classify the failure:

|-------|-----------|---------|------------|

| **Stale assertion** | Test checks for text/value that was deliberately removed | `expect(prompt).toContain('CONSTRAINT: [what NOT to do]')` — template removed in refactor | Update the assertion to match current state |

| **Soft regression indicator** | Test checks a threshold the codebase has since exceeded | `expect(tokenCount).toBeLessThan(35000)` — prompt grew past limit | Fix the threshold or reduce the prompt; do not just document and ignore |

| **Genuine pre-existing** | Failure exists on `main` unrelated to any recent change | See the quarantine ledgers (`scripts/ci/quarantined-tests*.txt`) | Document in PR body; do not fix unless scoped |

**Stale assertions and soft regression indicators are actionable** — they signal drift between

tests and code. Genuine pre-existing failures are not your responsibility to fix in this PR,

but they must be documented.

---

## Reading CI Failure Logs

When a CI job fails, the GitHub Actions log shows the exact `file:line` of the failure.

Do not guess — read the log.

```bash

# Get the failing job URL from the PR

gh pr view <number> --json statusCheckRollup --jq '.statusCheckRollup[] | select(.conclusion=="FAILURE") | .detailsUrl'

# Fetch and search the log (if gh CLI available)

gh run view --log <run-id> | grep -E "FAIL|error" | head -20

```

Or open the `detailsUrl` directly in a browser / via WebFetch and search for:

- `(fail)` — Bun test failure marker

- `error:` — parse or runtime error

- `at <anonymous>` — stack frame pointing to the test file and line

Once you have `tests/unit/agents/some-file.test.ts:354`, reproduce locally:

```bash

bun --smol test tests/unit/agents/some-file.test.ts --timeout 30000

```

---

## Quick Reference: Common Failures and Causes

| Symptom | Likely cause | Fix |

|---------|-------------|-----|

| `scope_exceeded` returned from test_runner | Fan-out exceeded 50 test files during graph/impact resolution | Switch to per-file shell loop; reduce changed-files scope |

| Session killed during test_runner | Pre-fix: unbounded fan-out on multiple files | Now returns `scope_exceeded` instead — no more session kills |

| `mock.module` breaks unrelated tests | Missing spread of real module exports | Add `...realModule` spread |

| Windows tests fail with EBUSY | `mock.restore()` called while child process holds lock | Add `test.skipIf(process.platform === 'win32')` |

| Test output truncated, ID unreadable | Bash tool buffer exceeded | Pipe to `Out-File`/`tee` explicitly |

| `Select-String -Last N` error | Invalid PowerShell parameter | Use `Select-Object -Last N` |

| Token budget test failure | Prompt grew past hardcoded threshold | Treat as soft regression; update threshold |

| CONSTRAINT assertion fails after refactor | Test checks for removed format template | Update assertion to match current prompt |

| `package-check` CI failure | `package-check` validates the npm tarball (`npm pack` + tarball contents) — a source/build/package-manifest problem, not generated-file drift | `dist/` is generated and NOT committed — do not stage it; run `bun run build` locally only when you need the bundle. There is no longer a committed-dist drift check. |

## Tree-sitter / WASM test timeouts

Tests that exercise tree-sitter (any test calling `extractFileSymbols` or loading a `web-tree-sitter` grammar) may take several seconds on **first WASM module load**. Depending on the code path, tree-sitter is reached via the dynamic symbol-graph import or the externalized runtime import; either way, the first `Parser.init` / grammar load in a process is slow.

- Use `--timeout 60000` (not 30000) for test files that load tree-sitter grammars.

- If the `test_engineer` agent gets stuck (no output for extended time), run the test file directly via bash with a longer timeout (`--timeout 120000`) to determine whether it's a WASM first-load delay or a genuine code failure.

- **Classify the timeout** before returning the test_engineer to the coder — a WASM-load timeout is infrastructure, not a code bug.

- Each test process loads WASM independently (no cross-process cache), so every file's first grammar load is slow.