--- name: test-and-verify-walnut description: "4-agent pipeline: design + implement code tests + Playwright UI tests + quality gate. Reads walnut-console-ops first." --- # /test-and-verify-walnut Every test must answer: **"If I reverted my code changes, would this test fail?"** NO → delete it. **FIRST**: Read `.claude/skills/walnut-console-ops/SKILL.md` for UI layout and interaction patterns. ``` Main Agent (context) → Agent 1 (design) → Agent 2 (code) ∥ Agent 3 (Playwright) → Agent 4 (quality gate) ``` --- ## Phase 0: Main Agent — Context 1. `git diff --stat HEAD~1` + `git log --oneline -5` 2. Read plan files: `.plan`, `~/.claude/plans/`, `.tasks/*/TASK.md` 3. Classify change type: | Change Type | Primary Tests | |---|---| | Frontend-only | Playwright UI | | Backend-only | API + unit + server logs | | Full-stack | Console E2E + Playwright + unit | | Bug fix | Reproduce → fix → re-verify | 4. Bundle context → pass to all agents. --- ## Phase 1: Agent 1 — Test Designer (read-only) Design tests in 2 categories. Do NOT implement. **Category A — UI (Playwright)**: 2-4 scenarios. For each: pre-conditions, steps (real clicks, NO `page.goto()` SPA nav), assertions (must include downstream verification — see below), screenshot points. **Category B — Code (vitest)**: 2-4 tests, as E2E as possible. For each: name, tier, setup, exercise, assert (HTTP response + WS event + persisted data). **Self-check each test**: "Would this pass with code reverted?" YES = delete. **E2E means full consequence**: Every test must verify the **downstream effect** of the action, not just the immediate UI change. If the action triggers a backend round-trip, the test must wait for and verify the backend result. Anti-patterns (test is SHALLOW or WORTHLESS if it only checks these): - "page loads", "component renders", "no console errors" - "API returns 200" without checking response body - "clicked button and UI closed" without verifying the backend outcome - "status badge changed" without verifying the underlying operation succeeded - Testing mock config instead of production code --- ## Phase 2: Agent 2 — Code Test Writer (∥ Agent 3) Implement Category B. Rules: - Real server: `startServer({ port: 0, dev: true })`. Only mock Claude CLI (`tests/providers/mock-claude.mjs`). - Assert on HTTP responses, WS events, file contents — not internal state. - Place: `tests/e2e/` (pipeline), `tests/web/routes/` (API), `tests/core/` (logic), `tests/e2e/browser/` (Playwright code). - Run and iterate until green. --- ## Phase 3: Agent 3 — Playwright Executor (∥ Agent 2) **First**: Read `.claude/skills/walnut-console-ops/SKILL.md`. Execute Category A: - `mkdir -p /tmp/test-and-verify/`, build SPA: `cd web && npx vite build` - Real UI clicks only. Screenshot every step to `/tmp/test-and-verify/-step.png` - DOM snapshot before each action. Verify screenshots with `Read` tool. - Act → Wait (5-20s) → Screenshot → Snapshot → verify. **CRITICAL — Verify full consequence, not just UI reaction**: Every UI action that triggers backend work must be verified **through to the downstream effect**. The test is NOT done when the UI element changes — it's done when the **outcome is confirmed**. The question: **"Did the thing I triggered actually work?"** — not **"Did the UI react to my click?"** **BAD (shallow)** — stops at UI reaction: ``` # Testing model switch: Click "Opus 1M" → picker closes → screenshot → ✅ DONE # WRONG: only proved the picker UI works. Did the session actually restart # with the new model? Did the next message succeed? No idea. # Testing task creation: Click "Add" → task appears in list → screenshot → ✅ DONE # WRONG: only proved the React component rendered. Did the task persist # to disk? Can you reload and still see it? No idea. # Testing session start: Click "Start session" → status shows "Running" → screenshot → ✅ DONE # WRONG: only proved the status badge updated. Did the session actually # produce output? Did the CLI process spawn? No idea. ``` **GOOD (full consequence)** — verifies the downstream effect: ``` # Testing model switch: Click "Opus 1M" → picker closes → send a message in session input → wait for response → verify response text appears (no "API Error", no "invalid model") → screenshot response → ✅ DONE # Testing task creation: Click "Add" → task appears in list → click the task → verify detail panel shows correct title/category/project → ✅ DONE # Testing session start: Click "Start session" → status shows "Running" → wait for first assistant message to appear in chat → verify message content → screenshot → ✅ DONE ``` The pattern: **Action → UI reacts → wait for backend round-trip → verify outcome → screenshot proof**. If you stop at "UI reacts", the test is SHALLOW. --- ## Phase 4: Agent 4 — Quality Verifier (after 2+3) For each test, ask TWO questions: 1. "Would this pass if feature was reverted?" YES = **WORTHLESS**, delete it. 2. "Does this verify the **downstream effect**, or just the immediate UI change?" UI-only = **SHALLOW**, must extend. - **GENUINE**: Triggers action AND verifies the full backend/system consequence - **SHALLOW**: Verifies UI reacted (button clicked, picker closed, badge changed) but NOT that the triggered operation succeeded. Must be extended to verify downstream. - **WORTHLESS**: Would pass without code changes A Playwright test that clicks a button and screenshots is **SHALLOW** unless it also waits for and verifies what that button *caused* (response arrived, status updated, data persisted, no errors). Overall: PASS / NEEDS WORK / FAIL --- ## Phase 5: Report ```markdown ## Test & Verify Report: ### Build — Server: PASS/FAIL, Frontend: PASS/FAIL ### Code Tests — tests, genuine ### UI Tests — scenarios, genuine ### Overall — PASS / NEEDS WORK / FAIL ``` ## Phase 6: Learning Loop New patterns → append to Learned Patterns below. Console-related → also update `walnut-console-ops/SKILL.md`. --- ## Test Reference ```bash npm test # Unit + Integration (~5s) npm run test:e2e # E2E real server (~15s) npm run test:all # Everything npx vitest run # Single test cd web && npx vite build # Build SPA before Playwright npx playwright test # Browser tests ``` **Pyramid**: Live → Playwright → **E2E** (most important) → Integration → Unit **Mock only**: Claude CLI. Everything else real. **Production safety**: `OPEN_WALNUT_HOME=/tmp/walnut-test-*/`. Never `~/.open-walnut/`. Port 3456 = production, never touch. Live checks with a real CLI run on `npm run dev:ephemeral`. **Open source**: No internal tool names, employer references, or personal usernames. ### E2E Template ```typescript vi.mock('../../src/constants.js', () => createMockConstants('walnut-e2e-FEATURE')); import { startServer, stopServer } from '../../src/web/server.js'; let server, port; beforeAll(async () => { server = await startServer({ port: 0, dev: true }); port = (server.address() as any).port; }); afterAll(() => stopServer()); it('REST → Core → Bus → WS', async () => { const ws = await connectWs(); const event = waitForWsEvent(ws, 'task:created'); const res = await fetch(`http://localhost:${port}/api/tasks`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ title: 'test' }), }); expect(res.status).toBe(201); expect((await event).data.task.title).toBe('test'); ws.close(); }); ``` ### Session E2E: Mock CLI wrapper ```typescript const MOCK_CLI = path.resolve(import.meta.dirname, '../providers/mock-claude.mjs'); sessionRunner.setCliCommand(MOCK_WRAPPER); // before startServer ``` ### Architecture (for test design) - **Event Bus**: backbone — test WS delivery for entity CRUD - **Phases**: TODO → IN_PROGRESS → AGENT_COMPLETE → AWAIT_HUMAN_ACTION → COMPLETE - **Sessions**: 2-slot (plan + exec), PID monitoring, FIFO stall - **Concurrency**: 2-layer locks. Test concurrent writes. - **WS**: `{ type: 'event', name, data }` / RPC: `{ type: 'req', id, method, payload }` ### Gotchas - WS events async — use `waitForWsEvent()` with timeout - Build SPA before Playwright or you test stale HTML - Click sidebar links, don't `page.goto('/route')` in SPA - Concurrent tests need separate WALNUT_HOME dirs --- ## Learned Patterns - **Real-session browser specs are now possible**: `tests/e2e/browser/test-server.ts` wires a MockDaemon + mock CLI (`sessionRunner.setCliCommand` + `setTestDaemonUrl`) into the Playwright server, so specs can create LIVE sessions via the `session:start` WS RPC and exercise the real event pipeline (no `__capturedWs` injection needed). Disable with `PW_NO_MOCK_DAEMON=1` when bisecting whether a failure comes from this wiring. See `tests/e2e/browser/single-timeline-real-pipeline.spec.ts` for the pattern (start session → poll `/api/sessions/task/:id` for the `mock-session-*` id → open `/sessions?id=`). - **`chunk-delay:` mock-CLI prefix**: spaces out `content_block_delta` emissions in `stream-partial-thinking-then-text` so browser tests can observe partial text mid-turn (the default burst is synchronous and races any DOM poll). Composable: `chunk-delay:250 stream-partial-thinking-then-text`. - **MockDaemon JSONL is NOT where history REST reads**: the MockDaemon captures CLI output in its own tmpDir; `GET /:id/history` falls back to `SESSION_STREAMS_DIR/.jsonl`. Vitest tests that need real-parser history must copy `daemon.streamFilePath(sid)` → `SESSION_STREAMS_DIR` first (see `publishStreamsFile` in `tests/web/stream/single-timeline-real-pipeline.test.ts`). Symptom otherwise: history 200 with 0 messages forever. - **Full-suite failure triage**: `npx playwright test` failures may come from PARALLEL agents' uncommitted UI changes, not yours. Bisect by re-running the failing spec against the HEAD version of the shared harness file you touched (`git checkout ` → run → restore) before assuming a regression.