runtime-verification · diff
git:20260522.34a8d75 to git:20260909.6778255
23 added, 204 removed. Audit A to A.
---
name: runtime-verification
description: "Verify code works at runtime through build verification (mandatory), LSP diagnostics, ad-hoc verification for projects without frameworks, E2E and smoke tests, and visual verification (screenshot-analyze-verify for UI changes). Skip whitelist strictly enforced (markdown-only, config-only, dependency-bump-only with evidence); all other skips require Proactive-Autonomy escalation. Use after quality checks pass to confirm the code actually runs. This skill MUST be consulted because no test framework is not an excuse to skip; build failure IS a finding and must be fixed."
allowed-tools: Bash, Read, Glob, Grep, LSP, TaskCreate, TaskList, TaskUpdate
agent: Explore
---
# Runtime Verification
- Domain skill for verifying code works at runtime, beyond static analysis and unit tests.
-
- ## Iron Law
-
- **NO COMPLETION UNTIL THE CODE BUILDS, RUNS, AND BEHAVES CORRECTLY. If you cannot verify it yourself, build the infrastructure to verify it.**
-
- A green test suite is necessary but not sufficient. Runtime verification proves code actually works. "No test framework" is a problem to solve, not a reason to skip.
-
- ## Skip Whitelist (Enumerated — No Subjective Exemptions)
-
- Runtime verification is **MANDATORY** for every change. The only permitted skips are the three categories below. Any skip outside this whitelist is forbidden and must be escalated via the Proactive-Autonomy protocol (see below).
-
- | Skip Category | Definition | Required Evidence to Claim |
- |---------------|------------|----------------------------|
- | `markdown-only` | The diff touches **only** `.md`, `.markdown`, `.txt`, or `.rst` files. Zero code, config, or data files. | `git diff --name-only origin/$DEFAULT_BRANCH..HEAD` output showing only doc extensions. |
- | `config-only` | The diff touches **only** configuration files (`.json`, `.yaml`, `.yml`, `.toml`, `.ini`, `.env.example`, dotfiles) with no executable code path changes. Config syntax must still be validated (lint/schema check, dry-run apply). | Full file list plus syntax validation output. |
- | `dependency-bump-only` | The diff touches **only** lockfiles and manifest version strings (e.g., `package.json` version fields, `package-lock.json`, `poetry.lock`, `Gemfile.lock`, `go.sum`, `Cargo.lock`) with no source code, no config semantics, and no new dependencies. Build must still succeed. | Full file list plus successful build output. |
-
- **If the diff mixes any whitelisted category with anything else (a single `.py` or `.ts` file, a new dependency, a config value change that alters behavior), the skip is disallowed. Run full runtime verification.**
-
- ### If In Doubt, Run It
-
- **If you are uncertain whether the change qualifies for a whitelist skip — run runtime verification.** Uncertainty is never a reason to skip. The cost of an extra verification run is small; the cost of shipping unverified code is large.
-
- Forbidden reasoning patterns include "small change", "CI-only edit", "tests already cover this", "I read the diff and it looks safe", "just a refactor", and similar. None of those are whitelist categories. If your reasoning does not map cleanly to `markdown-only`, `config-only`, or `dependency-bump-only` with the explicit evidence shown in the table above, you MUST run verification.
-
- ### Escalation Protocol for Out-of-Whitelist Skips
-
- If you genuinely believe a skip outside the whitelist is warranted (e.g., infrastructure-only change, generated-code-only change, or something the whitelist does not yet cover), you MUST NOT proceed silently. Raise a Proactive-Autonomy escalation per [`references/escalation-format.md`](../../references/escalation-format.md) using `AskUserQuestion` and receive an explicit approval before proceeding. The Situation field MUST cite the specific change and why the whitelist does not cover it; the Tried field MUST list the standard paths attempted (fast-path verify script, build, smoke tests). Blanket "always skip for this repo" authorization is never valid — each out-of-whitelist skip requires its own escalation.
-
- ## Fast-Path Verification
-
- Check for a project-level verify script first:
-
- ```bash
- [ -x "verify.sh" ] && echo "FAST_PATH: verify.sh found"
- [ -x "scripts/verify.sh" ] && echo "FAST_PATH: scripts/verify.sh found"
- ```
-
- If found, run it and return results. Skip remaining steps.
-
- ## Build Verification
-
- **Mandatory build step for all project types.** Build failure IS the finding — do NOT skip to runtime checks.
-
- ```bash
- # Node.js / TypeScript
- [ -f "package.json" ] && npm run build 2>&1
-
- # Python
- [ -f "setup.py" ] || [ -f "pyproject.toml" ] && pip install -e . 2>&1
-
- # Go
- [ -f "go.mod" ] && go build ./... 2>&1
-
- # Rust
- [ -f "Cargo.toml" ] && cargo build 2>&1
-
- # Ruby
- [ -f "Gemfile" ] && bundle install 2>&1
- ```
-
- If build fails → iterate: read errors, fix, rebuild (up to `closedLoop.maxBuildIterations`, default 5). Do NOT proceed until the build passes.
-
- ## LSP Diagnostics Verification
-
- **Pre-check**: Read `lsp.enabled` from settings (default `true`). If `false`, skip this section entirely.
-
- When LSP is available and `lsp.diagnosticsAsQuality` is enabled in settings, collect language server diagnostics as an additional quality signal. This complements — never replaces — CLI-based quality commands.
-
- ### Process
-
- 1. Identify all files changed on the branch:
- ```bash
- git diff --name-only origin/$DEFAULT_BRANCH..HEAD
- ```
-
- 2. For each changed source file, use `LSP(documentSymbol)` to confirm the language server recognizes the file, then collect any diagnostics reported.
-
- 3. Map diagnostics to findings:
-
- | LSP Severity | Finding Priority | Action |
- |-------------|-----------------|--------|
- | Error | P1 | Must fix before proceeding |
- | Warning | P2 | Should fix |
- | Information/Hint | P3 | Consider |
-
- 4. Deduplicate against CLI tool output — if the same issue is reported by both LSP and a CLI tool (e.g., `tsc` and TypeScript LSP), keep only one entry.
-
- ### Timeout Handling
-
- Each LSP operation must complete within `lsp.timeout` (default 5000ms from settings). If an operation times out:
- - Mark that file's LSP check as "Timeout — skipped"
- - Continue with remaining files
- - Note timeout in output table
- - Never block the workflow on a slow LSP server
-
- ### Graceful Fallback
-
- If no LSP server is available, skip this section entirely with note: "LSP diagnostics: N/A — no language server configured." This is not an error and does not affect the verification outcome.
-
- ## Ad-Hoc Verification
-
- For projects without formal test frameworks, verify by running the code:
-
- | Project Type | Verification Approach |
- |-------------|----------------------|
- | Backend/API | Start server, curl endpoints, verify responses, check logs |
- | CLI tools | Build, run with --help, run with sample input, check exit codes |
- | Libraries | Write temporary test script, exercise public API, verify outputs, delete script |
- | Static sites | Build, serve locally, verify pages load |
- | Config-only | Validate config syntax, apply dry-run if supported |
-
- "No test framework" is a problem to solve, not a reason to skip verification.
-
- ## Iterative Debug Loop
-
- When any verification fails:
- 1. Read the FULL error message and stack trace — don't skim
- 2. Identify root cause (not just the symptom)
- 3. Fix the root cause
- 4. Re-verify
-
- If the same error persists after a fix attempt: re-read code paths, try a different approach. Max `closedLoop.maxDebugIterations` (default 5) iterations, then escalate to user.
-
- **The user should NEVER have to provide logs or tell you what went wrong.** You have access to the same errors — read them yourself.
-
- ## Discovery (dev server, port, E2E framework)
-
- The `capability-discovery` skill (invoked at the start of every verify-relevant command) already detects tech stack, dev server scripts, and E2E frameworks. Re-running discovery here is redundant; consume the existing output.
-
- When operating standalone (no prior `capability-discovery` invocation), use these probes:
-
- ```bash
- # Dev server: CLAUDE.md hints, package.json scripts, framework config files
- [ -f ".claude/CLAUDE.md" ] && grep -iE "(dev|server|start|serve):" .claude/CLAUDE.md
- [ -f "package.json" ] && python3 -c "import json; d=json.load(open('package.json')); [print(f'{k}: {v}') for k,v in d.get('scripts',{}).items() if k in ('dev','start','serve')]"
-
- # Port: running listeners on common ports
- lsof -i -P -n 2>/dev/null | grep LISTEN | grep -E ':(3000|4000|5000|8000|8080)' | head -5
-
- # E2E framework: config files
- [ -f "playwright.config.ts" ] || [ -f "playwright.config.js" ] && echo "Playwright"
- [ -f "cypress.config.ts" ] || [ -f "cypress.config.js" ] && echo "Cypress"
- ```
-
- ## Smoke Tests
-
- If a dev server is running, perform basic health checks:
-
- ```bash
- curl -s -o /dev/null -w "%{http_code}" http://localhost:$PORT/health
- curl -s -o /dev/null -w "%{http_code}" http://localhost:$PORT/
- ```
-
- A non-200 response is a P1 finding emitted via the canonical schema in [`references/finding-schema.md`](../../references/finding-schema.md) with `category=runtime`.
-
- ## Visual Verification (extracted to its own skill)
+ ## Contract
- Visual verification has been extracted to [`skills/visual-verification/SKILL.md`](../visual-verification/SKILL.md). That skill owns the screenshot-analyze-verify loop, the browser-tool priority cascade (Playwright MCP → Chrome DevTools MCP → CLI → external skill fallback), the responsive viewport checks, and the result vocabulary (PASS / FAIL / SKIP / SKIP_WARN / SKIP_USER_APPROVED / MANUAL / BLOCKED).
+ Iron law: **no completion until the code builds, runs, and behaves correctly; if you cannot verify it, build the infrastructure to verify it.** Invoked after quality checks pass by `/flow:start` Phase 4 step 2, `/flow:pr` Phase 4 via `Agent(integration-verifier)`, and the `address-pr`, `debug`, and `start-issue` workflows. Returns the Runtime Verification Results and Acceptance Criteria Verification tables, with failures as P1 findings (`category=runtime`). Permitted skips: only `markdown-only`, `config-only`, or `dependency-bump-only`, each with its listed evidence; any other skip needs an approved six-field escalation via `AskUserQuestion`.
- When a diff is UI-relevant (UI file extensions OR acceptance criteria with UI keywords — see the visual-verification skill for the detection rules), the consumer (`commands/start.md` Phase 4, `commands/pr.md` Phase 4) invokes BOTH skills in parallel:
+ ## Skip Whitelist
- ```
- Skill(runtime-verification) # build + server + smoke + E2E + LSP diagnostics
- Skill(visual-verification) # browser-rendered UI loop
- ```
+ | Category | Definition | Required evidence |
+ |----------|------------|-------------------|
+ | `markdown-only` | Only `.md`, `.markdown`, `.txt`, `.rst` files | `git diff --name-only origin/$DEFAULT_BRANCH..HEAD` showing only doc extensions |
+ | `config-only` | Only config files (`.json`, `.yaml`, `.yml`, `.toml`, `.ini`, `.env.example`, dotfiles) with no code-path change; syntax still validated (lint/schema, dry-run) | Full file list plus validation output |
+ | `dependency-bump-only` | Only lockfiles and manifest version strings (`package.json` version, `package-lock.json`, `poetry.lock`, `Gemfile.lock`, `go.sum`, `Cargo.lock`); no source, config semantics, or new dependencies; build still succeeds | Full file list plus build output |
- The two skills coordinate via the dev server URL: if `runtime-verification` cannot start the dev server, `visual-verification` returns SKIP with reason "dev server unavailable" and the completion gate treats the dev-server failure as the primary finding rather than emitting a SKIP_WARN that obscures the root cause.
+ Mixed diffs (a whitelisted category plus one `.py` file, a new dependency, or a behavior-changing config value) get full verification. "Small change", "CI-only", "tests cover this", "just a refactor" are not categories.
- ## Acceptance Criteria Verification
+ **Out-of-whitelist skips** (anything uncovered): raise a Proactive-Autonomy escalation per [`references/escalation-format.md`](../../references/escalation-format.md) via `AskUserQuestion` and wait for approval; Situation cites why the whitelist misses the change, Tried lists the fast-path, build, and smoke attempts. Blanket repo-wide authorization is never valid.
- Map each acceptance criterion to a verification method:
+ ## Sequence
- | Criterion Type | Verification |
- |---------------|-------------|
- | API behavior | curl/fetch endpoint, check response |
- | UI rendering | Screenshot-analyze-verify loop (see Visual Verification) |
- | UI responsive | Multi-viewport screenshot verification |
- | Data processing | Run with test data, check output |
- | Configuration | Verify config loads without error |
+ 1. **Fast path**: an executable `verify.sh` or `scripts/verify.sh` is run and its results returned.
+ 2. **Build** (mandatory, every project type; per-stack commands and tables in [`runtime-verification-probes.md`](../../references/runtime-verification-probes.md)). Build failure IS the finding: read errors, fix, rebuild up to `closedLoop.maxBuildIterations` (default 5).
+ 3. **LSP diagnostics** when `lsp.enabled` and `lsp.diagnosticsAsQuality` (both default `true`): Error=P1, Warning=P2, Info/Hint=P3, deduplicated against CLI output, bounded by `lsp.timeout` (default 5000 ms); timeout or no server is N/A, never a block.
+ 4. **Dev server**: use `capability-discovery` output. Won't start: read the error, fix, retry up to `closedLoop.maxServerRetries` (default 3); port busy: another port; startup wait `timeouts.devServerStartup` seconds (default 30).
+ 5. **Smoke**: `curl` `/health` and `/`; non-200 is a P1 finding (`references/finding-schema.md`).
+ 6. **E2E** when a framework exists, within `timeouts.e2eTest` (default 120); otherwise ad-hoc verification by project type (reference); "no test framework" is a problem to solve, not a skip.
+ 7. **Acceptance criteria**: map each to a method (reference table) and record Pass/Fail/N/A.
- ## Completion Gate
+ On any failure: read the full error, root-cause it (`debugging-patterns`), fix, re-verify, up to `closedLoop.maxDebugIterations` (default 5), then escalate; the user never supplies logs.
- Runtime verification is complete only when:
+ ## Completion
- - Every testable acceptance criterion has a verification result (Pass/Fail/N/A with reason)
- - "N/A" is justified (e.g., no dev server for a CLI tool), never used as a shortcut
- - Failed checks are reported as P1 findings, not silently noted
+ UI-relevant diffs (rules in `skills/visual-verification/SKILL.md`) also run `Skill(visual-verification)` in parallel; if the dev server cannot start, that skill returns `SKIP` ("dev server unavailable") and the server failure is the primary finding; tables render side by side. `visualVerification.*` settings belong there.
- If the dev server won't start, that IS the finding. Report it.
+ Complete only when every testable criterion has Pass/Fail/N/A with a reason and every failure, including a dev server that won't start, is a P1 finding.
## Output Format
```markdown
### Runtime Verification Results
| Check | Status | Details |
|-------|--------|---------|
| Dev server | {Running/Not found} | Port {N} |
| Health check | {Pass/Fail/N/A} | HTTP {status} |
| E2E tests | {Pass/Fail/N/A} | {framework} |
| Smoke tests | {Pass/Fail/N/A} | {details} |
| LSP diagnostics | {Pass/Fail/Skip/N/A} | {errors: N, warnings: N, files checked: N} |
### Acceptance Criteria Verification
| # | Criterion | Verified | Method |
|---|-----------|----------|--------|
+ | {n} | {ui criterion} | Pass/Fail | visual-verification `Observed:` blocks → bundle `### Visual analysis` |
```
-
- When the consumer also invokes `Skill(visual-verification)`, the visual rows (Visual check, Responsive, Console errors, Visual Evidence table) are produced by that skill — see its Output Format section. The consumer renders the runtime-verification table and the visual-verification table side by side rather than merging them, so each skill's output remains attributable.
-
- ## Timeout Configuration
-
- From `settings.json`:
- - `timeouts.devServerStartup`: Max seconds to wait for dev server (default: 30)
- - `timeouts.e2eTest`: Max seconds for E2E suite (default: 120)
-
- `visualVerification.*` settings (including `maxIterations`) are owned by `Skill(visual-verification)`; see that skill's documentation when both run together.
-
- ## Active Problem Solving
-
- Do NOT silently skip verification. Actively solve problems:
-
- | Problem | Action |
- |---|---|
- | No dev server | Attempt to start one. Report P1 if no start command exists and no alternative verification is possible. |
- | No E2E framework | Run ad-hoc smoke tests (curl endpoints, run CLI, exercise API) |
- | Server won't start | Read the error, fix the code, retry (up to `closedLoop.maxServerRetries`, default 3) |
- | Port already in use | Try alternative ports |
- | LSP server unavailable / times out | Skip LSP diagnostics; do NOT block on them. Note "LSP diagnostics: N/A — {reason}" in the output table. |
-
- Browser-tool problems (no Playwright, no Chrome DevTools, no npx) are handled by the `visual-verification` skill — see its "Active Problem Solving" section.