verify ยท diff
git:20261012.71f9400 to git:20261012.abeec18
55 added, 54 removed. Audit B to A.
---
name: verify
- description: Project recipe for /verify in code-mode. Drive the plugin in real headless Claude Code against the bench's fake MCP server, then read the transcript and the server's call log. Use when you verify a change to hooks/, runtime/ or hints/.
+ description: Project recipe for /verify in code-mode. Drive the plugin in real Claude Code against the bench's fake MCP server with drive.mjs, headless or in tmux, then judge by the transcript and the server's call log. Use when you verify a change to hooks/, runtime/ or hints/.
---
# Verify code-mode
- The surface is the agent: real `claude -p` with this plugin loaded. The model calls `run_code` or `search_tools`, the
- sandbox makes nested calls, and the fake server logs each call that reaches it. A docs-only or tests-only diff has no
- surface: report SKIP.
+ The surface is the agent: real Claude Code with this plugin loaded. The model calls `run_code` or `search_tools`,
+ the sandbox makes nested calls, and the fake server logs each call that reaches it. A docs-only or tests-only diff
+ has no surface: report SKIP.
- `tests/smoke.e2e.mjs` is a fixed regression test, not a verification. Use its `argv` as the reference for flags, but
- write a prompt that makes the changed code run.
+ `drive.mjs` (next to this file) does the setup, the evidence and the clean-up. The usage is at its top. Each run gets
+ its own folder, server name and session id. In a prompt or a tool name, `{srv}` is the server name.
- ## Drive
+ ## Pick the drive
- Run from the repo root. Each run gets its own server name, so it sees no shapes or argument names of earlier runs.
+ - `node tests/smoke.e2e.mjs` covers the fixed flows in one run: allow rule, no rule, deny rule, sandbox globals,
+ `saveFile`/`readFile`, `search_tools` after a run, a result that Claude Code saved to a file, and the SessionStart
+ text. Run it first. Do not drive these flows again one by one.
+ - Then drive the changed code path with `drive.mjs run` and a prompt that makes it run. Put several probes in one
+ prompt when they need the same flags.
+ ## Headless
+
```bash
- ROOT=$PWD; OUT=$(mktemp -d); mkdir $OUT/cwd
- SRV=verify_$(uuidgen | cut -c1-8 | tr A-Z a-z); SID=$(uuidgen | tr A-Z a-z)
- printf '{"mcpServers":{"%s":{"command":"node","args":["%s/bench/fake-server.mjs"],"env":{"BENCH_LOG":"%s/calls.jsonl"}}}}' $SRV $ROOT $OUT > $OUT/mcp.json
- PROMPT="Use mcp__code-mode__run_code to count the issues of project OPS with mcp__${SRV}__list_issues ... Then reply DONE."
- cd $OUT/cwd && claude -p "$PROMPT" --restricted --strict-mcp-config --mcp-config $OUT/mcp.json --plugin-dir $ROOT \
- --allowedTools "mcp__code-mode__run_code,mcp__code-mode__search_tools,mcp__${SRV}__list_issues" \
- --output-format json --session-id $SID --max-budget-usd 0.5 --model haiku > $OUT/out.json 2> $OUT/stderr.txt
+ D=.claude/skills/verify/drive.mjs
+ node $D run 'Call mcp__code-mode__run_code with exactly this code: const r = await call("mcp__{srv}__list_issues", { project: "OPS" }); return r.total' --allow 'mcp__code-mode__run_code,mcp__{srv}__list_issues'
+ node $D clean <dir> # after you read the evidence
```
- - Plugin options and permission rules go in `--settings`, for example
- `'{"permissions":{"deny":["mcp__<srv>__get_deploys"]},"pluginConfigs":{"code-mode":{"options":{"blockDirectMcp":true,"metrics":true}}}}'`.
- - Fake server tools: `search_messages` (pages, results over 20,000 characters), `list_issues`, `query_logs`,
- `create_ticket`, `get_deploys`. Data is in `bench/data.mjs`.
- - One run with Haiku costs less than $0.01.
+ `run` prints JSON: `reply`, `calls` (what reached the server), `toolUses`, `toolResults` (cut to 800 characters),
+ `hookContext` (the code-mode lines only), `savedResults`, `costUsd`. Judge by `calls` and `toolResults`, never by
+ `reply` alone. One run with Haiku costs less than $0.01.
- ## Evidence
+ Options: `--settings <json>` for plugin options and permission rules, `--max-output-tokens 5000` to make Claude Code
+ save a large result to a file, `--base <dir>` for the run folder.
- ```bash
- jq -r '.result, .is_error, .total_cost_usd' $OUT/out.json # reply of the model
- cat $OUT/calls.jsonl # nested calls that reached the server
- T=~/.claude/projects/$(cd $OUT/cwd && pwd -P | sed 's/[^a-zA-Z0-9]/-/g')/$SID.jsonl
- jq -c 'select(.message.content|type=="array")|.message.content[]|select(.type=="tool_use")|{name,input}' $T
- jq -c 'select(.message.content|type=="array")|.message.content[]|select(.type=="tool_result")' $T
- jq -r 'select(.attachment.type=="hook_additional_context")|.attachment.content[]' $T # SessionStart text
- ```
+ Fake server tools: `search_messages` (pages), `list_issues`, `query_logs`, `create_ticket`, `get_deploys`. The data
+ is in `bench/data.mjs`.
- Judge by the transcript and `calls.jsonl`, never by the words of the model.
+ ## Auto mode
- ## Flows worth driving
+ Run `--auto` two times:
- - Permission path: a nested call with an allow rule reaches the server. A call with no rule is refused, because
- `claude -p` cannot ask (`refused: Claude requested permissions ...`). A call with a deny rule is stopped before
- the run.
- - Sandbox: `typeof process` and `typeof require` are `"undefined"`. `saveFile` then `readFile` round-trips.
- - `search_tools` after a run shows `names from earlier calls` and `returns (earlier call)`.
- - Large results: `search_messages` with `limit: 100` on `#release`.
- - SessionStart text and `blockDirectMcp`: set the option in `--settings` and read the hook context.
+ 1. No allow rule: the nested call is refused with `auto mode classifier gave no verdict`, and `calls` stays empty.
+ This is the documented limit in `docs/permissions.mdx`. If the call reaches the server, Claude Code fixed
+ anthropics/claude-code#100575: tell the person, because the docs must change.
+ 2. `--settings '{"permissions":{"allow":["mcp__{srv}__list_issues"]}}'`: the call reaches the server.
- ## Clean up
+ ## The band (TUI)
- The run writes outside `$OUT`. Remove it after each run:
+ The band, the proposal cards and the `add_hint` row draw only in an interactive session. `band start` turns on
+ `projectHints`, so a proposal goes to the run's own project and not to `~/.claude`.
```bash
- SRV=$SRV node -e '
- const fs=require("fs"),p=require("path"),H=p.join(require("os").homedir(),".claude","code-mode");
- for (const f of ["shapes.json","args.json"].map(f=>p.join(H,f)).filter(fs.existsSync)) {
- const o=JSON.parse(fs.readFileSync(f,"utf8"));
- fs.writeFileSync(f, JSON.stringify(Object.fromEntries(Object.entries(o).filter(([t])=>!t.startsWith(`mcp__${process.env.SRV}__`))),null,2)+"\n") }'
- rm -rf ~/.claude/code-mode/files/$SID ~/.claude/code-mode/metrics/$SID.jsonl "$(dirname $T)"
+ node $D band start # prints dir and server; accepts the trust dialog of its own folder
+ node $D band send <dir> 'Load the tool mcp__code-mode__add_hint, then call it with server "{srv}", text "...", why "...", scope "project".'
+ node $D band wait 'Review'
+ node $D band capture # evidence: the row, the band, any "hook skipped: threw" line
+ node $D band click '[ Review ]' # then '[ Approve ]' or '[ Discard ]'
+ node $D band files <dir> # the hint files of the run's project
+ node $D band stop <dir> # stops tmux and cleans up
```
+ - After an edit to `hooks/`, the session reloads the plugin (`code-mode: reloaded`). To draw old rows again, use
+ `band restart <dir>`.
+ - `band start` also writes `.claude-plugin/types/`, which the type check needs.
+
+ ## Probes
+
+ - A long path: `--base` with a folder deeper than 200 characters. This found a `$.store` key over its limit.
+ - A tool name with a typo, a server that is not connected, a program that throws, a call that never answers.
+
## Gotchas
- - User-level plugins and hooks still load with `--restricted`. The SessionStart context can hold text from other
- plugins. Look only for the code-mode line.
- - Without `blockDirectMcp`, the model can call `ToolSearch` to load the server's tools directly. That is not a direct
- MCP call. Check `tool_use` names for `mcp__<srv>__*`.
- - macOS has no `timeout`. Use `--max-budget-usd` to cap the run.
- - The plugin denies Bash commands that name its hint folders or the function folder in `~/.claude`. To check a
- proposal from `add_hint`, use the Read tool. Do not drive `add_hint` against the real `~/.claude` without a reason:
- it writes a real proposal.
+ - User-level plugins and hooks still load with `--restricted`. Their text is in the transcript, but `hookContext`
+ keeps only the code-mode lines.
+ - Without `blockDirectMcp`, the model can call `ToolSearch` to load the fake tools. That is not a direct MCP call:
+ check `toolUses` for `mcp__{srv}__*`.
+ - The plugin denies Bash commands that name its hint folders or its function folder. Use `band files` or the Read
+ tool. Never drive `add_hint` with scope `user`: it writes a real proposal to `~/.claude`.