--- name: mcp-tool-surface description: How to change the whiteboard MCP tool table — rename, re-describe, reshape, add, consolidate or retire a tool — with the two scoreboards and the LLM-driven lane that ADR-0031 makes the evidence. Use before touching a tool's name, description, input schema, annotations or existence under packages/server-core/src/tools/** or packages/mcp-server/src/server/mcp/**, when a consolidation is proposed, when someone says "too many tools", or when adding an eval task. Not for the daemon's HTTP routes or the web app. --- # The tool surface is judged by an instrument A tool definition is read by a model on every turn of every conversation that has this server attached, and a model asked to do something reads the table before it reads the document. So a change to the table is not judged by whether the tests pass — they pass either way — but by four numbers nobody sees in the diff: what the table costs to read, how much of it explains itself, what an errand costs in calls, and whether a real model still completes the errands. The criteria are [ADR-0031](../../../docs/contributing/adr/0031-tool-surface-criteria.md) §1 (C1–C14, each tagged with the rung that checks it); this skill is the procedure. The count is not a criterion. The last consolidation would have steered by it, and the count cannot see the two largest debts the instrument found on its first run: one tool being 45% of everything the model reads, and 299 of 324 parameters undescribed. ## The ladder, and which rung a change runs | rung | what | command | runs | |---|---|---|---| | 1 | tool-surface scoreboard: per tool, model-visible bytes, wire bytes, description words, parameters and how many are undescribed, what a stray key gets, which neighbours the description names; plus totals, and C9 for every tool | `pnpm test --project mcp-node packages/mcp-server/src/server/mcp/tool-surface-quality.test.ts` | every push; pinned exactly | | 2 | errand scoreboard: calls and bytes per errand | `pnpm test --project mcp-node packages/mcp-server/src/server/mcp/tool-call-count-quality.test.ts` | every push; pinned exactly | | 3 | LLM-driven lane: a real model, only this server's tools, a seeded fixture, graded by outcome | `pnpm eval:tool-surface` (`--trials=3`, `--only=`, `--model=`, `--out=`, `--dry-run`) | on demand; spends quota (~$0.06 a task, ~$0.70 a run); skips without the `claude` CLI | | the change | rungs | |---|---| | a description or a `.describe()` on a parameter | 1, then 3 for the tasks that touch the tool — the one study that measured it found description fixes regress a sixth of cases, so a rewrite is not assumed to help | | an input schema reshaped | 1, 2 | | a tool added | 1, 2 (write its errand), 3 (write its task), plus the four-place list `mcp-smoke-coverage.ts` documents and a `smoke:e2e` call | | a rename, a consolidation, a retirement | 1, 2, and 3 with `--trials=3` BEFORE and AFTER, both readings in the PR | | a change to the server `instructions` | 3 | ## Procedure for a rename, consolidation or retirement 1. **Take the before reading.** On the base commit: rung 1 and 2 are the pinned values already in the tests; rung 3 is `pnpm eval:tool-surface --trials=3 --out=tmp/notes/tool-surface-before.json`. Three trials, because one is a reading and the pass column moves between trials on a tool that is reachable but not reliably reached (the tag-write task took three different routes in three trials). 2. **Make the change.** Rung 1 and 2 go red — that is the design. Re-pin each moved row, and write beside it WHY it moved, in the currency it moved in: bytes the model no longer reads, calls an errand no longer makes, parameters that now say what they are for. A row re-pinned with no reason is the regression the exact pin exists to refuse. 3. **Take the after reading** with the same command and trials. 4. **Decide by the columns, in this order.** C14 (rung-3 pass^k) may not fall. Then at least one of C1 (visible bytes), C13 (calls) or C3 (undescribed) must have moved down, and the PR says which and by how much. A change that cut the count and moved none of those has changed the table's shape and nothing a model pays for. 5. **Read the calls column as a diagnostic, never as the gate.** A task that passes in seven calls where the errand scoreboard's cheapest path is two has not failed; it says C4 or C5 is unmet on the tools it wandered through, and the `tools` column says which. Grading by the path an author expected is what the sources call brittle, and the lane deliberately does not do it. 6. **Put both readings in the PR body**, the way `visual-evidence` puts two panels: the rung-1 totals before and after, the rung-2 rows that moved, and the rung-3 summary (pass^k, mean calls, cost) before and after. Retirements are breaking for MCP clients under the 0.0.x no-compat policy; say so in the title (`feat(mcp)!:`) and update the product skills under `./skills/*` — `skills-tool-surface.test.ts` catches a skill still naming a retired tool. ## In the development flow - **Inline (the default for a surface change):** load this skill, take the before reading, change, re-pin, take the after reading, then run `review.workflow.mjs` over the diff with `dimensions` including `{ name: 'tool-surface', content: }` beside the defaults — the criteria file is `.claude/skills/review-gate/resources/tool-surface.md`, and the `review-gate` skill says how a caller globs and passes it. - **Through `dev-loop`:** its design prompt asks for the rows the change will move and, for a rename or retirement, the lane's before/after in `testScenarios`; pass `reviewArgs: { dimensions: [...] }` so the composed review carries the dimension. The `developer` agent has no `claude` CLI and no quota, so the rung-3 readings are the main session's: take them before launching and after folding. - **In the PR:** both readings, the way `visual-evidence` ships two panels. A reviewer reads C14 first, then which of C1/C13/C3 moved. ## Reading a rung-3 result The lane prints one line per task and a summary; `--out` keeps the runs with every tool call, tool-error text, token and cost figure. - **A failure is probed against the store, not read off the transcript.** Two of the baseline's failures looked like model mistakes and were the product's: a tag-filtered search that cannot stand alone, and a comment thread that does not survive a restart. Reproduce what the model saw with `connectWhiteboard` from `scripts/eval/lib/whiteboard-client.mjs` against a seeded directory before concluding anything about the model. - **`toolErrorTexts` is the C11 evidence, and sometimes the C5 evidence.** It is what the model was told when a call was refused. A refusal that names the wrong tool's parameter (`wb_document_list` telling the caller to pass `createWorkspace`) is a finding on the message and the fix is cheap; a refusal that says the tool cannot take what the task needs (`facets.tags` must match `{namespace}.{name}/v{n}`) is a gap in the surface, and the fix is a parameter. - **An errand that keeps counting past a refusal is a scoreboard that lies.** The errand corpus's `call` throws on `isError` for this reason: two of its four rows had measured a refused write and a crashed seam as one cheap call each. A harness that drives real tools supplies every seam the tools reach (`InMemoryVersionHistory`), or the tool answers an error and the count reads as success. - **`tools` says what the surface failed to offer — read the refusal texts before blaming the description.** Three trials of "tag this note" wandered through four tools, and the first reading was "the description never says tags". The refusal text said the tool could not take a tag at all: an errand step with no tool behind it (C5), which no description fixes. When a task wanders, read what each refusal told the model, then the description of the tool it should have reached, in that order. - **A batch tool's line names the ARM the model picked.** `tools` prints `wb_canvas_edit[region.set]` or `wb_canvas_edit[node.add]`, because the tool name alone cannot say whether a declarative op was reached or the model got the same outcome with the imperative one — and that is the whole question a change to one arm's shape is judged on. The `--out` JSON keeps each call's `inputs` beside it, so whether the model declared geometry or left it to placement can be read rather than guessed. - **A call AFTER the write is the lane grading the write's result.** The group-contents task passed every trial, and after a description got the model to leave geometry to placement, every trial then spent one more `node.patch` moving the placed box — placement had been putting it on top of what the group already held, since the op shipped. Read `inputs` for what the extra call undid; a pass with a repair call in it is a product finding, not a model one. - **A description is a promise the refusal text has to keep.** `within` said "the group grows to fit"; a node given `within` and a position past the edge was still refused, and the lane hit that refusal three trials of three — the model had done what the description said it could. When a parameter's description names a behaviour, grep the refusals on that path for the case the description covers. - **Look at the boards, not only the verdicts.** A write task that declares `boards: ['boards/x']` gets each named board rendered through `wb_scene_render` into `-boards/` beside `--out`; rasterise with resvg from `packages/mcp-server` (it owns the binding) and read the PNGs. The first look found two rendering bugs — a cropped group label and a coloured group painted over its members — on drawings the grader had passed, because the grader reads the store and a person reads the picture. A layout task without its picture has been half-graded. - **Read the `drawing` column before the picture.** Each board a write task names is also scored by canvas-render's `scoreDrawing` (the instrument `package-canvas-render.md` describes), and the run's line prints every non-zero DEBT column by name — `overlaps 1 straddles 1 nearMisses 5` — with the full score under `drawing` in `--out`. A pass with debt is a drawing the verifier accepted and a reader would not; the column says which mistake, and `drawing-quality.test.ts`'s drafted rows show what each looks like on a board. `--dry-run --out=` scores the fixture's own boards with no model call, which is how the wiring is checked. - **Every read costs ~69k input tokens even at two calls.** The table is ~8.7k of that; the rest is the CLI's own system prompt. That is why the lane reports cost beside tokens, and why a change is judged on C1 rather than on the lane's token column. - **pass^k, not pass@1, is the reliability number.** A task that passes one trial in three is answerable and not reliably answered — the case the pair exists to separate. ## Adding a task to the lane `scripts/eval/tasks.mjs`, following the rules in its header, which are Anthropic's mcp-builder rules: - **Read tasks** carry a single stable `answer` compared as a string. Phrase the question so it does not carry the target document's words or any tool's name — a task that can be answered by matching a word in a description measures the description, not the surface. The fixture (`scripts/eval/fixture.mjs`) is owned data, so every answer is true by construction; add what the task needs there. - **Write tasks** carry a `verify(wb, ids)` that reads the state back through the real tools. Run `--dry-run` after adding one: the verifier must FAIL on the untouched fixture, or it would pass a model that did nothing. That is the mutation check, and the dry-run refuses to exit 0 without it. - **One task per question the surface should answer**, held out from the descriptions: refreshing tasks when descriptions are rewritten is part of the description increment, and a description may not quote a task. - **A task that fails every trial on a frontier model is first suspected of being a broken task**, then of being a surface gap — the tag count was the second, and probing the store is what told them apart. ## Adding a tool Beside the four-place list and the smoke call that `mcp-smoke-coverage.ts` documents, a new tool lands with: - every input parameter `.describe()`d — C3 counts down, never up; - a `TOOL_PROFILES` entry with the honest worst-op annotation (C6, C7); - the Zod OBJECT registered, not its `.shape`, so `.strict()` reaches the boundary and a stray key is refused by name (C10) — the way `wb_body_edit` and `wb_workspace_edit` already do; - a description that says when to reach for it over its neighbours (C4); - its rung-1 row, an errand in `mcp-errand-corpus.ts` if it changes what an errand costs, and a rung-3 task. ## Where the bytes go, and which cuts were free Measured on the current table (ADR-0031 §3b): descriptions are ~30% of what a model reads and are the useful bytes; the rest is schema, and the schema carries three kinds of dead weight a `z.toJSONSchema` emits by default. Check these before designing a bigger cut: - **Safe-integer bounds.** `z.number().int()` emits `minimum: -9007199254740991, maximum: 9007199254740991` on every field — sixty times in one tool. Use the model's `integerSchema` / `nonnegativeIntegerSchema` (`.int()` validation, `.meta()` strips the bounds). `multipleOf(1)` is not the same integer: it accepts `1.4e-45`. - **A stored union inlined per op arm.** The SDK round-trips a raw JSON Schema through `z.fromJSONSchema` and emits `$ref`s inlined, so dedup is impossible on the wire; it has to be the SOURCE schema. A flat write-side object narrowed on parse (`nodeExtensionWriteSchema`) cut the node extension from 487 bytes x8 to 290 x8, and made a broken embed a refusal instead of a silent drop. - **`$schema` and `additionalProperties: false`.** 55 and 28 bytes a tool. The first is the SDK's and cannot be removed; the second is C10's price and is kept. Two cuts that look available and are NOT, each refuted by measurement rather than by argument. Both would pass every gate this repo has. - **Registering a DESCRIBED LEAF deletes its description.** Naming a schema in zod's registry emits it into `$defs` once and `$ref`s it at each site, which is where the 5,177 bytes the five existing registrations save come from (measured by removing each one: `LineEnd` -2,540 visible, `EdgeEnd` -1,382, `Bends` -609, `CanvasColor` -595, `CanvasPoint` -51, `NodeEmbed` 0 — output schemas only). But a description INSIDE a registered object survives (`EdgeEnd` keeps all three of its) while a description ON the registered schema is dropped: `$defs.BoxWidth` came back `{"type":"number","minimum":0}` with the sentence gone. Registering the described `width`/`height` reads as -837 bytes and 4 arms x (60 + 150) is the whole of it — the bytes ARE the descriptions. **Only a composite may be registered.** - **`allOf` + `$ref` inheritance: CLOSED, and not on the axis first claimed.** The four node arms spend 424 bytes each on the same shared base, 1,696 together, so extracting it looks worth ~1,100. Four things were measured, in the order they were found, and each on its own settles it: 1. **It never reaches the table.** The published node draft is derived `spatialNodeSchema.options` -> `.omit()` -> `.partial()` -> `.extend()` -> `z.discriminatedUnion`. Every step requires a ZodObject and every step FLATTENS, so making the stored schema an intersection changes the table by nothing. Reaching it means building the write-side drafts as intersections directly, which costs `discriminatedUnion` (refusal quality), `NODE_DRAFT_KEYS`, and the free derivation the code comment calls "the point". 2. **It does not compile or run.** An intersection is not an object: `.omit()` is gone (four sites) and `z.infer` degrades to `{}` / `unknown` across `canvas-edit.ts`. The server does not start. 3. **Spec-conformance and C10 cannot both hold.** ajv 8, 2020-12, four shapes: flat + strict accepts valid input and refuses a stray key; `allOf` with both halves strict REFUSES valid input; `allOf` with neither strict accepts a stray key (C10 lost); only `allOf` + `unevaluatedProperties` is correct, and zod 4.4.3 reads that keyword without ever emitting it. 4. **The model axis was never the blocker.** The table already publishes five `$ref`s and the round-14 reading was pass^3 1.0 / debtFree^3 1.0, so `$ref` comprehension is established. Only `allOf` specifically is unmeasured — and a rung-3 run would not move any of 1-3, which is why it was not spent. The first record of this said "publishes a schema that refuses every valid call" and stopped there. That is true of the artifact and was the wrong thing to lead with: ajv answers spec-conformance (defensibly — both MCP SDKs bundle it, and `fromJsonSchema` uses it for raw-JSON-Schema tools), and says nothing about C14. It also asserted "only a client validating against the published schema breaks" with no client checked. Checked since: this server validates with zod, and the MCP client SDK does not validate outgoing arguments against `inputSchema` at all. **Nothing else in the table is worth registering.** Re-scanned after the node extension landed: every remaining repeated fragment is a described leaf, which is the refuted shape above. The one exception is a description-free record type (`{type: object, propertyNames, additional Properties}`, four sites, ~99 bytes) — safe by the rule and 0.26% of the table, so not worth an increment on its own. **The same multiplication prices an ADDITION, and usually refuses it.** Registration is attractive because a fragment repeats per arm; accommodation is expensive for exactly that reason, and the arithmetic runs the other way. Worked case: a lane trial wrote `stencil` INSIDE `node` — where every other property of the box goes, and where a person thinks it belongs ("this box is a database") — and lost the batch to the refusal. Accepting it where the model already reaches looks like a one-field change. Measured on the four-arm node draft, `node.add` alone: | shape | visibleBytes | parameters | undescribed | |---|---|---|---| | refuse + redirect (ships today) | 38,300 | 345 | 221 | | accept in `node`, described | 39,320 (**+1,020**) | 349 | 221 | | accept in `node`, undescribed | 38,604 (**+304**) | 349 | 225 (**+4**) | Neither shape moves C1, C13 or C3 DOWN, so neither clears §1's rule on its own; the only column that could justify either is C13. **Its ceiling is already known from the before-reading, which is why no rung-3 run was spent**: the misplacement hit one task, one trial in three, and the trial still PASSED — the refusal names the key, says it is not a node field, and says where it goes, so the model repaired it in one call. The table is read on every turn of every conversation; a ten-turn conversation pays the cheap shape's +304 bytes ten times to save at most that one call. The reason the trade is this lopsided is that **the expensive half of the problem was already fixed by a change that cost nothing**. When the draft STRIPPED, the identical mistake was accepted silently, the boxes were drawn undressed, and one trial spent seventeen further calls rebuilding the vocabulary by hand. Making the draft `.strict()` with a redirect turned that into one refused call. What remained was a retry, and a retry is not worth 1,020 bytes a turn — nor 304. **So: price the refusal before buying the accommodation.** A good error message is the cheapest thing on this table — it costs bytes only when it fires, while a field costs bytes on every turn forever. Reach for the schema only when the refusal cannot name the repair. ## Traps, each paid for once - **Run the lane from an EMPTY directory.** The `claude` CLI loads the working directory's `CLAUDE.md`, which in this repo names the tools the model is being tested on. The runner makes its own empty `cwd`; do not point it at a checkout to "save a copy". - **The seeding server and the model's server are different processes.** Anything that survives only in memory is invisible to the model, and the first run found exactly that: comment threads and proposals were flattened to values by the workspace record's fold and came back unreadable after a restart (fixed; `comment-threads.durability.test.ts` is the shape of the test that crosses a reopen). A task that fails every trial on a read the fixture plainly supports is first checked for this: seed, close, reopen with `connectWhiteboard`, read. - **`--only` is a substring over task names** — `--only=tag` matched two tasks. Quote the whole name for one. - **Bytes, not tokens, on rung 1.** A tokenizer would make the pin depend on which model reads it. Divide by ~4 when quoting tokens, and say so. - **Register the Zod OBJECT, never its `.shape`.** Handed a shape, the SDK rebuilds a non-strict object around it and a misspelt optional parameter is dropped with the call reporting success; `wb_facet_list`, every parameter of which is optional, answered the unfiltered list to any input at all. Every registration is the object now (C10), the rung-1 `strays` column is `refused` for all 18, and a test holds it by name. The price was 29 visible bytes a tool for `additionalProperties: false`.