mcp-tool-surface git:20260912.ebd6df5A

Immutable. This exact content never changes and is served at /api/v1/blob/23b0051263884fdc.

---
name: mcp-tool-surface
description: How to change the whiteboard MCP tool table — rename, re-describe, reshape, add, consolidate or retire a tool — with the two scoreboards and the LLM-driven lane that ADR-0031 makes the evidence. Use before touching a tool's name, description, input schema, annotations or existence under packages/server-core/src/tools/** or packages/mcp-server/src/server/mcp/**, when a consolidation is proposed, when someone says "too many tools", or when adding an eval task. Not for the daemon's HTTP routes or the web app.
---

# The tool surface is judged by an instrument

A tool definition is read by a model on every turn of every conversation
that has this server attached, and a model asked to do something reads the
table before it reads the document. So a change to the table is not judged
by whether the tests pass — they pass either way — but by four numbers
nobody sees in the diff: what the table costs to read, how much of it
explains itself, what an errand costs in calls, and whether a real model
still completes the errands. The criteria are
[ADR-0031](../../../docs/contributing/adr/0031-tool-surface-criteria.md)
§1 (C1–C14, each tagged with the rung that checks it); this skill is the
procedure.

The count is not a criterion. The last consolidation would have steered
by it, and the count cannot see the two largest debts the instrument found
on its first run: one tool being 45% of everything the model reads, and
299 of 324 parameters undescribed.

## The ladder, and which rung a change runs

| rung | what | command | runs |
|---|---|---|---|
| 1 | tool-surface scoreboard: per tool, model-visible bytes, wire bytes, description words, parameters and how many are undescribed, what a stray key gets, which neighbours the description names; plus totals, and C9 for every tool | `pnpm test --project mcp-node packages/mcp-server/src/server/mcp/tool-surface-quality.test.ts` | every push; pinned exactly |
| 2 | errand scoreboard: calls and bytes per errand | `pnpm test --project mcp-node packages/mcp-server/src/server/mcp/tool-call-count-quality.test.ts` | every push; pinned exactly |
| 3 | LLM-driven lane: a real model, only this server's tools, a seeded fixture, graded by outcome | `pnpm eval:tool-surface` (`--trials=3`, `--only=<substring>`, `--model=<alias>`, `--out=<file>`, `--dry-run`) | on demand; spends quota (~$0.06 a task, ~$0.70 a run); skips without the `claude` CLI |

| the change | rungs |
|---|---|
| a description or a `.describe()` on a parameter | 1, then 3 for the tasks that touch the tool — the one study that measured it found description fixes regress a sixth of cases, so a rewrite is not assumed to help |
| an input schema reshaped | 1, 2 |
| a tool added | 1, 2 (write its errand), 3 (write its task), plus the four-place list `mcp-smoke-coverage.ts` documents and a `smoke:e2e` call |
| a rename, a consolidation, a retirement | 1, 2, and 3 with `--trials=3` BEFORE and AFTER, both readings in the PR |
| a change to the server `instructions` | 3 |

## Procedure for a rename, consolidation or retirement

1. **Take the before reading.** On the base commit: rung 1 and 2 are the
   pinned values already in the tests; rung 3 is
   `pnpm eval:tool-surface --trials=3 --out=tmp/notes/tool-surface-before.json`.
   Three trials, because one is a reading and the pass column moves
   between trials on a tool that is reachable but not reliably reached
   (the tag-write task took three different routes in three trials).
2. **Make the change.** Rung 1 and 2 go red — that is the design. Re-pin
   each moved row, and write beside it WHY it moved, in the currency it
   moved in: bytes the model no longer reads, calls an errand no longer
   makes, parameters that now say what they are for. A row re-pinned with
   no reason is the regression the exact pin exists to refuse.
3. **Take the after reading** with the same command and trials.
4. **Decide by the columns, in this order.** C14 (rung-3 pass^k) may not
   fall. Then at least one of C1 (visible bytes), C13 (calls) or C3
   (undescribed) must have moved down, and the PR says which and by how
   much. A change that cut the count and moved none of those has changed
   the table's shape and nothing a model pays for.
5. **Read the calls column as a diagnostic, never as the gate.** A task
   that passes in seven calls where the errand scoreboard's cheapest path
   is two has not failed; it says C4 or C5 is unmet on the tools it
   wandered through, and the `tools` column says which. Grading by the
   path an author expected is what the sources call brittle, and the
   lane deliberately does not do it.
6. **Put both readings in the PR body**, the way `visual-evidence` puts
   two panels: the rung-1 totals before and after, the rung-2 rows that
   moved, and the rung-3 summary (pass^k, mean calls, cost) before and
   after. Retirements are breaking for MCP clients under the 0.0.x
   no-compat policy; say so in the title (`feat(mcp)!:`) and update the
   product skills under `./skills/*` — `skills-tool-surface.test.ts`
   catches a skill still naming a retired tool.

## In the development flow

- **Inline (the default for a surface change):** load this skill, take the
  before reading, change, re-pin, take the after reading, then run
  `review.workflow.mjs` over the diff with `dimensions` including
  `{ name: 'tool-surface', content: <resources/tool-surface.md> }` beside
  the defaults — the criteria file is
  `.claude/skills/review-gate/resources/tool-surface.md`, and the
  `review-gate` skill says how a caller globs and passes it.
- **Through `dev-loop`:** its design prompt asks for the rows the change
  will move and, for a rename or retirement, the lane's before/after in
  `testScenarios`; pass `reviewArgs: { dimensions: [...] }` so the
  composed review carries the dimension. The `developer` agent has no
  `claude` CLI and no quota, so the rung-3 readings are the main
  session's: take them before launching and after folding.
- **In the PR:** both readings, the way `visual-evidence` ships two
  panels. A reviewer reads C14 first, then which of C1/C13/C3 moved.

## Reading a rung-3 result

The lane prints one line per task and a summary; `--out` keeps the runs
with every tool call, tool-error text, token and cost figure.

- **A failure is probed against the store, not read off the transcript.**
  Two of the baseline's failures looked like model mistakes and were the
  product's: a tag-filtered search that cannot stand alone, and a comment
  thread that does not survive a restart. Reproduce what the model saw
  with `connectWhiteboard` from `scripts/eval/lib/whiteboard-client.mjs`
  against a seeded directory before concluding anything about the model.
- **`toolErrorTexts` is the C11 evidence, and sometimes the C5 evidence.**
  It is what the model was told when a call was refused. A refusal that
  names the wrong tool's parameter (`wb_document_list` telling the caller
  to pass `createWorkspace`) is a finding on the message and the fix is
  cheap; a refusal that says the tool cannot take what the task needs
  (`facets.tags` must match `{namespace}.{name}/v{n}`) is a gap in the
  surface, and the fix is a parameter.
- **An errand that keeps counting past a refusal is a scoreboard that
  lies.** The errand corpus's `call` throws on `isError` for this reason:
  two of its four rows had measured a refused write and a crashed seam
  as one cheap call each. A harness that drives real tools supplies every
  seam the tools reach (`InMemoryVersionHistory`), or the tool answers an
  error and the count reads as success.
- **`tools` says what the surface failed to offer — read the refusal
  texts before blaming the description.** Three trials of "tag this note"
  wandered through four tools, and the first reading was "the description
  never says tags". The refusal text said the tool could not take a tag at
  all: an errand step with no tool behind it (C5), which no description
  fixes. When a task wanders, read what each refusal told the model, then
  the description of the tool it should have reached, in that order.
- **A batch tool's line names the ARM the model picked.** `tools` prints
  `wb_canvas_edit[region.set]` or `wb_canvas_edit[node.add]`, because the
  tool name alone cannot say whether a declarative op was reached or the
  model got the same outcome with the imperative one — and that is the
  whole question a change to one arm's shape is judged on. The `--out`
  JSON keeps each call's `inputs` beside it, so whether the model declared
  geometry or left it to placement can be read rather than guessed.
- **A call AFTER the write is the lane grading the write's result.** The
  group-contents task passed every trial, and after a description got the
  model to leave geometry to placement, every trial then spent one more
  `node.patch` moving the placed box — placement had been putting it on
  top of what the group already held, since the op shipped. Read
  `inputs` for what the extra call undid; a pass with a repair call in it
  is a product finding, not a model one.
- **A description is a promise the refusal text has to keep.** `within`
  said "the group grows to fit"; a node given `within` and a position past
  the edge was still refused, and the lane hit that refusal three trials
  of three — the model had done what the description said it could. When
  a parameter's description names a behaviour, grep the refusals on that
  path for the case the description covers.
- **Look at the boards, not only the verdicts.** A write task that
  declares `boards: ['boards/x']` gets each named board rendered through
  `wb_scene_render` into `<out>-boards/` beside `--out`; rasterise with
  resvg from `packages/mcp-server` (it owns the binding) and read the
  PNGs. The first look found two rendering bugs — a cropped group label
  and a coloured group painted over its members — on drawings the grader
  had passed, because the grader reads the store and a person reads the
  picture. A layout task without its picture has been half-graded.
- **Read the `drawing` column before the picture.** Each board a write
  task names is also scored by canvas-render's `scoreDrawing` (the
  instrument `package-canvas-render.md` describes), and the run's line
  prints every non-zero DEBT column by name — `overlaps 1 straddles 1
  nearMisses 5` — with the full score under `drawing` in `--out`. A pass
  with debt is a drawing the verifier accepted and a reader would not;
  the column says which mistake, and `drawing-quality.test.ts`'s drafted
  rows show what each looks like on a board. `--dry-run --out=<file>`
  scores the fixture's own boards with no model call, which is how the
  wiring is checked.
- **Every read costs ~69k input tokens even at two calls.** The table is
  ~8.7k of that; the rest is the CLI's own system prompt. That is why the
  lane reports cost beside tokens, and why a change is judged on C1
  rather than on the lane's token column.
- **pass^k, not pass@1, is the reliability number.** A task that passes
  one trial in three is answerable and not reliably answered — the case
  the pair exists to separate.

## Adding a task to the lane

`scripts/eval/tasks.mjs`, following the rules in its header, which are
Anthropic's mcp-builder rules:

- **Read tasks** carry a single stable `answer` compared as a string.
  Phrase the question so it does not carry the target document's words
  or any tool's name — a task that can be answered by matching a word in
  a description measures the description, not the surface. The fixture
  (`scripts/eval/fixture.mjs`) is owned data, so every answer is true by
  construction; add what the task needs there.
- **Write tasks** carry a `verify(wb, ids)` that reads the state back
  through the real tools. Run `--dry-run` after adding one: the verifier
  must FAIL on the untouched fixture, or it would pass a model that did
  nothing. That is the mutation check, and the dry-run refuses to exit 0
  without it.
- **One task per question the surface should answer**, held out from the
  descriptions: refreshing tasks when descriptions are rewritten is part
  of the description increment, and a description may not quote a task.
- **A task that fails every trial on a frontier model is first suspected
  of being a broken task**, then of being a surface gap — the tag count
  was the second, and probing the store is what told them apart.

## Adding a tool

Beside the four-place list and the smoke call that
`mcp-smoke-coverage.ts` documents, a new tool lands with:

- every input parameter `.describe()`d — C3 counts down, never up;
- a `TOOL_PROFILES` entry with the honest worst-op annotation (C6, C7);
- the Zod OBJECT registered, not its `.shape`, so `.strict()` reaches
  the boundary and a stray key is refused by name (C10) — the way
  `wb_body_edit` and `wb_workspace_edit` already do;
- a description that says when to reach for it over its neighbours (C4);
- its rung-1 row, an errand in `mcp-errand-corpus.ts` if it changes what
  an errand costs, and a rung-3 task.

## Where the bytes go, and which cuts were free

Measured on the current table (ADR-0031 §3b): descriptions are ~30% of
what a model reads and are the useful bytes; the rest is schema, and the
schema carries three kinds of dead weight a `z.toJSONSchema` emits by
default. Check these before designing a bigger cut:

- **Safe-integer bounds.** `z.number().int()` emits `minimum:
  -9007199254740991, maximum: 9007199254740991` on every field — sixty
  times in one tool. Use the model's `integerSchema` /
  `nonnegativeIntegerSchema` (`.int()` validation, `.meta()` strips the
  bounds). `multipleOf(1)` is not the same integer: it accepts `1.4e-45`.
- **A stored union inlined per op arm.** The SDK round-trips a raw JSON
  Schema through `z.fromJSONSchema` and emits `$ref`s inlined, so dedup is
  impossible on the wire; it has to be the SOURCE schema. A flat
  write-side object narrowed on parse (`nodeExtensionWriteSchema`) cut
  the node extension from 487 bytes x8 to 290 x8, and made a broken embed
  a refusal instead of a silent drop.
- **`$schema` and `additionalProperties: false`.** 55 and 28 bytes a tool.
  The first is the SDK's and cannot be removed; the second is C10's price
  and is kept.

Two cuts that look available and are NOT, each refuted by measurement
rather than by argument. Both would pass every gate this repo has.

- **Registering a DESCRIBED LEAF deletes its description.** Naming a
  schema in zod's registry emits it into `$defs` once and `$ref`s it at
  each site, which is where the 5,177 bytes the five existing
  registrations save come from (measured by removing each one: `LineEnd`
  -2,540 visible, `EdgeEnd` -1,382, `Bends` -609, `CanvasColor` -595,
  `CanvasPoint` -51, `NodeEmbed` 0 — output schemas only). But a
  description INSIDE a registered object survives (`EdgeEnd` keeps all
  three of its) while a description ON the registered schema is dropped:
  `$defs.BoxWidth` came back `{"type":"number","minimum":0}` with the
  sentence gone. Registering the described `width`/`height` reads as -837
  bytes and 4 arms x (60 + 150) is the whole of it — the bytes ARE the
  descriptions. **Only a composite may be registered.**
- **`allOf` + `$ref` inheritance publishes a schema that refuses every
  valid call.** The four node arms spend 424 bytes each on the same
  shared base, 1,696 together, so extracting it is worth ~1,100. What
  `z.intersection` emits is `allOf: [{$ref}, {...additionalProperties:
  false}]`, and `additionalProperties` only sees its own subschema's
  properties. Checked with ajv 8 against 2020-12, three shapes side by
  side: flat (today) accepts valid input and refuses a stray key;
  allOf+`additionalProperties` **REFUSES valid input**; allOf +
  `unevaluatedProperties: false` is correct. zod 4.4.3 can READ
  `unevaluatedProperties` (`from-json-schema.ts`) and never EMITS it
  (`to-json-schema.ts`, zero occurrences), and handing the SDK a raw JSON
  Schema inlines the `$ref`s instead, so both doors are shut. The danger
  is the shape of the failure: zod validates with its own logic, so the
  server works, the tests pass, and only a client validating against the
  published schema breaks — silently and completely.

A description costs ~50 bytes each time it is emitted, and a field of the
node union is emitted once per arm per op: describing every parameter
would add ~40% to the table. Describe what the lane shows a model getting
wrong (the refusal texts), and let JSON Canvas fields mean what the spec
says.

## Traps, each paid for once

- **Run the lane from an EMPTY directory.** The `claude` CLI loads the
  working directory's `CLAUDE.md`, which in this repo names the tools the
  model is being tested on. The runner makes its own empty `cwd`; do not
  point it at a checkout to "save a copy".
- **The seeding server and the model's server are different processes.**
  Anything that survives only in memory is invisible to the model, and
  the first run found exactly that: comment threads and proposals were
  flattened to values by the workspace record's fold and came back
  unreadable after a restart (fixed; `comment-threads.durability.test.ts`
  is the shape of the test that crosses a reopen). A task that fails
  every trial on a read the fixture plainly supports is first checked
  for this: seed, close, reopen with `connectWhiteboard`, read.
- **`--only` is a substring over task names** — `--only=tag` matched two
  tasks. Quote the whole name for one.
- **Bytes, not tokens, on rung 1.** A tokenizer would make the pin depend
  on which model reads it. Divide by ~4 when quoting tokens, and say so.
- **Register the Zod OBJECT, never its `.shape`.** Handed a shape, the SDK
  rebuilds a non-strict object around it and a misspelt optional parameter
  is dropped with the call reporting success; `wb_facet_list`, every
  parameter of which is optional, answered the unfiltered list to any
  input at all. Every registration is the object now (C10), the rung-1
  `strays` column is `refused` for all 18, and a test holds it by name.
  The price was 29 visible bytes a tool for `additionalProperties: false`.