git:20260922.c4d457d to git:20260923.158603f

---

name: screen-annotation

description: Point at things on the screen the user is showing you

compatibility: "Designed for Vellum personal assistants"

metadata:

emoji: "🫵"

vellum:

display-name: "Screen Annotation"

category: "system"

activation-hints:

- "User asks where a control is, or to be walked through doing something themselves, in an app they share"

- "User wants to be shown, not have it done for them"

- "The answer is a place on the user's screen"

avoid-when:

- "User asks the assistant to do it (click, type, edit), even on a call: use computer-use"

- "Nothing is being shared, so there is no surface to point at"

---

Drawing on the screen the user is showing you, so they can go and do the

thing themselves.

This is the opposite errand from computer use. Nothing here clicks, types or

drives anything: the marks are a way of pointing while you talk, for someone

who wants to learn where a control is rather than have it operated for them.

A mark is drawn clear of what it indicates and never takes the mouse, so

what you point at stays visible and clickable the whole time.

## Requires a screen share

Marks are drawn on the frame around the surface the user is sharing with the

call. With nothing shared there is nowhere to draw, and `screen_point_at`

fails saying so. Ask them to share their screen from the call, then point.

## Say what to point at

The name is looked up on the surface itself, which knows where its controls

actually are, and an arrow is drawn at it.

**The label, not a description of it.** What is matched is the control's own

name. Casing, spacing and punctuation are forgiven, so `Color Balance` finds

`color balance`; nothing beyond that is, so "the stabilization button" finds

nothing, because no control is called that. Give the label on its own:

`stabilization`, `Send`, `Search`.

The arrow points at the middle of the control and stops just short, so what

you are sending someone to stays visible the whole time.

You are answered with what was drawn and the accessibility name it resolved

to. Some apps expose internal names such as `ToolbarExportControl` instead of

the visible label. When a returned name identifies the intended control in

the shared image, use that exact name as `target`; use the visible label in

the caption and speech. Do not invent an internal name or choose an unclear

candidate.

Fractions of the shared surface, `0` to `1`, measured against **the picture of

that surface you were last shown**. `x` and `y` are the top-left corner,

`width` and `height` the size. Give the bounds of the thing itself: the ring is

drawn around them, so a box tight on a button reads as a ring around that

button, and a box drawn where you think the ring should go puts the ring

outside that instead.

Moving the share is the one kind of drift that is caught for you. A mark

measured against the surface before the move is refused rather than drawn,

because those fractions land somewhere arbitrary on the surface that replaced

it. Wait for a frame of the new one and point again.

## How to point

**One thing at a time.** A mark is where to look next. A screen with four

marks on it is not four times as helpful; it is a diagram, and nobody knows

which one to start with. Point at the current step, talk, then point at the

next one.

**Captions are imperatives, not explanations.** "Click Share", "Type the name

here", "This is the tempo". Whatever else needs saying, say out loud: the

caption is drawn over the user's own work in a window they cannot scroll or

dismiss, and it is capped at 80 characters for that reason.

**Say it as well as draw it.** The marks are a gesture that accompanies

speech, the way a person points while explaining. A mark with no words is a

riddle.

**Take them down when they stop being true.** Call `screen_clear_marks` when

the step is done, when the user has moved on, or when the conversation has

left the screen behind. Marks come down on their own if the share ends or

moves, but a mark left standing over a finished step is one the user has to

work out is stale.

## Walking someone through several steps

Sometimes the answer is one pointer. Sometimes it is a route: four places to

click, in order, before the thing they asked about happens. Decide which it is

before you draw anything, because the two are paced differently.

**Say the route before you start it.** "There are three steps. First the

Share menu, then the format, then Export." If they asked how and you inferred

they want to be walked through it rather than told, this is where they wave it

off and just want the answer. Keep it to the count and the landmarks; the

detail belongs to each step as you reach it.

**One step, then stop.** Point at it, say what to do, and then wait. The

temptation is to narrate the next step while the mark for this one is still

up, and that leaves them doing step one with instructions for step two in

their ear. Silence is the cue that it is their turn.

**Advance on evidence, not on time.** Move to the next step when a fresh

picture shows this one done, or when they tell you it is. Do not move on

because a plausible amount of time has passed.

The most direct telling is automatic. When the user clicks the control an

arrow is pointing at, a message arrives as their turn saying they clicked it,

by name, and that mark comes down on its own. Treat it as the step done: say

what comes next and point at it. Nothing arrives for a click anywhere else,

and nothing arrives for a ring, so for those the picture and their words are

still the evidence. If the next picture shows the

step not done, or done to the wrong thing, point at the same place again with

a shorter caption and say what you saw. Pointing at the next step while the

previous one is still open is how someone ends up two steps behind a mark.

**Going back is just pointing again.** There is no undo. If they went past

something, or want to see step two again, point at step two. Say which step

it is, so the words and the mark agree about where you both are.

**Close it out.** When the last step is done, clear the marks and say so, in a

word. A mark left on the final button is one the user has to work out is

stale, and a walkthrough that ends without an ending leaves them waiting for

step five of four.

## Shapes