git:20260506.19937d7 to git:20260506.0234756

1 added, 1 removed. Audit A to A.

---

name: computer-use

description: "Use when driving native macOS apps -- click buttons, type text, screenshot windows -- without moving the user's cursor or stealing focus. Triggered by requests to control desktop apps (Slack, Notes, Xcode, Telegram Desktop, VS Code), automate macOS workflows, or take screenshots of native windows. macOS-only."

allowed-tools: Bash(valor-computer:*)

user-invocable: false

---

# Computer Use (macOS Desktop Control)

## When to Use

- User wants the agent to drive a native macOS app: Notes, Slack, Telegram Desktop, VS Code, Xcode, Finder, etc.

- Capturing screenshots of native (non-browser) app windows

- Automating multi-step desktop workflows (open app, type text, click button)

- Inspecting accessibility-tree state of any visible window

Do **not** use for:

- Browser automation -- that's BYOB MCP tools (`mcp__byob__browser_*`)

- Keyboard/mouse simulation that should move the user's actual cursor -- bcu drives windows headlessly via the macOS Accessibility API, deliberately leaving the user's pointer alone

## Platform Constraint

Computer-use is **macOS-only**. The `valor-computer` CLI enforces this at its entry point: on non-macOS hosts it prints `computer-use is macOS-only. This machine runs <platform>; skipping.` to stderr and exits 78 (`EX_CONFIG`). Skill body never reaches the bcu HTTP layer on Linux/Windows.

## Prerequisites

- bcu (background-computer-use) installed via `/setup` opt-in. The `/setup` skill prompts the user with "Do you want to enable computer-use?". On yes, it writes `~/.config/valor/computer-use-enabled`, downloads the bcu binary, and prompts the user to grant **Accessibility** + **Screen Recording** permissions in System Settings.

- bcu app must be running. It writes `$TMPDIR/background-computer-use/runtime-manifest.json` containing the loopback `base_url`. The CLI reads that manifest on every call. If absent, the CLI returns `{"error": "computer_use_unavailable", ...}` with a clear message and exits 78.

## Quick Start

```bash

# Discover what's open

valor-computer list_apps # all visible apps

valor-computer list_windows # all open windows

valor-computer list_windows --bundle-id com.apple.Notes

# Inspect a window's AX tree

valor-computer get_window_state <window_id>

# Drive the window

valor-computer click <window_id> --x 100 --y 200

valor-computer type_text <window_id> "Hello world"

valor-computer screenshot_window <window_id> --output /tmp/notes.png

# Press a key with modifiers

valor-computer press_key <window_id> a --mod cmd # Cmd-A

valor-computer press_key <window_id> return # Return

```

## Core Workflow

```

1. list_apps # find the bundle_id

2. list_windows --bundle-id ... # pick the window_id

3. get_window_state <window_id> # get AX tree for inspection (optional)

4. click / type_text / press_key # drive the window

5. screenshot_window <window_id> # capture proof

```

## Electron Apps (Race 3 Mitigation)

Electron apps lazily build their accessibility tree, so an AX node ref returned by `get_window_state` can become invalid before your next call -- even with the window still open. Known Electron bundles include:

- `com.tinyspeck.slackmacgap` (Slack)

- `com.microsoft.VSCode` (VS Code)

- `org.telegram.desktop` (Telegram Desktop)

- `com.hnc.Discord` (Discord)

- `com.electron.notion`, `com.figma.Desktop`, `com.spotify.client`

For these targets, **pass a `--selector` JSON instead of a raw `--ref`**. The module re-queries `get_window_state` internally on every call and resolves the selector to a fresh AX ref:

```bash

# Click the "Send" button in Slack regardless of stale refs

valor-computer click <slack_window_id> \

--selector '{"role":"AXButton","label":"Send","bundle_id":"com.tinyspeck.slackmacgap"}'

# Set the value of a Discord text field

valor-computer set_value <discord_window_id> "hello" \

--selector '{"role":"AXTextField","label":"Message","bundle_id":"com.hnc.Discord"}'

```

The `bounds` field (a `[x, y, w, h]` list) tie-breaks when multiple AX nodes match `role` + `label`.

## Loopback-Only

The bcu HTTP server binds to `127.0.0.1` only. There is no remote control surface. All requests go through `urllib.request` to the loopback URL stored in `$TMPDIR/background-computer-use/runtime-manifest.json`.

## Browser vs Desktop

Browser automation is BYOB MCP (`mcp__byob__browser_*`) — real Chrome, the user's logged-in session. `computer-use` operates on **native macOS apps**, not browsers. If the agent asks "click this button on a webpage", route to BYOB.

## BYOB and `BYOB_ALLOW_EVAL`

- When BYOB MCP tools are invoked, the registrar at `scripts/update/mcp_byob.py` keeps `BYOB_ALLOW_EVAL=0` -- `browser_eval` (arbitrary JS execution) stays disabled by default per the BYOB security model. Computer-use does not interact with BYOB; this note is here so the agent does not conflate the two surfaces.

+ When BYOB MCP tools are invoked, the registrar at `scripts/update/mcp_byob.py` keeps `BYOB_ALLOW_EVAL=1` -- `browser_eval` is enabled by default in this repo so skills like `mermaid-render`, `do-discover-paths`, and `do-design-system` work out of the box. Computer-use does not interact with BYOB; this note is here so the agent does not conflate the two surfaces.

## Common Workflows

### Open Notes, type, screenshot

```bash

# 1. Find Notes

valor-computer list_apps

# pick bundle_id "com.apple.Notes"

# 2. Find the window

valor-computer list_windows --bundle-id com.apple.Notes

# pick window_id, e.g. 12345

# 3. Click in the body (use coords for stable native apps)

valor-computer click 12345 --x 400 --y 300

# 4. Type

valor-computer type_text 12345 "Reminder: ship plan #1256"

# 5. Verify

valor-computer screenshot_window 12345 --output /tmp/notes-after.png

```

### Drive Slack via selector (Electron)

```bash

valor-computer list_windows --bundle-id com.tinyspeck.slackmacgap

# pick slack_window_id

# Click the channel "engineering"

valor-computer click <slack_window_id> \

--selector '{"role":"AXStaticText","label":"engineering","bundle_id":"com.tinyspeck.slackmacgap"}'

# Click the message composer

valor-computer click <slack_window_id> \

--selector '{"role":"AXTextArea","label":"Message engineering","bundle_id":"com.tinyspeck.slackmacgap"}'

# Type a message

valor-computer type_text <slack_window_id> "Build complete"

# Send (cmd-return)

valor-computer press_key <slack_window_id> return --mod cmd

```

## Error Handling

- `{"error": "computer_use_unavailable", ...}` -> bcu not installed, not opted in, or not running. Exit code 78. Tell the user to run `/setup` and answer "yes" to the computer-use opt-in.

- `{"error": "window_not_found", "window_id": N}` -> the window closed between `list_windows` and the action. Re-call `list_windows` and retry.

- `{"error": "selector_no_match", "selector": ...}` -> the selector didn't resolve. Inspect via `get_window_state` and refine the role/label/bounds.

- `{"error": "timeout", ...}` -> bcu took longer than 10s. Bcu is loopback HTTP; transient timeouts are unusual -- check that the bcu app is responsive.