autonomous-computer-use git:20261102.b647ce4A

Immutable. This exact content never changes and is served at /api/v1/blob/347205ad50583ef2.

---
name: autonomous-computer-use
description: "Autonomous: Computer Use. Background-only macOS app automation with a private virtual input lane. Use whenever a task asks to open, inspect, click, type, edit, or send through a native desktop app such as Notes, Messages, Slack, Teams, Finder, System Settings, or Office. Use WorkIQ first for Microsoft 365 work (Outlook, Teams, OneDrive, SharePoint), and the Slack MCP server for Slack when connected. Keeps the user's mouse, keyboard, focus, and current app untouched; launches and drives target apps in the background; handles one-time permissions; validates ambiguous actions; prevents duplicate sends; and prefers APIs or browser automation when truly headless execution is available."
---

# Autonomous: Computer Use

Operate desktop applications without taking over the user's Mac. The user should be able to keep
working in another app while the task runs. Detailed targeting, editor, and app-specific gotchas are
in the
[`Computer Use reference`](../autonomous-skill-glossary/references/computer-use.md).

## The contract

1. **Use a private background input lane.** Address every read, click, key, scroll, drag, and text
   edit through the Computer Use tools with an explicit app and, when needed, window id. On macOS
   these app-targeted actions are the automation mouse and keyboard. Never activate the app, move
   the real pointer, synthesize foreground keys, or use AppleScript, System Events, shell launch
   commands, or window-manager tricks.
2. **Be invisible by default.** The target app may have a real window because native GUI automation
   is not literally headless, but it stays backgrounded. Do not ask the user to bring it forward,
   watch it, keep their hands off the whole laptop, or manage windows for you. If a supported API,
   browser tool, or CDP route can complete the task with no native window, prefer that truly
   headless route.
3. **Start automatically.** If Computer Use tools are available, use them without asking the user
   to run `/computer on` again. If the server is not yet exposed, use tool discovery once. Only
   surface the one-time setup when the tools remain unavailable.
4. **Share the machine.** User input in another app is harmless. If input in the target app causes
   `interrupted`, re-read that app and retry the same action once. If it keeps happening, defer that
   app, continue independent work, and return later rather than fighting for control.
5. **Fail closed.** A refused or ambiguous action is not permission to foreground the app or switch
   to unguarded OS input. Re-read, choose a safer semantic target, or stop that step with exact
   evidence.

## One-time macOS setup

The host needs:

```json
"chat.agentHost.computerUse.enabled": true
```

The first activation may use `/computer on`. Grant `Copilot Computer Use.app`:

- **Privacy & Security -> Screen & System Audio Recording**
- **Privacy & Security -> Accessibility**

Fully quit and reopen the Copilot host after changing either permission. If macOS attributes the
helper to the parent process and the helper still reports a missing grant, grant the active host
(for example, VS Code Insiders) under the same two panes.

Do not request Input Monitoring, Full Disk Access, Contacts, Automation, Camera, or Microphone for
ordinary desktop control. After screenshots/reads and clicks work, `no_viable_candidate` is a
safe-targeting problem, not a missing-permission problem.

## Headless-first routing

Choose the least intrusive route that can finish the task:

1. Service API or dedicated tool. For Microsoft 365 work, use WorkIQ (see below). For Slack, use the
   Slack MCP server when its tools are loaded.
2. Browser automation or CDP for browser-only and owned Electron surfaces.
3. Computer Use text/accessibility actions against a backgrounded native app.
4. Computer Use screenshot coordinates only when the accessibility tree cannot address the custom
   surface.
5. Stop that step if none is safe. Never fall back to foreground OS input.

Do not use Computer Use for simple file, folder, or URL opening when filesystem, shell, or browser
tools can do it without UI automation.

## Microsoft 365 with WorkIQ

WorkIQ reads and writes the signed-in user's Outlook, Teams, OneDrive, and SharePoint data without a
window. Use it before opening those apps:

- **Teams:** read with `fetch`, and send with `create_entity`, on `/me/chats/{chat-id}/messages` or
  `/teams/{team-id}/channels/{channel-id}/messages`.
- **Your own Teams self chat (Notes):** `/me/chats/48%3Anotes/messages` works for reads and creates,
  and it doesn't appear in the `/me/chats` listing. Create with jsonBody
  `{"body": {"contentType": "text", "content": "..."}}`.
- **Outlook:** mail, calendar, and contacts. Send with `do_action` on `/me/sendMail`. Reply,
  forward, and move with the matching `do_action` routes.
- **OneDrive and SharePoint:** search with `retrieve`, or with `call_function` on
  `/me/drive/root/search(q='...')`. Download file content with `fetch_blob`.

Call `search_paths` before using any path you have not verified. Use `fetch` with `$select` and
`$top`, and call `get_schema` only when a write returns 400. Use Computer Use for these apps only
when WorkIQ has no path for the step, and then apply the message and ambiguity rules below.

## Slack

Prefer the Slack MCP server when its tools are loaded in the session. It can search, read channels
and threads, send, schedule, and draft messages, and add reactions. Its OAuth user token may need
workspace admin approval, so connecting it needs the user's explicit approval.

- To message or read yourself, use your own Slack user ID as `channel_id`. The Slack tool
  descriptions state it for the current user. Sending and reading both resolve to the same self-DM.
- Slack tools can appear partway through a chat. If they're missing, start a new chat before
  troubleshooting the connection.
- If Slack isn't connected, use Computer Use for Slack and follow the composer and transcript rules
  in this skill.

## Operating loop

1. **Read text first.** Call `get_window_state` directly with a known app name. Use `list_apps` only
   when the app is unknown or a specific window is needed. Let `get_window_state` launch a missing
   app in the background.
2. **Verify the destination.** Confirm the window, document, conversation, recipient, or channel
   before entering content.
3. **Use the latest index.** Every state-changing action can invalidate element indexes. Re-read
   before the next action when the UI changed.
4. **Use semantic actions first.** Click or edit indexed accessibility nodes. Request a screenshot
   only when a custom surface has no usable node, then use coordinates from that exact image.
5. **Choose the advertised text primitive.**
   - Simple whole-value `(settable, string)` field: `set_value`.
   - `(patchable)` rich text, document, slide, Monaco, or Slack composer: `patch_text`.
   - Insertion at a proven focused input: `type_text`.
   - If the advertised primitive returns `no_viable_candidate`, re-read once and stop that route.
     Don't cycle other primitives. Use the app's API route, or report the blocker.
6. **Verify the effect from the returned state.** Do not take an extra screenshot merely to confirm
   text already exposed by accessibility.

For large Electron trees, reduce `max_children` while retaining enough `max_depth` to reach the main
view. Search offloaded output narrowly, then read only the relevant range.

## Ambiguous actions

Treat `dispatched_unverified`, `post_verify_failed`, timeouts, and unknown outcomes as actions that
may already have happened:

1. Do not repeat the action.
2. Fresh-read the target.
3. Inspect both input state and output state.
4. Continue only after proving the result.

For a message, inspect both the composer and transcript. A non-empty composer does not prove that
no message was sent, and an empty composer does not prove that exactly one message was sent.

For a WorkIQ send that times out or returns an unknown result, fetch the recent chat messages
(`/me/chats/{chat-id}/messages` with `$top=5` and `$select=id,body,createdDateTime`) or Sent Items
before retrying. A repeated create or sendMail can duplicate the message.

## Messages, forms, and other consequential actions

Respect the Computer Use action policy, which also covers WorkIQ and Slack sends. Sending a message
needs explicit approval of the exact recipient and body, whichever route sends it. Approval covers
one send.

Before sending:

- Verify the exact one-to-one recipient or channel.
- Verify the complete draft and formatting.
- Verify an ambiguous edit did not already post it.
- Ignore separate prompts to add, invite, subscribe, or notify someone unless separately approved.
- If WorkIQ cannot resolve a one-on-one chat (they may be missing from `/me/chats`), ask the user for
  the chat ID, or use Computer Use after verifying the chat title.

Send once. Then verify an empty composer and exactly one matching outgoing transcript entry. Never
delete a duplicate without separate approval.

For a send from another machine, start a remote session on that host (`create_remote_session`, using
the host ID from `list_agent_hosts`). Put the approval and exact text in its prompt, have it report to
`origin`, then verify each destination yourself before reporting. Use the user's exact words for the
text. If the approval leaves the text open, use the closest literal wording and state it in your
report.

## Keep the user out of the loop

Do not narrate routine setup, ask the user to watch the app, or request window management. Handle
transient failures by re-reading and trying the correct semantic route. Ask only when policy
requires approval or when a one-time OS grant is genuinely missing. During unattended work, record
the blocked step, continue everything else, and leave exact recovery instructions.