computer-use ยท diff
git:20260828.b89ba11 to git:20260901.2b8f73c
116 added, 180 removed. Audit A to A.
---
name: computer-use
description: Control local desktop applications through Computer Use for tasks that require reading or operating app UI. Prefer purpose-built connectors, APIs, or CLIs when available.
---
- ## `node_repl` + `@qwen-code/cua-sdk` (Computer Use)
-
- - Use `node_repl` (JavaScript) for all Computer Use actions.
- - Do not use other technologies besides `node_repl` for computer interactions
- unless specifically requested by the user. This includes AppleScript,
- `osascript`, JXA, System Events, and synthesized input.
- - Prefer a dedicated plugin or skill when it can complete the task; use
- Computer Use for app interactions that are not exposed through a more
- specific interface.
- - Use only the typed `ComputerUse` API. Do not use a generic SDK `callTool`,
- direct `CuaDriver` imports, or a Qwen global bridge.
- - `node_repl` state persists across calls.
- - Use `nodeRepl.write(...)` for text output. It takes a string, so wrap objects
- with `JSON.stringify(...)`.
-
- ## Install automatically
-
- Run `qwen mcp list` to check whether the `node-repl` server is configured. If it
- is not configured, run both commands yourself:
+ # Computer Use with the CUA SDK
- ```bash
- qwen mcp add --scope user node-repl npx -y @qwen-code/node-repl-mcp@0.1.1
- npm install --no-save --package-lock=false @qwen-code/cua-sdk@0.20.2
- ```
+ - Prefer a dedicated connector or API. Use Computer Use only for UI state or
+ interactions the dedicated interface does not expose.
+ - Perform Computer Use through `node_repl` and the typed `ComputerUse` API.
+ Do not use generic `callTool`, direct driver imports, AppleScript, JXA, or
+ synthesized-input utilities.
+ - Observe the exact current window before acting. Prefer current element tokens
+ over screenshot coordinates; use coordinates only when accessibility is
+ incomplete and the screenshot provides the target.
+ - Treat an action result as delivery evidence, not task completion. Decide from
+ fresh state and require stable postcondition evidence.
- Tell the user to restart Qwen Code after adding the MCP server, then stop. Do
- not ask the user to copy or run the commands.
+ ## Setup
- If `node_repl` is available but the SDK import fails, run the SDK installation
- command yourself from the current workspace, then retry the import:
+ If `node_repl` is unavailable, run these commands yourself:
```bash
- npm install --no-save --package-lock=false @qwen-code/cua-sdk@0.20.2
+ qwen mcp add --scope user node-repl npx -y @qwen-code/node-repl-mcp@0.1.2
+ npm install --no-save --package-lock=false @qwen-code/cua-sdk@0.20.3
```
- ## Bootstrap
+ Tell the user to restart Qwen Code, then stop. If only the SDK import is
+ missing, run the second command and retry.
- Import the SDK once per fresh `node_repl` kernel:
+ Create one persistent client per REPL kernel:
```js
globalThis.computer = await (
await import('@qwen-code/cua-sdk/computer-use')
).ComputerUse.create();
- ```
-
- ## API surface
-
- ```ts
- type WindowTarget = { pid: number; windowId: number };
- type ElementTarget = {
- pid: number;
- windowId?: number;
- elementToken: string;
- };
- type CoordinateTarget = WindowTarget & { x: number; y: number };
- type PointOrElementTarget = CoordinateTarget | ElementTarget;
- type App = {
- name?: string;
- bundle_id?: string;
- pid?: number;
- running?: boolean;
- launch_path?: string;
- };
- type Window = {
- window_id: number;
- title?: string;
- is_on_screen?: boolean;
- on_current_space?: boolean;
- };
- type Element = {
- element_token?: string;
- role?: string;
- label?: string;
- value?: unknown;
- actions?: string[];
- };
-
- type ComputerUse = {
- listApps: () => Promise<App[]>;
- listWindows: (args?: {
- pid?: number;
- onScreenOnly?: boolean;
- }) => Promise<Window[]>;
- observeWindow: (
- args: WindowTarget & {
- baseRevisionId?: string;
- forceFull?: boolean;
- includeScreenshot?: boolean;
- },
- ) => Promise<{
- text: string;
- elements: Element[];
- revisionId?: string;
- screenshot?: { images: object[] };
- }>;
- click: (
- args: PointOrElementTarget & {
- button?: 'left' | 'right' | 'middle';
- count?: number;
- },
- ) => Promise<object>;
- doubleClick: (args: PointOrElementTarget) => Promise<object>;
- rightClick: (args: PointOrElementTarget) => Promise<object>;
- setValue: (args: ElementTarget & { value: string }) => Promise<object>;
- typeText: (args: WindowTarget & { text: string }) => Promise<object>;
- pressKey: (args: WindowTarget & { key: string }) => Promise<object>;
- hotkey: (args: WindowTarget & { keys: string[] }) => Promise<object>;
- scroll: (
- args: PointOrElementTarget & {
- direction: 'up' | 'down' | 'left' | 'right';
- by?: 'line' | 'page';
- amount?: number;
- },
- ) => Promise<object>;
- drag: (
- args: WindowTarget & {
- fromX: number;
- fromY: number;
- toX: number;
- toY: number;
- deliveryMode?: 'background' | 'foreground';
- },
- ) => Promise<object>;
- performSecondaryAction: (
- args: ElementTarget & { action: string },
- ) => Promise<object>;
- close: () => Promise<void>;
- };
+ globalThis.cuaRevisions ??= new Map();
```
- ## Workflow
+ ## Target and observe
- ### 1. Initialize
+ Use `listApps({signal:nodeRepl.signal})` and filter in JavaScript; print only
+ likely matches. After selecting a real PID, call
+ `listWindows({pid,signal:nodeRepl.signal})` and choose from returned
+ metadata. Never guess a PID, window ID, element token, or coordinate. If the
+ app is not running, start it with ordinary Node.js process APIs and refresh the
+ lists.
- Resolve the exact running application and window named by the task. Filter
- inside `node_repl`; do not print the entire application list:
+ Maintain one revision cursor per window surface. The first observation has no
+ base; later observations use only the last revision actually consumed for that
+ same surface:
```js
- var apps = await computer.listApps();
- var matches = apps.filter(
- (app) => app.name === 'Target App' || app.bundle_id === 'com.example.target',
- );
- nodeRepl.write(JSON.stringify(matches));
+ globalThis.observeCuaWindow = async (target, options = {}) => {
+ const key = `${target.pid}:${target.windowId}`;
+ const state = await computer.observeWindow({
+ ...target,
+ ...options,
+ baseRevisionId: cuaRevisions.get(key),
+ signal: nodeRepl.signal,
+ });
+ if (state.revisionId) cuaRevisions.set(key, state.revisionId);
+ return state;
+ };
```
- After choosing the application from returned metadata, list only its windows:
+ Use accessibility text for efficient decisions. Request a screenshot when the
+ tree is incomplete, visual layout matters, or action evidence conflicts with
+ the tree. Emit only decision-relevant images:
```js
- var pid = matches[0].pid;
- var windows = await computer.listWindows({ pid });
- nodeRepl.write(JSON.stringify(windows));
+ for (const image of state.screenshot?.images ?? []) {
+ if (image?.dataBase64 && image?.mimeType) {
+ await nodeRepl.emitImage(
+ `data:${image.mimeType};base64,${image.dataBase64}`,
+ );
+ }
+ }
```
- Choose the window from returned metadata, then get its current accessibility
- state:
-
- ```js
- var target = { pid, windowId: windows[0].window_id };
- var state = await computer.observeWindow({ ...target, forceFull: true });
- nodeRepl.write(state.text);
- ```
+ If the SDK explicitly reports a missing/invalid base or a stale lineage,
+ perform one observation with `forceFull: true`, replace that surface's cursor,
+ then resume the normal helper. Do not make full observations the default.
- Never guess a PID, window ID, coordinate, or element token. `ComputerUse`
- discovers running applications but does not launch them; if necessary, start
- the application from `node_repl` with ordinary Node.js process APIs, then
- refresh the application and window lists.
+ ## Act and verify
- ### 2. Act and get the latest state
+ Choose the narrowest action supported by current state. Pass an observed
+ `element_token` as `elementToken`. Use `performSecondaryAction` only when that
+ exact action appears in the element's current `actions` list.
- Choose only the action needed for the user's task. Prefer current
- `element_token` values over coordinates. Pass an observed `element_token` as
- the camel-case `elementToken` action field:
+ Before acting, state a concrete observable postcondition. For postconditions
+ expressible as window or element state, use `actAndVerify` with `verifyState`:
```js
- await computer.setValue({
- ...target,
- elementToken,
- value: 'hello',
- });
-
- state = await computer.observeWindow({
- ...target,
- baseRevisionId: state.revisionId,
- });
- nodeRepl.write(state.text);
+ try {
+ globalThis.lastCuaOutcome = await computer.actAndVerify({
+ action: () =>
+ computer.setValue({
+ ...target,
+ elementToken,
+ value: expectedValue,
+ signal: nodeRepl.signal,
+ }),
+ verify: () =>
+ computer.verifyState({
+ ...target,
+ expect: [
+ {
+ element: {
+ selector: { role: expectedRole, label_contains: expectedLabel },
+ value_equals: expectedValue,
+ },
+ },
+ ],
+ stableSamples: 2,
+ signal: nodeRepl.signal,
+ }),
+ });
+ nodeRepl.write(JSON.stringify(lastCuaOutcome));
+ } catch (error) {
+ nodeRepl.write(JSON.stringify(error?.details ?? { message: String(error) }));
+ throw error;
+ }
```
- After one or more actions, always observe the exact window before deciding what
- to do next. If the updated state shows the requested result, stop acting,
- clean up, and answer the user. If the UI does not behave as expected, get a
- fresh full state before choosing a different action.
+ `verifyState.expect` accepts one to eight AND-combined predicates:
- Use a secondary action only when the current accessibility state advertises
- that exact action. Prefer accessibility text for efficiency; use a screenshot
- when it is incomplete or visual layout matters.
+ - `{window:{exists, bounds?}}`
+ - `{element:{selector:{role?, label_contains?}, exists:true?,
+ value_equals?, enabled?, selected?}}`
- ## Reading screenshots
+ Element absence is not provable. `unknown` and `stable:false` are not success.
+ When the postcondition is visual or unsupported, observe the exact window
+ again with a screenshot and inspect the fresh result before deciding. If state
+ is unexpected, observe again rather than repeating the action blindly.
- ```js
- var state = await computer.observeWindow({
- ...target,
- forceFull: true,
- includeScreenshot: true,
- });
+ Read every action result. `effect` is `confirmed`, `partial`, `unverifiable`,
+ `suspected_noop`, or `refused`; `route`, `delivery`, `evidence`, `escalation`,
+ and `operation` explain what actually happened. A committed operation can
+ still have `cancellationRequested:true`, so it still requires verification.
+ Follow an advertised escalation only after fresh state shows it is needed.
- for (var image of state.screenshot?.images ?? []) {
- if (image?.dataBase64 && image?.mimeType) {
- await nodeRepl.emitImage(
- `data:${image.mimeType};base64,${image.dataBase64}`,
- );
- }
- }
- ```
+ ## Interaction details
- ## Finish
+ - After navigation, dialogs, menus, or other surface changes, refresh the
+ relevant window list and observe the new exact surface.
+ - Use returned state to determine text-field behavior; do not assume typing
+ replaces existing text. Use the platform-appropriate select-all action when
+ replacement is required.
+ - Prefer background delivery. Use `deliveryMode:'foreground'` only when the
+ action result or fresh state shows the background route is unavailable or
+ ineffective.
+ - Stop as soon as the requested postcondition is stably satisfied. Do not add
+ extra cleanup actions that could undo the result.
- When the task is complete, close the SDK client:
+ ## Finish
```js
await computer.close();
globalThis.computer = undefined;
+ globalThis.cuaRevisions = undefined;
+ globalThis.observeCuaWindow = undefined;
```
- Call `node_repl_reset` when no other REPL state is needed.
+ Reset the REPL only when no other persistent state is needed.