git:20260913.2eaed75 to git:20260913.e6647b3

- npm install --no-save --package-lock=false @qwen-code/cua-sdk@0.20.5

- Import the `ComputerUse` API directly once per fresh `node_repl` session:

- ```

-

- ## API surface

-

- ```ts

- type WindowTarget = { pid: number; windowId: number };

- type ElementTarget = {

- pid: number;

- windowId?: number;

- elementToken: string;

- };

- type CoordinateTarget = WindowTarget & { x: number; y: number };

- type PointOrElementTarget = CoordinateTarget | ElementTarget;

- type ExactActionTarget = WindowTarget | ElementTarget;

- type DeliveryOptions = { deliveryMode?: DeliveryMode };

-

- type ComputerUse = {

- click: (

- args: PointOrElementTarget &

- DeliveryOptions & { button?: MouseButton; count?: number },

- ) => Promise<object>;

- doubleClick: (

- args: PointOrElementTarget & DeliveryOptions,

- ) => Promise<object>;

- rightClick: (

- args: PointOrElementTarget & DeliveryOptions & { modifier?: string[] },

- ) => Promise<object>;

- drag: (

- args: WindowTarget &

- DeliveryOptions & {

- fromX: number;

- fromY: number;

- toX: number;

- toY: number;

- },

- ) => Promise<object>;

- observeWindow: (

- args: WindowTarget & { disableDiff?: boolean; includeScreenshot?: boolean },

- ) => Promise<WindowObservation>;

- listApps: () => Promise<Array<App>>;

- listWindows: (args: {

- pid: number;

- onScreenOnly?: boolean;

- }) => Promise<Array<Window>>;

- performSecondaryAction: (

- args: ElementTarget & { action: string },

- ) => Promise<object>;

- pressKey: (

- args: ExactActionTarget &

- DeliveryOptions & { key: string; modifiers?: string[] },

- ) => Promise<object>;

- hotkey: (

- args: ExactActionTarget & DeliveryOptions & { keys: string[] },

- ) => Promise<object>;

- scroll: (

- args: PointOrElementTarget &

- DeliveryOptions & { direction: Direction; amount?: number },

- ) => Promise<object>;

- setValue: (args: ElementTarget & { value: string }) => Promise<object>;

- typeText: (

- args: ExactActionTarget & DeliveryOptions & { text: string },

- ) => Promise<object>;

- close: () => Promise<void>;

- };

-

- type App = {

- name?: string;

- bundle_id?: string;

- pid?: number;

- running?: boolean;

- launch_path?: string;

- };

-

- type Window = {

- window_id: number;

- title?: string;

- is_on_screen?: boolean;

- on_current_space?: boolean;

- };

-

- type WindowObservation = {

- pid: number;

- windowId: number;

- mode: 'full' | 'diff' | 'no_change';

- resyncReason?: string;

- text: string;

- elements: Array<Element>;

- screenshot?: Screenshot;

- };

-

- type Element = {

- element_token?: string;

- role?: string;

- label?: string;

- automation_id?: string;

- value?: unknown;

- actions?: string[];

- };

-

- type Screenshot = {

- images: Array<{ mimeType: string; dataBase64: string }>;

- };

-

- type Direction = 'up' | 'down' | 'left' | 'right';

- type MouseButton = 'left' | 'right' | 'middle';

- type DeliveryMode = 'background' | 'foreground';

- ```

-

- ## Workflow

-

- ### 1. Initialize

-

- Start by getting the state for the app and window you want to use. When the task

- names an app, filter for that name directly:

-

- The snippets below show individual steps. When the app and visible window

- unambiguously match the task, combine discovery, selection, and observation in

- one `node_repl` call. Print candidate lists for selection when they are ambiguous.

-

- ```js

- var apps = await computer.listApps();

- var matches = apps.filter(

- (app) => app.name === 'Target App' || app.bundle_id === 'com.example.target',

- nodeRepl.write(JSON.stringify(matches));

- Use the matching app's observed PID as `targetPid`, then list its visible windows:

-

- ```js

- var windows = await computer.listWindows({

- pid: targetPid,

- onScreenOnly: true,

- });

- nodeRepl.write(JSON.stringify(windows));

- ```

-

- Select the intended document or dialog from the returned titles and window IDs;

- the first entry is not necessarily the task window. Set `target` to

- `{ pid: targetPid, windowId: selectedWindowId }` using that observed ID, then read

- its state:

-

- ```js

- var state = await computer.observeWindow(target);

- nodeRepl.write(state.text); // This will return the accessibility tree

- ```

-

- If you cannot identify an app from the task, prior context, or builtin apps,

- start by discovering the available apps:

-

- ```js

- var apps = await computer.listApps();

- nodeRepl.write(JSON.stringify(apps));

- ```

-

- After performing one or more UI actions, call `observeWindow(...)` before

- deciding what to do next. This keeps you in the current UI state. Read the

- current actionable `element_token` values from `state.elements`; tokens for

- unchanged elements remain valid across `diff` and `no_change` observations.

-

- For token efficiency, when appropriate, the accessibility tree will be returned

- as a diff from the most previous accessibility tree, listing only the elements

- that were removed, added, or changed. Prefer this default diff output; pass

- `disableDiff: true` only when you need a fresh full accessibility tree. If you

- disregard the text from a previous call to `observeWindow`, such as when you only

- emit the screenshot, get the full tree next time you inspect AX text.

- `state.elements` remains the current full actionable element list when

- `state.text` is a diff or reports no change.

-

- ### 2. Actions using app

-

- Choose the delivery mode and current window before batching actions:

-

- - On macOS, app menu items are not owned by a document window, and process-wide background key events cannot reliably target one of several windows. For an interactive task that permits foreground use, pass `deliveryMode: 'foreground'` for app menu clicks and for window-targeted keyboard input when the app has other windows. If the task must stay in the background, use a supported exact element action instead.

- - End the input batch when opening a dialog, sheet, or menu. For a new dialog or sheet, call `listWindows({ pid, onScreenOnly: true })` and observe the matching returned window ID before typing, including a file panel's Go To Folder dialog. Do not guess an ID if no matching window is returned. For ordinary menus that remain in the current window, refresh that window's observation and use its current menu tokens.

- - A parent window's tree may include an attached sheet. If the sheet has a separately discovered window ID, observe that window and use tokens from that observation for actions in the sheet.

- - An AX action can return an error after the app has already opened or closed a window. Check the current windows and state before retrying; do not assume the error means nothing happened or repeat the old token immediately.

-

- Batch actions whose target remains the same, then fetch the latest state:

- await computer.click({ pid: target.pid, elementToken });

- await computer.setValue({ pid: target.pid, elementToken, value: 'openai.com' });

- await computer.pressKey({ ...target, key: 'Enter' });

- await computer.typeText({ ...target, text: 'hello' });

- await computer.scroll({

- pid: target.pid,

- elementToken,

- direction: 'down',

- amount: 1,

- });

- await computer.performSecondaryAction({

- pid: target.pid,

- elementToken,

- action: 'Show Menu',

- });

- nodeRepl.write((await computer.observeWindow(target)).text);

- Notes:

-

- - Prefer `element_token`-based actions over coordinate actions. If AX actions or AX text are unavailable or behave unexpectedly, switch to screenshots, coordinate clicks, and key presses.

- - For window-targeted coordinate actions, use pixels in the PNG returned by `observeWindow` for that exact window, with its top-left corner as `(0, 0)`. AX `frame` values are screen-space logical points, not PNG pixels; do not pass an AX frame's center directly as `x`/`y`. Use a current element token or locate the control in the screenshot. The driver already accounts for Retina scale and image downscaling.

- - `doubleClick` and `rightClick` invoke their dedicated SDK actions; `rightClick` also accepts optional modifiers.

- - On macOS, `drag` requires `deliveryMode: 'foreground'`; the default background mode is unsupported. Pass it explicitly with the exact window target and drag coordinates.

- - If the UI is not behaving as expected, try fetching the latest `observeWindow(...)` state to make sure you have the latest context.

- - Prefer using accessibility text over screenshots for efficiency, but if the interface is not fully working or not providing enough context, make sure to fetch a screenshot to get more context. The accessibility interface may be incomplete in some applications, so a screenshot helps fully understand what is going on.

- - `performSecondaryAction` invokes an accessibility action that an element exposes besides a normal click, such as expanding a disclosure row, showing a menu, incrementing a control, or cancelling something. It requires an action actually exposed for that element in the accessibility text. Do not guess action names.

- - `pressKey` presses one key and accepts optional modifiers; `hotkey` sends a key combination such as `{ keys: ['ctrl', 'c'] }`. Single-key examples include `"a"`, `"Enter"`, `"Tab"`, and `"Up"`.

- - Take care when passing strings containing `\n` or `\r` to `typeText`, as it simulates pressing the return key. Many apps with message composers or forms will respond by sending the message or submitting the form rather than inserting a newline.

-

- ## Reading screenshots

-

- Request a screenshot with the observation, read its accessibility text, and

- emit each returned image:

- `includeScreenshot: true` is the parameter that requests a screenshot.

- ```js

- var state = await computer.observeWindow({

- ...target,

- includeScreenshot: true,

- });

- nodeRepl.write(state.text);

- for (const image of state.screenshot?.images ?? []) {

- await nodeRepl.emitImage(`data:${image.mimeType};base64,${image.dataBase64}`);

- }

- ```

- When all Computer Use work is complete:

- ```js

- await computer.close();

- globalThis.computer = undefined;

- ```

- Reset the Node REPL only when no other persistent state is needed.