computer · diff
git:20261005.efe72ae to git:20261012.3afd440
34 added, 0 removed. Audit A to A.
---
name: computer
description: Read or operate browser and desktop UI when a task needs interaction, login state, visual inspection, or content that web search/fetch cannot obtain.
---
# Computer
Prefer `web_search` / `web_fetch` for information that can be retrieved without operating UI.
Use `computer` when the task needs browser interaction, an existing login or tab, desktop controls,
visual inspection, or when search/fetch fails or returns insufficient content. If UI interaction is
clearly required, start here without a redundant search/fetch attempt. Prefer a suitable dedicated
skill, connector, API, or CLI when available. Respect an explicit user request to use the browser.
Use the `computer` tool for browser and desktop work. It runs JavaScript inside the existing Computer
container, using its logged-in Chromium profile. `browser`, `agent`, `sky`, and `nodeRepl` are ready.
If the tool is not visible, call `load_tools` with `{"plugin":"computer"}`. Its exact tool name is
`computer`.
Bindings persist within this Akashic Session while Computer is awake; a timeout, cancellation, reset,
idle sleep, or workload restart invalidates them. First use wakes the saved personal profile.
An active Turn prevents sleep. After the Turn and identity refresh finish, 10 minutes without
actual input releases the desktop. A panel left open does not keep it alive.
After a script error, bindings and pages remain if input release was confirmed. Read the error and
API, inspect the current page, and reuse the existing browser. Acquire fresh bindings only when the
receipt says they were reset or the browser was closed.
Earlier actions may have taken effect: inspect the current page before retrying an action.
Correct an invalid method and continue; do not treat a script error as task completion.
+ Keep each call focused: observe, choose a small action, then check its visible effect. Once one
+ reliable UI signal confirms the requested result, finish. A second view is useful only when the
+ first is unclear or contradicted. Do not keep inspecting a completed task, look up its implementation,
+ or repeat it to improve a displayed score. A highlighted row alone does not prove its checkbox or
+ icon changed; inspect the specific control's selected/checked state.
+
Before the first browser action, read the installed API in a separate call:
```js
nodeRepl.write(await browser.documentation());
```
Use only methods shown by that API. Create a tab with `await browser.tabs.new()`;
`browser.tabs.create()` does not exist. For example:
```js
var tab = await browser.tabs.new();
await tab.goto("about:blank");
await tab.ax.write();
```
To inspect an existing tab:
```js
var tabs = await browser.tabs.list();
nodeRepl.write(tabs);
var tab = await browser.tabs.get(tabs[0].id);
await tab.ax.write();
```
Read additional documents
with `agent.documentation.get(name)` using names listed in that guidance. The bundled API is the reference Browser API; optional capabilities must be
listed before use. macOS application AX and optional desktop audio are unavailable here.
+ This is a bounded Browser API, not the full Playwright library. Locators do not have `inputValue()`
+ or `boundingBox()`. Read a visible input's current value or an element's rectangle through the
+ documented read-only `evaluate`, for example:
+
+ ```js
+ var field = await tab.playwright.getByRole("textbox", {name: "Name"}).evaluate(el => ({
+ value: el.value, readOnly: el.readOnly, rect: el.getBoundingClientRect()
+ }));
+ nodeRepl.write(field);
+ ```
+
+ Keep page evaluation to short DOM reads. Do calculations and timing loops in the outer REPL;
+ the read-only page scope does not expose every browser global. Use `Number(text)` for numeric
+ conversion. Do not discover APIs by reflection, source reads or guessing many method/key names.
+ Use `var` for bindings reused across calls; do not redeclare top-level `const` or `let` names.
+ Output only what is needed with `nodeRepl.write` or `nodeRepl.emitImage`. Assignments and returned
+ values do not print automatically. Keep screenshot bytes in a variable and emit the image;
+ never write a byte array as JSON. Methods such as `tab.ax.write()` already emit their output.
+
1. Inspect the current page before acting. Prefer `tab.ax` element indices or `tab.playwright` locators.
Derive indices from the latest AX state; derive selectors from observed page structure.
2. Use `fill()` for typed controls such as dates and range sliders, and `selectOption()` for native
selects. Do not type an ISO date into Chromium's segmented date editor.
+ If an input is read-only, open its picker and use the visible controls instead of retrying fill.
3. Verify the resulting UI state before claiming success; partial content does not prove completeness.
Batch related actions and their resulting observation in one call. The driver waits after input;
use page state or locator waits when loading is asynchronous.
+ If the UI confirms there are no matching items to change, complete the remaining requested
+ confirmation or submission; do not invent a target or search for hidden state.
4. Use `sky` for native dialogs, browser chrome, and coordinate actions. Coordinates refer to the
container's 1280 × 800 desktop. It supports click, move, drag, scroll, press_key, type_text and screenshot.
5. New tabs belong to this Turn and close at its end unless marked deliverable or handoff. Preserve
useful output with `await tab.markDeliverable()`. Never close a human tab without explicitly claiming it.
6. Keep `drag_handle()` inside one call and use `try/finally` with `end()`. The driver releases remaining
input when the call ends. A cancelled action may already have changed the page; observe before retrying.
7. Do not launch Chromium yourself, alter profile files, or bypass the Workload owner. Use the Computer
panel for human login and takeover. File chooser paths refer to the container, not the Akashic host.
```js
await tab.playwright.getByRole("textbox", {name: "Name"}).fill("花月");
await tab.playwright.locator('input[type="date"]').fill("2026-09-05");
await tab.playwright.getByRole("combobox").selectOption({label: "China"});
await tab.ax.write();
```
```js
await sky.click({x: 640, y: 400, mouse_button: "right"});
await nodeRepl.emitImage((await sky.get_screenshot())[0].bytes);
```
Images are saved by the existing screenshot owner; inspect the returned path with `read_file`.
It supplies image content to the current multimodal model. Use `read_image_vision` only if `read_file`
reports that the current model cannot accept images.
`sky.get_screenshot()` returns bytes and a data URL, without a separate temporary filepath. AX state is
currently full text; its formatting and compression are not claimed identical to the original WASM.
+ The returned screenshot path belongs to Akashic, not the Computer container's filesystem. Pass it
+ unchanged to `read_file`; do not open it with `require('fs')` inside Computer. For a small visual
+ target, emit a clipped screenshot with `tab.screenshot({clip:{x,y,width,height}})` and inspect it
+ before changing browser zoom or probing image libraries. Browser coordinates are relative to the
+ tab viewport; `sky` uses desktop coordinates, including browser chrome. Refresh geometry after
+ scrolling and bring a target into view before a coordinate action.
OpenCLI remains a separate ordinary shell command and uses the same browser.
For anonymous E2E, use the managed extension `await agent.browsers.create()`. It returns the same
Browser API with an empty, isolated Context in a separate shared headless Chromium. It never copies
the personal profile, login state, or OpenCLI extension. `browser` still refers to the main browser.
At most eight anonymous browsers can be open. Close them with `await testBrowser.close()`;
ending the Turn, an interrupted call, Session reset, or sleep also closes them. Ordinary script errors
keep them when cleanup was confirmed. Another Turn cannot use them. Each instance has a stable
`browserId`: `await agent.browsers.get(testBrowser.browserId)` returns that same browser; use
`await agent.browsers.list()` to find live instances before creating another. Reuse the browser and tab
across tool calls instead of restarting the E2E for every step. An instance idle for 10 minutes is
released; running operations and explicit human takeover prevent idle release. Watching does not.
A closed or expired ID fails explicitly and never selects the main browser.
The last Context releases the headless process and its temporary data. A process crash affects all
anonymous Contexts; close them and acquire fresh bindings. Saved screenshots keep their normal owner.
```js
var testBrowser = await agent.browsers.create();
var testTab = await testBrowser.tabs.new();
await testTab.goto("http://localhost:3000");
await testTab.ax.write();
await testBrowser.close();
```