phone-computer · diff
git:20261115.2ea8b65 to git:20261115.2fb8894
10 added, 4 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: phone-computer
description: Use a phone (a cloud Android or an iPhone) through computer_observe, computer_act, computer_screenshot and a codemode script's `computer`. It covers reading the screen's elements, acting on them by ref, running known steps as one script, checking what happened, recovering, and what a phone cannot do. Use when a run has a phone and the task needs its screen or apps.
---
# Using a phone
The run has a phone, a real one, rented while the run lives. Where it lists its elements (the `ui` group, an Android's
today), you read the screen as lines of elements and act on them by their refs. Where it doesn't (an iPhone today), you
see it through screenshots and act with one finger at their pixels. The tools say which one you have.
## The tools
**`computer_observe` `{app?, image?, full?}`** shows the app in front as compact lines, one per element:
```
state s2 · com.android.settings "Settings" · window 360×800 pt
@e1 generic
@e3 searchbox "Search settings"
@e5 button "Network & internet"
@e9 button "About phone"
```
Each line is a ref (`@e5`), a role, a name, a value after `=`, what else the element says in parentheses (a row's
second line: `("Mobile, Wi-Fi, hotspot")`) and flags in brackets. A child is indented under its parent. The head line names the **state**. `image: true` adds a screenshot, for what the elements can't tell you:
layout, images, colours. Without the `ui` group, observe is the screenshot.
**`computer_act` `{steps, state?, expect?, observe?}`** runs steps in order:
| Step | What it does |
|---|---|
| `{press: "@e9"}` | The element's own tap. The surest step. |
| `{set: "@e3", value: "wifi"}` | Sets a field's text, or a switch's `"on"` or `"off"`. |
| `{click: "@e9"}` or `{click: [x, y]}` | A tap; `button: "right"` is a long press. |
| `{type: "text", into?: "@e3"}` | Types, into the element when named, else into the field in focus. |
| `{nav: "back"}`, `{nav: "home"}`, `{nav: "app_switch"}` | The system's own navigation. |
- | `{scroll: "@e1", dy: 600}` | Scrolls the element by points: `dy` above 0 shows what is below. |
+ | `{scroll: [180, 420], dy: 600}`, with `foreground: true` on the call | Swipes from that point: `dy` above 0 shows what is below. A swipe is the phone's own input, refused unless the call passes `foreground: true`. |
| `{drag: [[x1, y1], [x2, y2]], ms?}` | One finger along the path. |
| `{key: "enter"}` | A key, where the phone has keys. |
Each step answers its route and whether a readback verified it; the answer ends with whether the screen changed and its
- new state. `expect: {text: "Android version"}` waits after the steps until that text shows (`gone: true`: until it is
+ new state. `expect: {text: "Device name"}` waits after the steps until that text shows (`gone: true`: until it is
gone). `observe: true` adds the new outline: one call instead of two. The first refused step stops the rest.
Prefer refs. A step at a point is read against a state and refused as **stale** once the screen moved under it:
observe again and use the new state. Without the `ui` group, points are screenshot pixels and there are no refs,
states or expects.
**`computer_screenshot` `{region?}`** is the whole screen, or `region: [x, y, w, h]` of it, in pixels.
**Scripts: `codemode`.** A `codemode` script has the phone as `computer`, and codemode's description gives its types
for this phone. The body is an async function: `await` the calls and `return` what you need. Each call is a call of its
own, asked about and recorded like the tools'.
```js
await computer.apps.open("com.android.settings");
const settings = computer.app("com.android.settings");
await settings.observe();
+ for (let i = 0; i < 4 && !settings.find({ text: "About phone" })[0]; i++) {
+ await settings.scroll([180, 420], { dy: 600 }, { foreground: true });
+ await settings.observe();
+ }
const about = settings.find({ text: "About phone" })[0];
- const page = await settings.press(about.ref, { expect: { text: "Android version" }, observe: true });
- return { met: page.expect, lines: page.after.outline.split("\n").filter((line) => /version/i.test(line)) };
+ const page = await settings.press(about.ref, { expect: { text: "Device name" }, observe: true });
+ return { met: page.expect, lines: page.after.outline.split("\n").filter((line) => /name/i.test(line)) };
```
+
+ An `expect` is met only by what shows on the screen: a row below the fold is not there until you scroll to it.
App ids come from `computer.apps.list()`: call it before `computer.apps.open`. The tools are there too, as
`tools.computer_observe(...)` and `tools.computer_act(...)`.
**On Codex** there is no `codemode` and no `computer`: Codex runs scripts in its own code mode, which calls the tools in
your list, and each call answers JSON text to parse. The phone's apps are `computer_op` there, one call per op with its
arguments, named as `computer`'s methods are: `computer_op({op: "apps.open", args: {id: "com.android.settings"}})`.
**The screen itself.** Prefer an app's steps. A script's `computer.input.act([...])` is the screen-level way out, at
screenshot pixels, for what the app's window can't reach: a swipe from the screen's top edge opens the notification
shade (`{ type: "swipe", path: [[540, 5], [540, 1200]] }`), and anything outside the app.
**Only what you were given.** An agent's computer setting may leave some out: a code-only agent drives the phone from
`codemode` alone, with no `computer_observe`, `computer_act` or `computer_screenshot` in its tool list; an agent whose
exec is off has no shell of any kind. Use what your tool list and codemode's types show.
## How to work: observe, then act by ref, then check
1. **Observe** first. Find your target by its name, role or id, and note its ref. A list row is often an unnamed
button whose title sits on a child text: the outline shows it indented under the row.
2. **Act** by ref. A step that changes the screen (opening a page, a back) ends the call, unless you put an `expect`
on it for what the next screen shows.
3. **Check** the answer's routes, `changed` and the expect. When it doesn't settle it, observe again.
4. **Recover.** `stale`: observe again. Nothing changed: a dialog, the keyboard or a banner may cover the target.
`{nav: "back"}` closes what covers it; `computer.apps.open(id)` in a script starts an app afresh on its first
screen, the surest way back to a known place.
## When to write a script
A `codemode` script is one call of yours where tools would be many.
- **Known steps**: open an app, press through a known path, read one value: one script, an `expect` on each page.
- **A few facts out of a long screen**: `find()` inside the script and return only those, never whole observations.
`find({ role, text, id })`: a role is the same name on every phone, a text matches the name, the value or what the
parentheses say, and an id is the app's own (an Android resource-id).
- **Loops**: scroll a list until a row shows.
When you are unsure what is on the screen, observe before you act, and decide from what it shows.
## What the screen looks like
- **A page's title may not be an element.** On current Android the title is drawn in a header that shrinks as you
scroll. The page opened when its own items show and the list you pressed is gone.
- **The keyboard** covers the lower part of the screen while a field has focus; `{nav: "back"}` closes it.
- **The status bar** across the top (time, battery) is not the app.
## What a phone cannot do here
- No pinch or zoom, and no gesture with more than one finger.
- No shell, files or installs.
- Plain ASCII typing only: other characters are refused in words before anything is typed.
Say in your answer what the task needed that the phone could not do.
## When a tool answers an error
Errors come in words, and they say what was wrong. Read them, act on what they say, and don't repeat the same call.
- **"The person had this computer"**: a person took the phone over and has handed it back, and your act was not done.
The screen may have changed: observe again before you act.
- **A step refused as `unsupported`** names what this phone can't do (an element step without the `ui` group, a key
on a phone with no keys). Use another step.
- **The phone is gone or unreachable**: the run's phone ended. Don't keep calling. Say what you finished and what you
could not.
## Time and cost
The phone may be billed by the minute and lives only for the run's lifetime. Plan first (which app, which screens,
what will show success), script what is certain, and stop once the task is done or can't be done.
## Finishing
Before you call the task done, check the result with an observe (or a screenshot when it is visual). Say what it shows
in your answer, and name anything you could not finish or check.