git:20261009.682c703 to git:20261011.c440b60

73 added, 69 removed. Audit B to B.

---

name: ictrl-phone-control

description: Control an iPhone over HTTP to automate real apps — tap, type, swipe, read the screen's view tree, launch apps, take screenshots. Use when the user asks to operate a phone, drive an iOS app, automate a mobile task (search a product, check an order, fill a form), or inspect what's on a device's screen. The service is iCTRL, JSON-RPC 2.0 on port 8200; requests are encrypted, so drive it with a client downloaded from the device (a static CLI needing no interpreter, or a Python one) rather than hand-written HTTP.

- skill_version: "1.35.0"

+ skill_version: "1.96.0"

---

# iCTRL — control an iPhone over HTTP

- iCTRL runs on the iPhone and serves UI automation as **JSON-RPC 2.0 over HTTP, port 8200**, encrypted — use a client downloaded from the device (below). `GET /schema` (no auth) lists every method and parameter.

+ iCTRL runs on the iPhone and serves UI automation as **JSON-RPC 2.0 over HTTP, port 8200**, encrypted. This file is under 15K characters — **read it whole, once**; don't `head`/`grep` it in pieces.

+ ## Quick start

+ On the phone (e.g. Minis) — the device is `127.0.0.1`:

+ ```bash

+ C=~/.ictrl/bin; I="sh $C/ictrl.sh"

+ [ -f $C/ictrl ] || { mkdir -p $C && wget -qO $C/ictrl http://127.0.0.1:8200/skill/clients/cli && wget -qO $C/ictrl.sh http://127.0.0.1:8200/skill/clients/cli.sh; }

+ $I call device_info # 401 → $I pair (user taps Allow)

+ $I call resolve_app '{"text":"<brand>|<store name>|<alias>"}' # every name it may go by

+ $I call launch_app '{"bundle_id":"<id>"}'

+ $I call page_source '{"visible_only":true,"max_children":50}'

+ $I call find_element '{"element_id":"<label>"}' # → ref

+ $I call tap '{"ref":"<ref>"}'

+ ```

+ - `device_info.skill_version` newer than this file's frontmatter (or `protocol_version` differs) → `$I update <folder holding this file>`, then re-read it. Device older: don't downgrade; use only its `capabilities[]`.

+ - Keep the client in `~/.ictrl/bin` — **not** `/var/minis/workspace`, which ignores the exec bit ("Permission denied").

+ - Elsewhere: `curl -o ictrl_rpc.py http://<host>:8200/skill/clients/python`, then `python3 ictrl_rpc.py --host <host>:8200 call …` (LAN pairing: `pip install cryptography`). The CLI takes `--host`/`ICTRL_HOST`.

+ - Keys: `~/.ictrl/token.json`, one per `host:port`; reuse them (expire after 30 days unused). Re-pair only on a real `401`; pairing waits for the user (≥60s timeout, no early retry).

+ - `/schema` unreachable → the runner isn't running; ask the user to start it.

## Rules — read these first

- 1. **Open apps by bundle id, never by tapping home-screen icons.** `resolve_app` → `launch_app{bundle_id}`. By common sense, pass every name it may go by in **one** call: `resolve_app{text:"Trip.com|Ctrip|Trip"}` (brand, store title, English/Chinese). The store is `country` (default `us`); pick it from context (`cn`: China-only).

+ **IMPORTANT — the rules agents break most, each costing minutes per task:**

+ - **IMPORTANT: no `sleep` in your commands.** Actions already wait for the screen to settle (`settle.stable`) — read right away (see Waiting).

+ - **IMPORTANT: never discard or cut a reply.** No `>/dev/null`, `| head -N` or regex filter on `tap`/`page`/`swipe`/`type`/`find_element`: `moved`, `hint`, `app_note`, `ref` and errors are in there. Check a `ref` is non-empty before tapping it.

+ - **IMPORTANT: read a screen once.** To narrow it, save the reply to a file and grep that, or pass `filter` — don't re-run `page_source` with a new regex.

+ 1. **Open apps by bundle id, never by tapping home-screen icons.** `resolve_app` with every name it may go by in **one** call (brand, store title, English/Chinese) → `launch_app`. `country` picks the store (default `us`; `cn` for China-only apps).

2. **Read before you act.** On any screen you haven't read, call `page_source{visible_only:true, max_children:50}` (~0.4s) to learn which view/tab/date you're really on.

- 3. **Tap by `ref`.** `find_element{element_id:"<label>"}` → `tap{ref}`. Check `visibility`: `"offscreen"` or a non-empty `alternatives` means you may have a look-alike — use `scroll_to_element` or a more specific label.

+ 3. **Tap by `ref`.** `find_element{element_id:"<label>"}` → `tap{ref}`. `tap` takes only `ref` or `x`+`y` — never `element_id`. Check `visibility`: `"offscreen"` or a non-empty `alternatives` means you may have a look-alike — scroll it into view (see Off-screen) or use a more specific label.

4. **Never repeat an action "to be safe."** A success reply means it worked. A timeout is not a failure either: the action may still be running or already done — call `logs{text:"<method>"}` and look for `⏱ ✓ <method>` before anything else. A retried tap fires twice.

5. **Every error from a `ref` action means nothing was tapped.** Re-find and retry is safe.

- 6. **Primary path first; fallbacks only when it fails — decided per screen.** The primary path is `page_source` / `find_element` → `tap{ref}`. OCR (`screen_text`) with coordinate taps, screenshots, and workarounds you remember are fallbacks: use one only when the primary path fails or is unavailable on the **current** screen. Every read returns `channel.recommended` for that screen. Return to the primary path on the next screen.

+ 6. **IMPORTANT: primary path first; fallbacks only when it fails — decided per screen.** The primary path is `page_source` / `find_element` → `tap{ref}`. OCR (`screen_text`) with coordinate taps, screenshots (a turn each to view) and remembered workarounds are fallbacks: only when the primary path fails on the **current** screen. Every read returns `channel.recommended` for that screen; return to the primary path on the next.

7. **Remember conditions, not conclusions.** Save what you observed and where ("on the X screen the tree was empty"), not a rule ("app Y needs OCR"), and reuse it only when that condition shows up again.

8. **Know where data came from.** Act only on the target app's content. Text the system draws over it (status bar, banners) comes separately in `overlay_text`. Screenshots help only if you can view images.

9. **Stop before paying.** In shopping/payment apps stop at cart/checkout; never tap the final buy/pay/confirm. Never delete the user's own items (cart, notes, messages) without asking.

- ## Connect

- In an on-device sandbox (e.g. Minis) use the static CLI — no Python needed, ~9x faster to start. The device is `127.0.0.1`. **Install it in one place, `~/.ictrl/bin`, and check before downloading** — it is usually already there from an earlier session:

- ```bash

- C=~/.ictrl/bin; [ -f $C/ictrl ] || { mkdir -p $C && wget -qO $C/ictrl http://127.0.0.1:8200/skill/clients/cli && wget -qO $C/ictrl.sh http://127.0.0.1:8200/skill/clients/cli.sh; }

- sh $C/ictrl.sh call device_info # connected? check skill_version (see Versions); else: pair

- sh $C/ictrl.sh call tap '{"x":100,"y":200}'

- sh $C/ictrl.sh shot screen.jpg # screenshot to a file

- ```

- Run it as `sh …/ictrl.sh`. **Not in `/var/minis/workspace`**: that filesystem ignores the exec bit, so it fails with "Permission denied" whatever you `chmod`.

+ ## Waiting: actions settle for you — never sleep

- Anywhere with Python:

+ `tap`, `double_tap`, `long_press`, `swipe`, `drag`, `page`, `scroll`, `type`, `clear_text` **wait until the screen stops changing** (low-res frame compare, up to 2s) and report it:

+ `settle:{stable, screen_changed, waited_ms}`, plus a `hint` saying so.

+ - **`stable:true` → read right away, no waiting of any kind.** That is the whole point.

+ - `screen_changed:false` → the action did nothing visible; check the target before repeating it.

+ - `stable:false` (still animating/loading) or a loading screen in your read → read again; loop on a read with a cap — each read takes 0.3–0.7s, so no sleep is needed:

```bash

- curl -o ictrl_rpc.py http://<host>:8200/skill/clients/python

- python3 ictrl_rpc.py --host <host>:8200 pair

- python3 ictrl_rpc.py --host <host>:8200 call device_info

+ for i in 1 2 3 4 5 6; do $I call screen_text '{"element_id":"<new screen text>"}' | grep -q '"count" *: *[1-9]' && break; done

```

- The CLI defaults to `127.0.0.1:8200`; `--host` or `ICTRL_HOST` targets another device. The Python client needs `pip install cryptography` for LAN pairing (not for `--plain` over localhost/USB via `iproxy 8200 8200`).

- Both share `~/.ictrl/token.json`, keyed by `host:port` — **one key per device**, never another device's; Store `device_name` with each key (hosts move; the name says which phone). Reuse the key: it expires only after 30 days *unused*, whatever its `expires` says. Re-pair only on a real `401`. Pairing blocks until the user taps Allow — ≥60s timeout, no early retry (each pops another prompt).

- ## Core workflow

+ - Data that arrives over the network after the screen settled (a spinner, then results) also needs one more read. `page_source` says `loading:true` then.

+ - `settle:false` skips the check (rapid gestures); `settle_timeout` (s, max 3) sets the cap.

- ```

- GET /schema every method + params (once)

- device_info screen size in points, versions, state

- resolve_app{text} → launch_app{bundle_id} then wait 2–3s once (cold render)

- page_source{visible_only:true, max_children:50} where am I?

- find_element{element_id} → tap{ref} act

- type{text, submit:true} enter text

- page{direction:"down"} read further

- ```

+ ## Methods

| Method | Required | Useful optional |

|---|---|---|

| `find_element` | `element_id` (label or accessibility id) | `element_type`, `match:"contains"`, `bundle_id` |

| `scroll_to_element` | `element_id` | `element_type`, `match`, `direction`, `distance` |

| `tap` / `double_tap` / `long_press` | `ref`, or `x`+`y` | `duration` |

| `type` | `text` | `submit`, `clear` |

| `resolve_app` | `text` | — |

| `launch_app` / `activate_app` / `terminate_app` | `bundle_id` | — |

| `open_url` | `url` | — |

- | `page_source` | — | `visible_only`, `max_children`, `filter`, `bundle_id` |

+ | `home` | — | — |

+ | `page_source` | — | `visible_only`, `max_children`, `filter`, `bundle_id`, `enhanced` |

| `page` | — | `direction`, `distance` (screens) |

- | `swipe` | — | `direction`, `distance`, `velocity`, `x`, `y` |

+ | `collect` | — | `pages`, `min_length`, `fresh` |

+ | `swipe` | — | `direction`, `distance` (points), `velocity`, `x`, `y` |

| `screenshot` | — | `scale`, `format`, `quality` |

- The method is `type`, not `type_text`. `element_type` only filters — `element_id` is always required.

+ The method is `type`, not `type_text`. `element_type` only filters — `element_id` is always required. `GET /schema` lists every method and parameter.

Most responses carry a **`hint`** for that exact result — which limits applied, what to do next. Read it, above all when a result is empty or negative.

- ## Timing, timeouts and recovery

+ ## Timeouts and recovery

- - Taps and swipes are synthesized and return **without waiting for the UI to settle**. After a tap that changes screens, the next read can catch a half-drawn transition. If a read looks like the old screen or is sparse, wait ~1s and re-read. Don't sleep blindly after every tap.

- - Typical: `tap` ~0.3s, `page_source` (visible_only) ~0.4s, `find_element` ~1s, `launch_app` ~0.9s + 2–3s render. Much slower usually means a wrong label or a mid-transition screen.

- - The server answers a UI call after **6s** (15s for `page_source`, `tap_element`, `find_and_tap`) even if the work is still running. UI calls run **one at a time**, and a stuck one can't be cancelled. Calls behind it time out too. Their error says *"has NOT run yet: queued behind X"*: they will still run later, so **do not resend them**.

- - After two timeouts in a row, UI calls are rejected immediately. The message names the stuck call and how long it has run. Recovery: `reset` (clears refs and the app binding), then **wait** for the stuck call to end (typically 15–45s; watch `logs`), then retry. `reset` can't stop a call already running.

+ - The server answers a UI call after **6s** (15s for `page_source`, `collect`, `screen_text`, `tap_element`, `find_and_tap`) even if still working. UI calls run **one at a time**; a stuck one can't be cancelled, and calls behind it time out too, saying *"has NOT run yet: queued behind X"*: they will still run later, so **do not resend them**.

+ - After two timeouts in a row, UI calls are rejected immediately. The message names the stuck call and how long it has run. Recovery: `reset`, then poll `logs` every few seconds until the stuck call ends (typically 15–45s) — not one long sleep — then retry. `reset` can't stop a call already running.

- **Only `logs`, `config` and `reset` answer while the UI queue is stuck.** `device_info` shares the UI queue, so it's no health probe; use `logs`.

- Client timeout ≥30s. After restarting your side, reuse the cached key and `bundle_id`s and return via a deep link (`open_url`).

## Finding and tapping

- - `find_element` matches exact label → accessibility id → (with `match:"contains"`) substring. **An element in view always beats an off-screen one**: a calendar holds "25" in three months. The result has `visibility` (`in_view`/`partial`/`offscreen`), and when several matched, `alternatives[]` + `match_count`.

- - A `ref` re-resolves the live element at tap time. If its app has gone to the background (a system sheet, another app), the tap is refused at once: *"belongs to X, which is now in the background"*. Look at the screen before continuing.

- - **Off-screen** → `scroll_to_element{element_id}`. It drags straight to a target already in the tree, else searches (reversing once at an end when no `direction` is given). **"NOT anywhere in this view" means it isn't on this screen at all** — check page/tab/date or whether the data exists; stop scrolling.

+ - `find_element` matches exact label → accessibility id → value → (with `match:"contains"`) substring. **An element in view always beats an off-screen one**: a calendar holds "25" in three months. The result has `visibility` (`in_view`/`partial`/`offscreen`), and when several matched, `alternatives[]` + `match_count`.

+ - A `ref` re-resolves the live element at tap time. If its app has gone to the background (a system sheet, another app), the tap is refused at once: *"belongs to X, which is now in the background"*.

+ - **Off-screen** → loop `page{direction}` → `find_element` until `in_view`; `moved:false` = the end, it isn't there. `scroll_to_element` is slower (~1.3s a step): only for a target `find_element` called `offscreen`. Its **"NOT anywhere in this view"** means it isn't on this screen at all — stop scrolling.

- **Text on screen ≠ accessibility label** (common). Copy the label from `page_source`, or use `match:"contains"`. An accessibility `identifier` shown in `page_source` is the most stable `element_id` of all. Duplicate labels → add `element_type`.

- **Nothing matches but you can see it** → it's a custom control with no label. The failure message lists what *is* on screen. Take its frame from `page_source` and `tap{x,y}`.

- - A coordinate `tap` returns `hit` (the element under the point, sampled before tapping) or `hit_warning` (nothing labelled there — you probably hit empty space).

- - **Web pages don't appear in the accessibility tree.** In Safari or any in-app web view, `page_source` shows only browser chrome (`foreground_is_browser:true` in `device_info`). Use `screenshot` → locate visually → `tap{x,y}`. Native alerts over a page *are* in the tree.

- - A **system alert** covering the app blocks ref/element actions. Read it with `page_source{bundle_id:"com.apple.springboard"}` and tap its button by coordinates.

+ - A coordinate `tap` returns `hit` (the element under the point, sampled before tapping) or `hit_warning` (nothing labelled there — you probably hit empty space). `covering_layer:true`: a dialog/login layer covers the point and got the tap.

+ - **Web pages often don't appear in the accessibility tree** (Safari: chrome only, `foreground_is_browser:true`; some in-app web views expose it). Trust `channel.recommended`; on `ocr` use `screenshot` → locate → `tap{x,y}`.

+ - A **system alert** over the app (`system_alert:true`) blocks its ref actions. Answer it: `find_element{element_id:"<button>", bundle_id:"com.apple.springboard"}` → `tap{ref}`.

+ - **`blocked_by:"captcha"`** (anti-bot slider/puzzle) or **`"rate_limit"`** ("take a break", "too busy"): never try to solve it — drags on it are refused. Use `close_candidates` or back out, wait, or report the step blocked. `promo_popup`: close it, never claim.

## Typing and searching

- 1. The search "bar" on a home screen is often a Button that opens a search page. `find_element` returns `is_text_input`; if false, tap it, then find the real field: `find_element{element_id:"<placeholder>", element_type:"textfield", match:"contains"}` → `tap{ref}` (brings up the keyboard).

+ 1. The search "bar" on a home screen is often a Button that opens a search page. `find_element` returns `is_text_input`; if false, tap it, then find the real field: `find_element{element_id:"<placeholder>", element_type:"textfield", match:"contains"}` → `tap{ref}`.

2. `type{text, submit:true}`. `submit` presses Return/Search.

- 3. **Trust success.** If `type` returned no warning, the text went in. Old content in the next read means the page hasn't updated yet — wait ~1s, re-read, don't re-type.

- 4. **Typing appends.** To replace a field's text use `type{text, clear:true}` or `clear_text` first (select-all + delete). An `append_warning` in the response means you just appended to existing text.

+ 3. **Trust success.** If `type` returned no warning, the text went in. Old content in the next read = not updated yet: read again, don't re-type.

+ 4. **Typing appends.** To replace a field's text use `type{text, clear:true}` or `clear_text` first. An `append_warning` in the response means you just appended to existing text.

If `submit` didn't trigger the search, tap the screen's own search/confirm control — don't re-type.

5. `page_source` carries `keyboard` (`visible`, `covers_from_y`). Coordinates read before the keyboard appeared are stale: everything above it has shifted.

## Reading the screen

- **Default: `page_source{visible_only:true, max_children:50}`** — on-screen nodes only, well under 1s even on long feeds. Omit `visible_only` only when you need off-screen structure.

- `filter:{text|label|type:"…"}` returns matching nodes + ancestors — the cheap way to harvest repeated values (prices, titles).

+ - **`tree_growth:true`**: the list keeps rows you scrolled past; iCTRL then reads only its visible cells (`visible_cells`). Long reads: `collect`; relaunching resets it.

+ - `enhanced:true`: whole tree + text drawn in images (`ocr_only`). Slower; only when asked for everything.

- **Thin tree?** When `channel.recommended` is `ocr` (or `sparse_hint`/`viewport_warning` appears), read with `screen_text`: text plus tappable point coordinates.

- - `max_depth` small (3–4) returns structure without labels — layout preview only.

- - `max_elements` (default 2000, cap 4000): if the response says `truncated:true`, narrow with `filter`/`visible_only` instead of raising it. An app scrolled for a long time accumulates rows and slows every read; relaunching it clears them.

+ - `max_elements` (default 2000, cap 4000): if the response says `truncated:true`, narrow with `filter`/`visible_only` instead of raising it.

## Scrolling

- - **`page{direction:"down"}`** is how to read long content. Each step moves 75% of the screen (25% overlap) at a speed with no momentum, so nothing falls between reads. Don't use `swipe` for reading — it barely moves; keep it for carousels, partial drags and pull-to-refresh.

- - `page`/`swipe`/`scroll` report `moved`. **`moved:false` (`at_end:true`) means stop** — nothing more that way. (A page still loading also looks unmoved; wait ~1s if more is expected.)

- - `position` (`percent`, `fraction`, …) says where you are. Trust it only when `confidence` is `estimated` or `exact`. Lists that build rows lazily pin it near 0/100% with few `rows_seen`.

+ - **Long lists/feeds: `collect{pages:N}`** (≤8) reads, pages, reads… in ONE call (~1.5s a screen), returning only new text; call again to continue. It switches to OCR by itself when the tree is thin, slow or stops answering. `page` moves ⅔ of a screen, no momentum. Don't read with `swipe`.

+ - `page`/`scroll` report `moved`. **`moved:false` (`at_end:true`) means stop** — nothing more that way. (A page still loading also looks unmoved; read once more if more is expected.) `keyboard_closed:true`: only the keyboard closed; page again.

+ - `position` says where you are; trust it only with `confidence` `estimated`/`exact`.

## Apps

- `resolve_app` looks offline first (`opened_here` = launched on this device before — use it directly; then a built-in catalog), else the App Store. Only `opened_here` says anything about *this* device.

- - **`launch_app` is the real install test**: success = installed, error = not. Newer iOS refuses `list_apps` enumeration; it then returns only apps opened via iCTRL before (`source:"opened_history"`). Never report "not installed" from a lookup or an empty list — if unsure, check Spotlight (home, swipe down, type the name).

+ - **`launch_app` is the real install test**: success = installed, error = not. Newer iOS refuses `list_apps` enumeration; it then returns only apps opened via iCTRL before (`source:"opened_history"`). Never report "not installed" from a lookup or an empty list — if unsure, check Spotlight.

- `open_url{url}` opens any scheme / deep link / universal link. Finish by opening the caller's scheme to hand control back, if you have one.

- - After `launch_app`/`activate_app` that app is the **bound target**. Coordinate and `element_id` actions bring it back to the front if it drifted (`app_note` says so). `ref` actions don't — they refuse instead. If the bound app leaves the screen, reads follow whatever is in front. Pass `bundle_id` to read a specific app.

+ - **No target app**: reads and actions apply to what is on screen. Apps jump (login, payment, ads): after a surprise, read first. A `ref` tap is refused once its app leaves the front.

+ - Coordinate actions and `type` return `page_check`; **`same_page:false`** = the screen had changed, before the action, since you last saw it: read, check what you hit, fix.

## Coordinates and screenshots

- `tap`/`swipe`/`drag` take **points**. `page_source` frames and OCR boxes are points.

- - Screenshots are pixels: `point = pixel / scale`, using the **`scale` from that screenshot response** (default 2), not `device_info.screen_scale` (the panel's 3). Mixing them puts taps ~200pt off.

- - Screenshots are JPEG by default; `format:"png"` only when lossless matters.

+ - Screenshots are pixels: `point = pixel / scale`, using the **`scale` from that screenshot response** (default 2), not `device_info.screen_scale` (the panel's 3).

## Diagnostics

- - `device_info.system`: `screen_locked` (black screenshots, taps doing nothing — `unlock` only presses Home; a passcode needs the user), `foreground_app`, `foreground_is_browser`, thermal/memory/battery. Screen sleep doesn't stop the runner; for long runs suggest Auto-Lock → Never.

- - `logs{text?, max_elements?}`: recent log lines over the normal channel. `⏱ ✓ <method>` = completed, `⏱ TIMEOUT`, `⛔️`/`✅` channel state. `GET /logs/files` and `GET /logs/history?date=YYYY-MM-DD` keep logs across crashes; byte endpoints (`/logs*`, `/stream`) need a single-use `?ticket=` from the `stream_ticket` RPC.

- - `reset`: clears refs and the binding and rebuilds the UI channel state, then `launch_app` the target. `config{stop_server:true}` stops the runner (keys persist); it must be relaunched on the device.

- - If `/schema` is unreachable, the runner isn't running. The user must start it on the phone, and the Developer Disk Image must be mounted (it is lost on every reboot).

- - On cellular the device isn't reachable from the LAN (on-device clients still work).

- ## Versions

- On your first `device_info`: if its `skill_version` is **newer** than this file's frontmatter (or `protocol_version` differs), update before anything else: `sh ~/.ictrl/bin/ictrl.sh update <folder holding this SKILL.md>` (Python: re-download `/skill` and `/skill/clients/python`), then re-read this file. Device *older* than this file: don't downgrade; use only its `capabilities[]`.

+ - `device_info.system`: `screen_locked` (black screenshots, taps doing nothing — `unlock` only wakes the screen and never leaves the app: use `home` for that; a passcode needs the user), `foreground_app`, `foreground_is_browser`, thermal/memory/battery.

+ - `logs{text?, max_elements?}`: recent log lines over the normal channel. `⏱ ✓ <method>` = completed, `⏱ TIMEOUT`, `⛔️`/`✅` channel state. `GET /logs/history?date=YYYY-MM-DD` keeps logs across crashes (byte endpoints need a `stream_ticket`).

+ - `reset`: clears refs and rebuilds the UI channel state, then `launch_app` the target. `config{stop_server:true}` stops the runner (keys persist); it must be relaunched on the device.

+ - `/schema` unreachable = the runner isn't running; the user must start it (the Developer Disk Image is lost on every reboot).

+ - On cellular the device isn't reachable from the LAN.

## Protocol

- Only if you can't use a bundled client: the Python client (`GET /skill/clients/python`) is a complete, readable reference for pairing (X25519) and the v1 request envelope (HMAC-SHA256). Follow it exactly.

+ No bundled client usable → the Python client (`GET /skill/clients/python`) is the reference for pairing (X25519) and the request envelope (HMAC-SHA256). Follow it exactly.