--- name: using-agent-browser description: "Drives the preinstalled agent-browser Chromium in an orb for browser interaction, UI inspection, screenshots, recordings, and web E2E checks. Use before any agent-browser work: covers localhost dev servers, sessions that survive pause/resume, 2x screenshots, hover and touch emulation limits, and known command pitfalls." metadata: sandbox-only: true --- # Using agent-browser in an orb agent-browser and Chrome for Testing (Chromium) are preinstalled in every orb. Run `agent-browser skills get core --full` first; it is the version-matched reference for commands, refs, sessions, restore, and recording. This skill adds only what the orb environment changes. ## Environment - The browser runs inside the orb. It can open `http://localhost:` dev servers directly. Only the user needs portal URLs, so share the portal URL with the user and never the localhost URL you browsed. - If Chrome is missing, run `agent-browser install`. - For a self-signed HTTPS dev server, pass `--ignore-https-errors` and `--args "--allow-running-insecure-content"` on the first `open`. Prefer an HTTP port when the server offers one. - Pass the identical `--args` value on every command for that session, or wrap the invocation in a script. A changed or missing `--args` value relaunches Chrome without the flags, silently. `--args` splits on every comma, so a comma-valued Chrome flag cannot go through it; use an `--executable-path` wrapper instead (see Hover-gated UI). - Save every screenshot and recording the user should see under `.amp/in/artifacts/` by absolute path. The thread renders media only from the workspace, so files under `/tmp` show as broken placeholders. A path starting with `.` is read as a CSS selector. ## Sessions - Reuse one named `--session` for a whole workflow. Keep at most three live sessions in one namespace; reuse or close one instead of opening a fourth. Keep session names to 20 characters or fewer so the Unix socket path stays under its 103-byte limit. - Close sessions as soon as their part of the workflow ends, and before memory-heavy work such as test suites or builds. - The browser daemon dies on orb pause and resume, on CLI restarts, and after an idle hour. Pass bare `--restore` on every command for a session that must keep login state. Restore keeps cookies and storage only: after a relaunch, `open ` (not bare `reload`) and reapply `set viewport`, `set device`, and `set media`. - `[agent-browser] restore: missing; save: saved` is startup metadata, not a failure. - A shell helper that creates sessions installs an `EXIT` trap as soon as it knows the session names; a Node or TypeScript helper closes them in `finally`. Call cleanup explicitly on the success path too; a hard command timeout can skip the trap, so run cleanup in the next command after a timeout. ```bash export AGENT_BROWSER_NAMESPACE=checkout-flow SESSIONS=(buyer admin) cleanup() { for session in "${SESSIONS[@]}"; do agent-browser --session "$session" --restore close >/dev/null 2>&1 || true done } trap cleanup EXIT agent-browser --session buyer --restore open "$BUYER_URL" agent-browser --session admin --restore open "$ADMIN_URL" for session in "${SESSIONS[@]}"; do agent-browser --session "$session" --restore set viewport 1280 720 2 done # workflow cleanup trap - EXIT ``` ## Screenshots - Screenshots must be 2x (`devicePixelRatio` 2); 1x looks blurry on retina displays. Run `set viewport 2` after the first `open` of a new or restarted session and after any `set viewport` that omits the scale, which resets that tab to 1x. Check with `eval devicePixelRatio` before capturing. Image pixel coordinates are then 2x the CSS values; selector clips are unaffected. - Set screenshot scale with `set viewport`, not Chromium's `--force-device-scale-factor` launch flag. Combining the launch flag with viewport DPR produces incorrectly sized full-page captures. - `set viewport`, `set device`, and init scripts apply to the current tab only. A popup opened by page code (`window.open`, `target=_blank`) gets neither; `tab new ` and `click --new-tab` replay init scripts but not the viewport. Open new pages with `tab new ` instead of triggering a popup, and after `tab ` onto any new tab check `eval devicePixelRatio` and rerun `set viewport 2` if it is not 2. - Wait for the element, text, URL, or app-specific ready signal before every capture. After a viewport change, wait for two animation frames: `eval "new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r)))"`. - Pass a selector or ref as the first `screenshot` argument only when the element sits inside the initial unscrolled viewport. Selector captures clip against the unscrolled surface, so below-the- fold or scrolled elements come out blank white and `scrollintoview` does not fix it. For those, scroll and capture the viewport without a selector, or capture `--full` and crop. - Capture every representative affected state, not only the default one: open, selected, empty, loading, and error states when the change touches them. - Inspect every capture with the media viewing tool before reporting it. For exact colors or measurements, sample pixels with Python and Pillow instead of judging by eye. - Use `set viewport 2` at several widths for breakpoint checks. A 390-pixel Chromium capture proves the narrow layout only; it is not a phone capture (see Touch and mobile). ## Hover-gated UI Headless Chromium reports no input device: `(hover: none)` matches and `(pointer: fine)` does not. Hover-gated UI such as tooltips never appears on `hover` or `mouse move`, and touch fallbacks render in screenshots. To test hover, relaunch through a wrapper that adds the Blink settings: ```bash cat > /tmp/chrome-desktop-pointer.sh <<'EOF' #!/usr/bin/env bash set -euo pipefail chrome=$(ls -d "$HOME"/.agent-browser/browsers/chrome-*/chrome | sort -V | tail -n 1) exec "$chrome" \ '--blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4' \ "$@" EOF chmod +x /tmp/chrome-desktop-pointer.sh agent-browser --session app close agent-browser --session app --executable-path /tmp/chrome-desktop-pointer.sh open "" ``` Verify `eval "matchMedia('(hover: hover)').matches"` is `true` before trusting hover results. A relaunch without the wrapper reverts. ## Interaction recordings Record a clip when a change affects how UI responds to the pointer or keyboard: hover states, menus, popovers, inline forms, focus, or transitions. A clip shows what a person sees while moving through the UI; screenshots of the end states do not. - Launch through the hover wrapper above first, or the clip shows touch fallbacks. - Start with `record start --cursor`. The cursor overlay follows the real pointer and shows click feedback. Recordings use the tab's device pixel ratio, so a 2x viewport records at 2x. - Move with real input: `mouse move --human` and `click --human`. Real moves fire hover events on every element along the path. `hover ` jumps straight to the target and skips everything between. - Tour the area the way a person would. Pass over neighboring rows and similar controls before the target, click, wait for the result to appear, then keep moving inside the new UI so its hover states show too. Hold about one second after each move, since many hover effects appear after a delay. - Type with `click --human` and then `keyboard type ""`, so each keystroke shows. Never type secrets into a recording. - Cut idle stretches between commands before sharing, and keep clips under about 30 seconds. - The transcript plays a video once. For a clip that loops, also embed an animated WebP: `ffmpeg -i clip.mp4 -vf fps=25 -c:v libwebp_anim -quality 75 -loop 0 -an clip.loop.webp`. - Check the clip frame by frame. Video analysis samples about one frame per second and misses short hover states, so extract frames and view them as a contact sheet: `ffmpeg -i clip.mp4 -vf fps=5,scale=640:-1 /tmp/f%03d.png` and `magick montage /tmp/f*.png -tile 4x -geometry +4+4 sheet.png`. ## Touch and mobile agent-browser is Chromium-only and never emulates touch. `set viewport` and `set device` change size, DPR, and user agent only, so `(pointer: coarse)` stays unmatched and `(hover: none)` matches. Touch-only styles change sizes, spacing, and layout, so never present a `set device` screenshot as what the UI looks like on a phone. For mobile screenshots, touch verification, and Safari-only rendering bugs, use Playwright device profiles from a scratch directory outside the workspace: ```bash mkdir -p /tmp/wk && cd /tmp/wk && npm i playwright >/dev/null npx playwright install webkit --with-deps # iOS Safari approximation ``` ```js const { webkit, chromium, devices } = require('playwright') const browser = await webkit.launch() const context = await browser.newContext({ ...devices['iPhone 14'], ignoreHTTPSErrors: true }) const page = await context.newPage() await page.goto(url) if (!(await page.evaluate(() => matchMedia('(pointer: coarse)').matches))) throw new Error('no touch') await page.screenshot({ path: '/absolute/workspace/.amp/in/artifacts/mobile.png' }) ``` - The full device descriptor (`hasTouch`) is what makes `(pointer: coarse)` match; use `.tap()`. - Android: `chromium.launch()` with a Pixel descriptor covers Android Chrome touch behavior. Orbs have no KVM, so an emulator is not an option; never present this as a real device test. - Linux WebKit approximates macOS Safari; system colors and native selection painting differ. Confirm macOS-specific bugs on real Safari. - Say which engine produced a mobile capture and that `(pointer: coarse)` was verified. ## Fake microphone Chromium's fake media stack turns a WAV file into the microphone signal: ```bash sudo apt-get install -y espeak-ng espeak-ng -s 145 -v en-us -w /tmp/raw.wav "Your dictation text here." ffmpeg -y -i /tmp/raw.wav -af "adelay=700:all=1,apad=pad_dur=1" -ar 48000 -ac 1 /tmp/speech.wav agent-browser --session mic \ --args "--use-fake-device-for-media-stream,--use-fake-ui-for-media-stream,--use-file-for-fake-audio-capture=/tmp/speech.wav,--autoplay-policy=no-user-gesture-required" \ open "" ``` The WAV must be mono 48 kHz `pcm_s16le`; it loops forever. Repeat the identical `--args` on every command for that session. Verify with `eval "navigator.mediaDevices.getUserMedia({audio:true}).then(s=>s.getTracks()[0].label)"`, which returns `Fake Default Audio Input`. ## Troubleshooting - There is no `mouse click` subcommand. Use `click `; for coordinates, use `mouse move `, `mouse down left`, `mouse up left`. - Do not retry a hanging command in a loop. Run `agent-browser doctor --offline --quick`, inspect the session, then close and reopen it. - If `open` reports `CDP command timed out: Page.navigate`, run `get url` once. If it shows the expected destination, continue. - `batch` step strings strip quotes. For quoted JavaScript or selectors, pass a JSON array of argument arrays on stdin: `agent-browser batch <<'JSON'` … `[["open","https://example.com"],["eval","document.title"]]`. - Wrap multi-statement `eval` in an IIFE or block. Top-level `const` and `let` persist across calls in a session and cause `Identifier has already been declared`. - The orb's agent-browser build has no runtime `addinitscript` command. Pass `--init-script ` when launching the browser instead. - If `open` lands on `about:blank` or `snapshot` shows `(no interactive elements)` for a local HTTPS server, relaunch with the HTTPS flags from Environment on the first `open`.