arch-benchmark-browser · diff
git:20260927.f614618 to git:20260928.8202ef9
127 added, 79 removed. Audit A to A.
---
name: arch-benchmark-browser
description: "Run the benchmark prompt in the chatgpt.com, claude.ai, gemini.google.com, and grok.com products, signed in, and save each answer with its conversation URL as proof."
- allowed-tools: Read, Write, Bash(mkdir:*), Bash(date:*), Bash(python3:*), mcp__claude-in-chrome__tabs_context_mcp, mcp__claude-in-chrome__tabs_create_mcp, mcp__claude-in-chrome__tabs_close_mcp, mcp__claude-in-chrome__navigate, mcp__claude-in-chrome__computer, mcp__claude-in-chrome__read_page, mcp__claude-in-chrome__find, mcp__claude-in-chrome__get_page_text, mcp__claude-in-chrome__browser_batch
+ allowed-tools: Read, Write, Bash(mkdir:*), Bash(date:*), Bash(python3:*), Bash(git ls-remote:*), mcp__claude-in-chrome__tabs_context_mcp, mcp__claude-in-chrome__tabs_create_mcp, mcp__claude-in-chrome__tabs_close_mcp, mcp__claude-in-chrome__navigate, mcp__claude-in-chrome__computer, mcp__claude-in-chrome__read_page, mcp__claude-in-chrome__find, mcp__claude-in-chrome__get_page_text, mcp__claude-in-chrome__browser_batch
disable-model-invocation: true
---
# arch-benchmark-browser
Ask four products the same question and keep the answers with their
proof. The prompt, the contract, the size map, and the schema live at
`${CLAUDE_SKILL_DIR}/../../benchmark/browser/` and
`${CLAUDE_SKILL_DIR}/../../benchmark/schema/browser-session.schema.json`.
If any is missing, stop and say the installation is incomplete.
## Input
`$ARGUMENTS` names two sizes, `model=<size> effort=<size>`, each one of
`xs`, `s`, `m`, `l`, `xl`. When one is missing, it is `m`. It may also
name a subset of sites (`sites=claude.ai,gemini.google.com`); the
default is all four. It may ask for a comparison with earlier runs:
`compare=<run_id>,<run_id>` names them, and `compare=all` means every
earlier run. Without `compare=`, nothing is compared.
## What the pages are like
Facts that decide how the steps below go. Read them before the browser.
- The read tools are permitted per domain, and not as a set. On some
domains `get_page_text` and `read_page` are refused while a
screenshot works; on others `find` is refused while `read_page`
works. When a read tool is refused, use a screenshot and `zoom`, and
transcribe from it. Say in the output which site was read that way.
- The composers on chatgpt.com, claude.ai, and grok.com are rich text:
a line that starts with a dash and a space becomes a bullet. On
chatgpt.com and claude.ai the next line gets one too, and backticks
become inline code; grok.com keeps backticks as typed.
gemini.google.com's composer is plain text. Step 5 types the contract
so that no dash becomes a bullet.
- Enter sends on all four composers. A new line inside the message is
`shift+Return`. Type one line, press `shift+Return`, type the next;
a blank line is two `shift+Return`.
- A `browser_batch` has a deadline of its own: at most five ten-second
waits and one screenshot per batch, or it times out. A tool that is
refused once on a domain with "Permission denied for this action" may
succeed on the next call; retry once before treating it as a fact.
- The tab group can hold tabs the skill did not open: an empty New Tab,
or another agent's tabs. Touch only the tabs you create.
- A dialog that promotes a model or a feature, or asks to allow
notifications, is closed by its most privacy-preserving choice ("Not
now", "No thanks"), never by accepting it. The session's `note` says
so.
- - An account can lock a model or a mode: clicking it opens a plan page
- or an upgrade prompt, or the entry does not stay checked. Never click
- Upgrade, or anything that buys.
+ - An account can lock a model, a mode, or an effort: clicking it opens
+ a plan page or an upgrade prompt, or the entry does not stay checked.
+ Never click Upgrade, or anything that buys.
- chatgpt.com: the control at the right of the composer opens a
"Thinking effort" popover with a five-stop slider. The label above
- the slider names the stop the handle is on: Instant, Medium, High,
- Extra High, and 6 Pro, left to right. A chevron beside that label
- opens the model list ("Latest" and older models). After a pick, the
- chip shows the stop's label, not the model.
+ the slider names the stop the handle is on; `sizes.yaml` lists the
+ five labels, left to right. A chevron beside that label opens the
+ model list ("Latest" and older models). After a pick, the chip shows
+ the stop's label, not the model.
- claude.ai: one menu at the right of the composer holds the current
models, then "Effort" with five levels, then "More models" with the
older ones. Picking a model closes the menu and resets the effort to
its default, so the model is picked first and the effort second. The
page may say "Computer actions available": the product can then run
commands on the person's machine, and step 4 does not send there.
- gemini.google.com: click the composer once before opening the chip.
The list is Flash-Lite, Flash, Pro, then, below a separator,
"Extended thinking", which is a toggle layered on the current model
and not a model of its own; the chip then reads "Flash Extended". The
size map never asks for it. The chip shortens "3.1 Pro" to "Pro".
- grok.com: the chip at the right of the composer shows the mode. Its
accessible name is "Model select", the name to give `find`. It opens
one list of modes: Fast, Build, Auto, Expert, Heavy, and a
"SuperGrok" row with an Upgrade button. There is no effort control;
the "+" at the left holds files, projects, skills, and connectors.
Auto, Expert, and Heavy need a SuperGrok plan. On an account without
one, clicking them opens a plan page (the URL ends in `#subscribe`)
and the chip keeps its mode; the × at the page's top right closes it.
Never switch to "Private" at the top right. An answer advances only
while its tab is in front.
- A finished answer is announced in text on three sites. chatgpt.com
writes "Worked for" and a duration above it. claude.ai's page text
contains "Claude finished the response". gemini.google.com shows the
answer with the composer empty and no stop control. On these three,
read that rather than the send button's shape.
- grok.com announces nothing in text. It writes "Worked for" above its
answer while the text is still revealing, so that line is not the
signal. Its answer is done when the page text is the same on two
- polls a minute apart. The button at the
- right of the composer is a check, not the signal. Before sending,
- with the composer empty, it is "Enter voice mode", drawn as a blue
- waveform in a circle. A stop control in its place means the answer
- is still coming.
+ polls in a row, which step 6 spaces at least a minute apart. The
+ button at the right of the composer is a check, not the signal.
+ Before sending, with the composer empty, it is "Enter voice mode",
+ drawn as a blue waveform in a circle. A stop control in its place
+ means the answer is still coming.
- The page text before and after an answer is chrome: the time worked,
the echoed prompt, the speaker label ("ChatGPT said:", "Claude
responded:", "Gemini said"), a tool count, and after the answer the
- chip's label ("Fast") or a hover line ("Add to chat"). The score is
- the line that matches `Score: NN/100`, wherever it is.
- gemini.google.com's page text also holds the sidebar's conversation
- titles. They are the person's: never copy them.
+ chip's label ("Fast") or a hover line ("Add to chat"). The answer
+ starts after the speaker label and ends at its own last line.
+ grok.com writes no speaker label: its answer starts after the echoed
+ prompt, whose last line is the contract's last item, and after the
+ "Worked for" line where one shows. Step 6 and step 7 both mean the
+ answer so bounded. The score is the line that matches
+ `Score: NN/100`, wherever it is. gemini.google.com's page text also
+ holds the sidebar's conversation titles. They are the person's:
+ never copy them.
## Procedure
1. Read `prompt.md`, which holds both the prompt and the contract, and
`sizes.yaml`. The browser README is for people and is not an input.
Resolve, per site, the model and effort labels the sizes ask for.
- Say them before touching the browser; in a run nobody watches, they
- go into the run's `note` in `results.json`.
- 2. Note the run's start with `date -u +%Y-%m-%dT%H:%M:%SZ`; make the
- run folder `~/Downloads/benchmark_browser/<YYYYMMDD-HHMMSS>/` from
- the same moment, in UTC. Every file of the run goes there.
- 3. Call `tabs_context_mcp` with `createIfEmpty`, then `tabs_create_mcp`
- one tab per site, and navigate each to its new-chat URL:
+ Say them before touching the browser, and always put them in the
+ run's `note` in `results.json`.
+ 2. Note the run's start with `date -u +%Y-%m-%dT%H:%M:%SZ`, the form
+ of every time the run records; make the run folder
+ `~/Downloads/benchmark_browser/<YYYYMMDD-HHMMSS>/` from the same
+ moment, in UTC. Every file of the run goes there. Then read
+ the commit the repository's default branch points at, with
+ `git ls-remote <url> HEAD`, where `<url>` is the repository URL in
+ the prompt. The first field it prints is the run's
+ `repository_head`, whatever each answer says it read. When the
+ command fails, it is `unknown` and the run's `note` gives the error.
+ 3. Call `tabs_context_mcp` with `createIfEmpty`; never use or close an
+ empty tab it makes. Then `tabs_create_mcp` one tab per site, and
+ navigate each to its new-chat URL:
`https://chatgpt.com/`, `https://claude.ai/new`,
`https://gemini.google.com/app`, `https://grok.com/`. Take a
screenshot of each. A page that shows a sign-in button, a login
form, or no composer is `not-signed-in`: record it, tell the person
which site to sign in to, and go on with the sites that are. Never
type an email or a password, ever.
- 4. Per site, set the model and the effort, then verify with a
+ 4. Per site, first look, with `find` or a screenshot, for a line that
+ says the product can act on the person's machine, such as
+ claude.ai's "Computer actions available". When one shows, on any
+ site, set nothing and send nothing: the session is `not-run`, and
+ its `note` asks the person to turn that setting off. Never change
+ it yourself.
+
+ Otherwise set the model and the effort, then verify with a
screenshot of the chip. `model_label` is the model picker's checked
entry as the list writes it; `effort_label` is the checked effort
entry or the stop's label. The chip is the check.
- chatgpt.com: open the popover, open the chevron, click the model
the size names; then click the slider stop whose label the size
names.
- - claude.ai: first look for "Computer actions available" with `find`
- or a screenshot. When it shows, set nothing and send nothing: the
- session is `not-run`, and its `note` asks the person to turn
- computer actions off. Never change that setting yourself.
- Otherwise open the menu and click the model, under "More models"
+ - claude.ai: open the menu and click the model, under "More models"
when it is not at the top; open the menu again, open "Effort",
click the level. The chip must read `<model> <effort>`.
- gemini.google.com: click the composer, open the chip, click the
model the size names. The chip must read the model's short name.
- grok.com: open the chip, click the mode the size names. The chip
must read the mode.
Click the size's effort even when the chip already shows it. Where
the size map's effort is `none`, `effort_label` is `none`. If the
label does not match what the size asked for, try once more, then
record the label the page shows and go on: the results carry what
was actually used, never what was asked for.
- A locked model or mode is the exception, on every site. Close what
- opened and pick the next smaller entry that stays checked (Fast on
- grok.com). Record that entry, name the locked one in `note`
- ("Expert needs a SuperGrok plan"), and step 7 records the session
- `smaller-mode`. A locked entry is not tried again.
+ A locked model, mode, or effort is the exception, on every site, and
+ is not tried once more. Close what opened. Then take that site's
+ entries in `sizes.yaml`, in the same map (model or effort), for the
+ smaller sizes, from the next smaller one down to `xs`: skip an entry
+ already tried, and click the first one that stays checked (Fast on
+ grok.com, for Expert or Heavy). Record that entry, name the locked
+ one in `note` ("Expert needs a SuperGrok plan"), and step 7 records
+ the session `smaller-mode`. A locked entry is not tried again in the
+ run. When no smaller size's entry stays checked, send nothing: the
+ session is `not-run`, and `note` names the locked entries.
5. Click the composer and type the prompt from `prompt.md`, a blank
line, and the contract, line by line with `shift+Return` between
lines. Type each of the contract's items without its leading dash:
this is the one place that rule lives, and `contract` records the
text as typed. Send with `Return`. The send time, from `date -u`, is
the session's `started_at`. Take one screenshot showing the sent
message and the chip; it is a check and is not saved.
6. Do step 4 and step 5 for every site first, then poll each site at
most thirty times, no more often than once a minute, and count its
polls into `polls`. A poll is one batch: first a scaled (0.4)
screenshot, which brings the tab to the front, then up to five
ten-second waits. On claude.ai and grok.com the signal is in the
page text: end the batch with one `get_page_text` and keep it to
four waits, so the batch stays inside its deadline. Where
`get_page_text` is refused, on either site, compare the poll's
screenshot with the previous one instead: the same last line of the
answer, and no stop control.
- grok.com: the text is the same when the answer's length and its
own last line match the previous poll's; the chrome after the
answer is ignored. When they match, check the button beside the
composer with `find` or `read_page`, or with a `zoom` where both
are refused. "Enter voice mode" confirms the answer is done. A
stop control means it is not, whatever the text did: keep
polling. A tab that was not in front for its waits is never called
done.
Done is the signal named above. Record the finish time from `date -u`
- when the signal is seen. On grok.com, run `date -u` at every poll:
- the finish time is the first of the two polls that matched. When an
- answer is already done at the first poll, the finish time is the send
- time plus the site's own "Worked for" figure where it shows one,
- except on grok.com, or else the first poll's time, and `note` says
- which. After thirty polls or thirty minutes, record `timed-out` with
- what the page shows so far.
+ when the signal is seen. On grok.com, run `date -u` at the end of
+ every poll: the finish time is the first of the two polls that
+ matched. An error the product prints in place of an answer ends the
+ polling of that attempt, and step 7's `errored` says what follows.
+ When an answer is already done at the first poll, the finish time is
+ the send time plus the site's own "Worked for" figure where it shows
+ one, except on grok.com, or else the first poll's time, and `note`
+ says which. After thirty polls or thirty minutes from the send,
+ record `timed-out` with what the page shows so far.
7. When a site is done, read the conversation URL from
`tabs_context_mcp` and drop its query string. Read the answer with
`get_page_text`, or from screenshots where that is refused. Find the
`Score: NN/100` line, and what the answer's Method says it read: a
commit, a tag, or a branch and a date, for `read_version` ("not
stated" when it names none). Save `<site>.md` in the run folder:
```text
# <site>
- URL: <url>
- Model: <model_label>
- Effort: <effort_label>
- Sent: <started_at>
- Finished: <finished_at> (how it was read)
- Status: <status>
- Score: <NN>/100
- Read: <read_version>
- Polls: <polls>
- Note: <note>
## Answer
<the answer>
```
- The answer starts after the speaker label and ends at its own last
- line. The chrome on either side is left out; tool steps, citation
- chips ("GitHub", "10 sources"), and image captions stay as the page
- gave them. The statuses:
+ Where `score` is null, the header's line is `- Score: none`; where
+ there is no note, it is `- Note: none`. The answer is bounded as
+ "What the pages are like" says, and the chrome on either side is
+ left out; tool steps, citation chips ("GitHub", "10 sources"), and
+ image captions stay as the page gave them. The statuses:
- `ok`: a score was found.
- `smaller-mode`: in place of `ok`, when step 4 fell back from a
- locked model or mode.
+ locked model, mode, or effort.
- `no-score`: an answer and no score line; the number the answer
gives elsewhere goes in `note`.
- `refused`: the product declined.
- `errored`: the product printed its own error in place of an answer
- ("I seem to be encountering an error"). Start a new chat on that
- site, do step 4 again without clicking a locked entry, and send
- once more before recording it.
+ ("I seem to be encountering an error"). Read that attempt's URL
+ from `tabs_context_mcp`, without its query string, and try once
+ more: navigate the site's tab to its new-chat URL, do step 4 again
+ without clicking a locked entry, do step 5, and poll as step 6
+ says. Every field of the session is the second attempt's, and its
+ `polls` count from 0, with thirty polls and thirty minutes of its
+ own. A fallback on the first attempt holds for the second: it is
+ `smaller-mode` in place of `ok`, and `note` still names the locked
+ entry. `note` keeps the first attempt, as one remark: the error in
+ the page's words, its URL, and its poll count. When the second
+ attempt errors too, the session is `errored`, and there is no
+ third.
- `timed-out`: as in step 6.
- `not-signed-in` and `not-run`: as in steps 3 and 4. There is no
- answer: `score` is null, `polls` is 0, both times are the moment
- of recording, and `<site>.md` holds the header only.
+ answer. `url` is the address the tab shows, without its query
+ string; `model_label` and `effort_label` are `not set`;
+ `read_version` is `not stated`; `score` is null; `polls` is 0;
+ both times are the moment step 3 or step 4 found it; and
+ `<site>.md` holds the header only.
An answer cut short by a tool-use limit that still satisfies the
contract is `ok` with a `note`; do not press Continue. `note` holds
every remark on the session, in the order they arose, separated by
- "; ".
+ "; ". A session with no remark has no `note` key.
8. Write `results.json` in the run folder in the schema, with `python3`:
- `run_id` is the folder name, `started_at` from step 2 and
- `finished_at` from the moment of writing, `prompt` and `contract`
- the two texts as typed, `sizes` the two sizes, and one entry per
- site. Read it back and check every required key of the schema is
- there and no other.
- 9. Per site, bring the score line into view: click an empty margin of
- the page to clear a stray selection or menu, `find` the score line,
- and `scroll_to` its reference. Take one `zoom` of the conversation
- column, right of the sidebar, with `save_to_disk`: it shows the
- score line and the chip, and no conversation title. Copy it into the
- run folder as `<site>.jpg` with `python3` (`shutil.copyfile` from
- the path the tool reports); the tool saves JPEG. These are the
- evidence a pull request carries.
- 10. Close the tabs you created, and no other.
+ `run_id` is the folder name, `started_at` and `repository_head` from
+ step 2, `finished_at` from the moment of writing, `prompt` and
+ `contract` the two texts as typed, `sizes` the two sizes, `note`
+ with the labels step 1 resolved, and one entry per site, whose
+ `response_path` is `<site>.md`. Read it back with `python3` and hold
+ it to the schema file, at the top and in each session: every
+ required key is there, no key is there that the schema does not
+ list, and each value is inside the `type`, `enum`, `minimum`,
+ `maximum`, and `format` the schema gives its key. A value outside
+ them is corrected, never bent to fit.
+ 9. Per site that sent the prompt, bring its score line into view, or,
+ where there is none, the first line the page wrote back: an answer,
+ a `refused` decline, or an `errored` message. Click an empty margin
+ to clear a stray selection or menu, `find` the line, and
+ `scroll_to` its reference. Where `find` is refused, scroll the
+ conversation with the `computer` tool's `scroll` until a screenshot
+ shows the line. Take one `zoom` of the conversation column with
+ `save_to_disk`. Its region is right of the sidebar and below the
+ page's header, where claude.ai writes the conversation's own title:
+ it shows that line and the chip, and no conversation title. Copy it
+ into the run folder as `<site>.jpg` with `python3`
+ (`shutil.copyfile` from the path the tool reports); the tool saves
+ JPEG. These are the evidence a pull request carries.
+ 10. Close the tabs `tabs_create_mcp` opened in step 3, and no other.
11. Only when `$ARGUMENTS` has `compare=`. List the earlier runs with
`python3`:
`sorted((Path.home() / "Downloads" / "benchmark_browser").glob("*/results.json"))`,
since the tools have no `ls` and `glob` does not expand `~`. With
run ids named, read those only, and name a named run that has no
`results.json` as missing. Keep the runs whose `prompt` and
`contract` are both the same text as this run's, and name each run
left out and why. Runs with other `sizes` are kept apart, in a table
of their own. The comparison is a table in the output: a row per
- site, a column per run, and in each cell the score, the model and
- effort labels, and `read_version`. Only `ok` sessions are compared;
- any other shows its status.
+ site, a column per run headed by its `run_id` and
+ `repository_head`, and in each cell the score, the model and effort
+ labels, and `read_version`. Only `ok` sessions are compared; any
+ other shows its status. A key an earlier run lacks shows as
+ `not recorded`.
## Output
In prose: the run folder path; what was measured, which is the
- repository, the version each site says it read, and the two sizes; per
- site, the model and effort labels, the score, `read_version`, and the
- conversation URL; any site that was not signed in, not run, refused,
- errored, or timed out, with the reason in the page's own words where it
- gave one; which sites were read from screenshots. When the sites read
- different versions, say so. A `smaller-mode` session is named with the
- mode its size asked for and the mode that ran, and it is left out of any
- comparison of like for like, across sites or across runs. The table
- step 11 makes follows, when it was asked for.
+ repository, its `repository_head`, the version each site says it read,
+ and the two sizes; per site, the model and effort labels, the score,
+ `read_version`, and the conversation URL; any site that was not signed
+ in, not run, refused, errored, or timed out, with the reason in the
+ page's own words where it gave one; which sites were read from
+ screenshots. When the sites read different versions, say so. A
+ `smaller-mode` session is named with the entry its size asked for and
+ the entry that ran, and it is left out of any comparison of like for
+ like, across sites or across runs. The table step 11 makes follows,
+ when it was asked for.