write-a-spec · diff
git:20260921.8da7a96 to git:20260921.bbe08f8
67 added, 0 removed. Audit A to A.
---
name: write-a-spec
description: Write or extend an E2E journey spec in evals/specs that proves a PR's change to a human reviewer. Use when a PR changes user-visible behaviour, when a reviewer asks "show me", or when the coverage decision says a journey is missing.
---
# Skill: Write a Spec
Every `evals/specs/**/*.e2e.test.ts` a PR adds or changes runs in CI on the PR
head and is published as that PR's proof (private report + one PR comment).
Write the spec for the person who will read that report in thirty seconds,
not for the machine that runs it.
## Do not write one when…
- An existing journey covers the behaviour: extend it. One spec per user
journey, not per PR; bug fixes add a step to the journey they escaped from.
- The change is a pure function: write a colocated unit test, not evidence.
- You would import `../../apps|packages|ee`, read source files, or spawn another
runner. That is a unit test in disguise; the boundary ratchet rejects it.
+ No UI does not mean no spec. A server, proxy, or protocol bug gets a
+ browser-less world (see `evals/specs/session-title-recovery.test.ts`); the
+ steps still read as one person's before → after.
+
## The proof shape
A proof answers: **who** can now do **what** they could not before, and who
still cannot. Structure every spec as that story. Each beat is one `step()`
with a claim-sized name; each visual beat ends in `user.screenshot()`.
```
persona → the test title names a person: "an owner", "a teammate", "a member without access"
before → what that person sees or cannot do today (features only)
action → the person does the thing, through the UI, as they would
after → what they now see; screenshot at that moment
boundary → who else is affected, and the negative half: who is not
```
The review app is generated from the spec. It shows exactly four strings, and
each comes from one place in your code; write those strings for the reviewer:
| Reviewer sees | Comes from | Rule |
| --- | --- | --- |
| Section heading | the `test("…")` title | names the persona and what they can now do |
| Caption under a screenshot | the `step("…")` the `user.screenshot()` ran inside | the old state starts `before:`, the new state `after:`; otherwise a plain claim |
| Caption + judgment on a `looks()` image | the first expectation in `user.looks([...])` | judged later; pending until then, so CI proof stays `Incomplete` |
| Assertion line | `recordAssertionEvidence(claim, evidence, ok)` | `claim` is the caption, `evidence` the text under it |
A screenshot taken outside any `step()` is captioned "<title> artifact N",
which tells the reviewer nothing. No verbs like "assert", no selectors, no
internal names anywhere in those strings.
Example title and steps:
```ts
test("an owner enables Code Mode and a teammate turns one chat into a shared Workflow", async ({ user, probe, step }) => {
await step("before: the teammate's agent has no script tool", …); // screenshot
await step("the owner enables Code Mode with one switch", …); // screenshot
await step("after: the teammate's request runs as one script", …); // screenshot
await step("the result is saved as a Workflow the team can open", …); // screenshot
await step("a member outside the team cannot see it", …); // screenshot
});
```
Assertions still live inside each step; they just hang off user-visible
moments. A spec whose assertions are RPC responses collected in the world and
compared in one block is a mechanism check, not a proof. It is still allowed;
it just tells the reviewer nothing.
+ ## Same story in three places
+
+ The spec, the report, and the PR body tell one story in the same words. Write
+ the report you want to read first, then make the code produce those lines.
+ A bug fix with no screen still gets this shape; the "screenshot" is one
+ `recordAssertionEvidence` line per step whose `evidence` is the observed fact.
+
+ Report the reviewer reads (every line comes from the spec):
+
+ ```
+ ▶ a member's conversation stays readable while OpenWork is still checking who owns it
+ ✔ given an engine where the ownership check is slow and the messages read is fast
+ GET /session/:id answers in 150 ms; GET /session/:id/message in 2 ms
+ ✔ when the member opens the conversation and memory is reclaimed mid-check
+ three reads sent as the app sends them; gc() ran three times
+ ✔ then the runtime really reclaimed the engine's response
+ WeakRef → undefined (witness; without it a pass means "nothing happened")
+ ✔ after: all three reads return the conversation, none say internal_error
+ 200 / 200 / 200; body {"id":"ses_x", …}
+ ```
+
+ Spec that produces it:
+
+ ```ts
+ test("a member's conversation stays readable while OpenWork is still checking who owns it", async ({ world, step, evidence }) => {
+ await step("given an engine where the ownership check is slow and the messages read is fast", async () => {
+ evidence.recordAssertionEvidence("engine timings", `GET /session/:id ${world.ownershipMs} ms; GET /session/:id/message ${world.messagesMs} ms`, true);
+ });
+ await step("when the member opens the conversation and memory is reclaimed mid-check", async () => { /* act; gc(); */ });
+ await step("then the runtime really reclaimed the engine's response", async () => {
+ const collected = world.engineResponse.deref() === undefined;
+ evidence.recordAssertionEvidence("engine response reclaimed", collected ? "WeakRef → undefined" : "WeakRef still alive: window not exercised", collected);
+ expect(collected).toBe(true);
+ });
+ await step("after: all three reads return the conversation, none say internal_error", async () => {
+ evidence.recordAssertionEvidence("reads", `${statuses.join(" / ")}; body ${body.slice(0, 40)}`, statuses.every((s) => s === 200));
+ expect(bodies.join()).not.toContain("internal_error");
+ });
+ });
+ ```
+
+ PR body line that names it (`open-a-pr`):
+
+ ```markdown
+ ## Evidence
+ `evals/specs/conversation-stays-readable.e2e.test.ts` — before: the read fails
+ with "Response body object should not be disturbed or locked"; after: all
+ three reads return the conversation and the reclaim witness is true.
+ ```
+
+ Rules that make this parseable:
+
+ - Step names are `given / when / then / after:` (or `before: / after:` with
+ screenshots). One clause each, the person's words, no function names.
+ - Every step records exactly one evidence line: the observed fact, with the
+ numbers in it. The reviewer never opens the code to learn what happened.
+ - A **witness** step proves the risky condition actually occurred. If it did
+ not, that step fails; a pass without the witness is not a proof.
+ - The quoted "before:" string in the PR body is the exact error the user or
+ Sentry saw. CI shows it red on `dev` and green on the head; never narrate it.
+
Rules of thumb:
- At least three steps. Zero steps renders as "no claims declared".
- Every step has at least one `user.*` act or `user.see`. `probe.*` observes;
it does not carry a step alone.
- One screenshot per visual beat. Non-visual work records `recordAssertionEvidence`
with the command or response excerpt as the evidence text.
- For features, show the before state in the same world. Base-vs-head runs are
expensive; a step that starts with the switch off is not.
- For permissions, sharing, or scopes, always include the negative persona.
+ - The world runs the code on the runtime it ships on. A spec that passes on
+ the wrong runtime proves nothing (`run-tests` → Match the runtime).
- Prefer `seed.appWeb` (headless Chrome, real app). Use `seed.desktop` only for
a native capability a browser cannot show, and say why in `nativeReason`.
- No `seed.evalIn` / `probe.eval` in new specs. If you need one, comment why.
## Use the testkit channels
Import `spec` (and `expect`) from `@openwork/testkit`; bind the world:
```ts
const test = spec.world(myWorld, {
resources: { surfaces: ["appWeb"], services: ["den"] },
});
```
| Channel | Use it for |
| --- | --- |
| `seed` | Arrange the world: Den, orgs, members, workspaces, sessions, mocks, faults. Only `seed` writes state. All `seed.*` goes in the world, before the first act. |
| `user` | Act as the person: `click`, `type`, `press`, `reload`, `see`, `notSee`, `looks`, `screenshot`. Trusted CDP input; no JS evaluation. |
| `agent` | Drive the product's automation rail (`window.__openworkControl`): sends, session actions. |
| `probe` | Observe without changing state: `text`, `hash`, `storage`, `api` (GET), `dom`, `eventually`. |
| `step` | Name a claim. Nests. A failed step is recorded and rethrown; later steps show `not-reached`. |
Well-known targets: `"composer"` for the editor; otherwise role, label, text,
placeholder, or test id. Bound every wait; declare external requirements in
`needs()` so a missing dependency skips loudly.
## Claims and witnesses
- Every claim is machine-checkable with an observable assertion and its
negative half: what must not happen to another identity, account, or state.
- Prose is never proof. Screenshots explain an assertion; they cannot replace it.
- Describe product behaviour, not incidental layout.
- If product and claim diverge, change one explicitly; never bend the claim.
- Never smuggle the answer into the prompt: assert the user-facing request
carries no connector or resource IDs.
- Use `mcpMock()` witnesses under `evals/packages/labs/src/`; never call real
providers. Witnesses are deterministic, identity-scoped, and queryable.
## Evidence contract
- Evidence is ambient: `user.screenshot()` records an artifact, `user.looks()`
records a visual validation (judged later; pending until then),
`recordAssertionEvidence()` records a witness assertion. Never create or
pass recorder handles.
- Run it once before pushing and read your own record:
```sh
pnpm evals:e2e <slug> --local
ls evals/results/test-runs/<latest>/ # index.html, test-run.json, NN-*.png
```
If the screenshots would not convince you, they will not convince the reviewer.
Read the captions in `index.html` top to bottom: they should tell the
before → after story on their own.
- In CI the spec runs on `PR change proof`, one job per spec; the trusted
publisher aggregates every changed spec's records into one report. Failed,
skipped, and cancelled runs stay visible as such; nothing substitutes for them.