write-a-spec · diff
git:20260904.13c282d to git:20260920.e759462
96 added, 42 removed. Audit A to A.
---
name: write-a-spec
- description: Extend or write a journey spec in evals/specs. Use only after the coverage decision says a journey spec is missing an assertion or a new user journey exists.
+ description: Write or extend an E2E journey spec in evals/specs that proves a PR's change to a human reviewer. Use when a PR changes user-visible behaviour, when a reviewer asks "show me", or when the coverage decision says a journey is missing.
---
# Skill: Write a Spec
+ Every `evals/specs/**/*.e2e.test.ts` a PR adds or changes runs in CI on the PR
+ head and is published as that PR's proof (private report + one PR comment).
+ Write the spec for the person who will read that report in thirty seconds,
+ not for the machine that runs it.
+
## Do not write one when…
- - An existing journey spec covers the behaviour; extend it.
- - The change is a pure function; write a colocated unit test, not evidence.
+ - An existing journey covers the behaviour: extend it. One spec per user
+ journey, not per PR; bug fixes add a step to the journey they escaped from.
+ - The change is a pure function: write a colocated unit test, not evidence.
- You would import `../../apps|packages|ee`, read source files, or spawn another
- test runner. That is a unit test in disguise; the boundary ratchet rejects it.
+ runner. That is a unit test in disguise; the boundary ratchet rejects it.
- One spec per user journey, not per PR; bug fixes add an assertion to the journey they escaped from.
+ ## The proof shape
- Write new tests in `evals/specs/**/*.test.ts` and import `test` from
- `@openwork/testkit`. App-driving E2E tests use `.e2e.test.ts`; the PR lane excludes
- them. Model setup as resources in dependency order: `needs()` → `server()` →
- `app()`.
+ A proof answers: **who** can now do **what** they could not before, and who
+ still cannot. Structure every spec as that story. Each beat is one `step()`
+ with a claim-sized name; each visual beat ends in `user.screenshot()`.
- ## Use the testkit primitives
+ ```
+ persona → the test title names a person: "an owner", "a teammate", "a member without access"
+ before → what that person sees or cannot do today (features only)
+ action → the person does the thing, through the UI, as they would
+ after → what they now see; screenshot at that moment
+ boundary → who else is affected, and the negative half: who is not
+ ```
- - `server()` boots or reuses Den and provisions isolated organizations.
- - `app()` boots a signed-in desktop. Use `profileDir` for caller-owned profile
- continuity and `localServerDelayMs` for deterministic startup races.
- - `inviteMember()` adds a named member to an existing Den.
- - `faultProxy()` injects `faults.status()` or `faults.latency()` and exposes the
- `requests` log for assertions about attempts and recovery.
- - `eventually()` bounds polling and reports its last value or error.
- - `readDenClientState()`, `readConnectState()`, and `readConnectStateFile()`
- expose client, local-server, and persisted-profile state.
+ Example title and steps:
- ## Claims and witnesses
+ ```ts
+ test("an owner enables Code Mode and a teammate turns one chat into a shared Workflow", async ({ user, probe, step }) => {
+ await step("before: the teammate's agent has no script tool", …); // screenshot
+ await step("the owner enables Code Mode with one switch", …); // screenshot
+ await step("the teammate's request now runs as one script", …); // screenshot
+ await step("the result is saved as a Workflow the team can open", …); // screenshot
+ await step("a member outside the team cannot see it", …); // screenshot
+ });
+ ```
- - Make each claim machine-checkable with an observable assertion and its
- explicit negative half. Assert both the intended effect and what must not
- happen to another identity, account, request, file, or state.
- - Prose is never proof. Screenshots explain an assertion but cannot replace it.
- - Describe product behavior, not incidental layout. Claims such as "side by
- side" can disagree even when pixels are identical across runs.
- - Match claims to what the product actually says on screen. If product and
- claim diverge, explicitly change one; never silently bend the claim.
- - Never smuggle the answer into the prompt. Assert that the user-facing request
- does not contain connector or resource IDs.
+ Assertions still live inside each step; they just hang off user-visible
+ moments. A spec whose assertions are RPC responses collected in the world and
+ compared in one block is a mechanism check, not a proof. It is still allowed;
+ it just tells the reviewer nothing.
- ## Mocks
+ Rules of thumb:
- - Use `mcpMock()` witnesses; never exercise real providers from a spec.
- - Witnesses live under `evals/packages/labs/src/`, following `mock-mcp.ts` and
- the provider-specific `mock-*.ts` fixtures.
- - Keep witnesses deterministic, identity-scoped, and queryable for assertions.
+ - At least three steps. Zero steps renders as "no claims declared".
+ - Every step has at least one `user.*` act or `user.see`. `probe.*` observes;
+ it does not carry a step alone.
+ - One screenshot per visual beat. Non-visual work records `recordAssertionEvidence`
+ with the command or response excerpt as the evidence text.
+ - For features, show the before state in the same world. Base-vs-head runs are
+ expensive; a step that starts with the switch off is not.
+ - For permissions, sharing, or scopes, always include the negative persona.
+ - Prefer `seed.appWeb` (headless Chrome, real app). Use `seed.desktop` only for
+ a native capability a browser cannot show, and say why in `nativeReason`.
+ - No `seed.evalIn` / `probe.eval` in new specs. If you need one, comment why.
+ ## Use the testkit channels
+
+ Import `spec` (and `expect`) from `@openwork/testkit`; bind the world:
+
+ ```ts
+ const test = spec.world(myWorld, {
+ resources: { surfaces: ["appWeb"], services: ["den"] },
+ });
+ ```
+
+ | Channel | Use it for |
+ | --- | --- |
+ | `seed` | Arrange the world: Den, orgs, members, workspaces, sessions, mocks, faults. Only `seed` writes state. All `seed.*` goes in the world, before the first act. |
+ | `user` | Act as the person: `click`, `type`, `press`, `reload`, `see`, `notSee`, `looks`, `screenshot`. Trusted CDP input; no JS evaluation. |
+ | `agent` | Drive the product's automation rail (`window.__openworkControl`): sends, session actions. |
+ | `probe` | Observe without changing state: `text`, `hash`, `storage`, `api` (GET), `dom`, `eventually`. |
+ | `step` | Name a claim. Nests. A failed step is recorded and rethrown; later steps show `not-reached`. |
+
+ Well-known targets: `"composer"` for the editor; otherwise role, label, text,
+ placeholder, or test id. Bound every wait; declare external requirements in
+ `needs()` so a missing dependency skips loudly.
+
+ ## Claims and witnesses
+
+ - Every claim is machine-checkable with an observable assertion and its
+ negative half: what must not happen to another identity, account, or state.
+ - Prose is never proof. Screenshots explain an assertion; they cannot replace it.
+ - Describe product behaviour, not incidental layout.
+ - If product and claim diverge, change one explicitly; never bend the claim.
+ - Never smuggle the answer into the prompt: assert the user-facing request
+ carries no connector or resource IDs.
+ - Use `mcpMock()` witnesses under `evals/packages/labs/src/`; never call real
+ providers. Witnesses are deterministic, identity-scoped, and queryable.
+
## Evidence contract
- - Test evidence is ambient: `screenshot()` records screenshot artifacts,
- `validate()` records their visual validations whether they pass or fail, and
- `recordAssertionEvidence()` holds witness assertions.
- - Never create or pass test-evidence recorder handles in test bodies.
- - Bound every wait.
- - Declare every external requirement in `needs()` so missing dependencies skip
- loudly instead of timing out or weakening coverage.
+ - Evidence is ambient: `user.screenshot()` records an artifact, `user.looks()`
+ records a visual validation (judged later; pending until then),
+ `recordAssertionEvidence()` records a witness assertion. Never create or
+ pass recorder handles.
+ - Run it once before pushing and read your own record:
+
+ ```sh
+ pnpm evals:e2e <slug> --local
+ ls evals/results/test-runs/<latest>/ # index.html, test-run.json, NN-*.png
+ ```
+
+ If the screenshots would not convince you, they will not convince the reviewer.
+ - In CI the spec runs on `PR change proof`, one job per spec; the trusted
+ publisher aggregates every changed spec's records into one report. Failed,
+ skipped, and cancelled runs stay visible as such; nothing substitutes for them.