git:20260924.3aceaab to git:20260924.17234b5

222 added, 407 removed. Audit A to A.

---

name: pr-test-automation

- description: Run first-pass automated manual testing for PRs that are reviewed or nearly ready to merge. Use when the user asks you to test a PR, branch, or worktree, validate product flows, exercise browser or CLI workflows, map changed UI journeys with screenshots, verify analytics, webhook, payment, email, or SMS behavior through connected tools, or produce manual QA notes before human testing.

+ description: Run first-pass automated manual testing for PRs that are reviewed or nearly ready to merge. Use when the user asks Codex to test a PR/branch/worktree, validate product flows, exercise browser or CLI workflows, map changed UI journeys with screenshots, verify analytics/webhooks/payments/email/SMS behavior through connected tools, or produce manual QA notes before human testing.

---

- # PR test automation

- Validate as much of a PR as you can with local services, browser automation,

- CLIs, logs, and product connectors before the user does final manual testing.

- This is first-pass QA: prove what works with evidence, identify what still

- needs a human, and leave a reproducible trail.

- ## Before you start: preflight

- QA evidence is current-head evidence, and the Manual tests checklist is the PR

- body's contract. Anything that would change either runs first:

- - On a large PR (over 10 files or 300 hand-written lines) that has not had a

- `refactor` pass, say so and offer one before driving anything. A refactor

- landed after QA means this whole pass runs again.

- - Run `cold-read` on the PR body before QA, so the checklist you execute is the

- one the reader will see.

+ # PR Test Automation

- Then confirm the PR is testable. Pass the PR number or URL to every

- `gh pr view` call; bare `gh pr view` uses the current branch's PR, which may

- differ from the test target.

+ ## Overview

- - **Mergeability.** Check `gh pr view <PR> --json mergeable,mergeStateStatus`.

- A conflicting PR may get no gating CI run, and its head will change on merge.

- If it conflicts, report blocked and stop before driving.

- - **Head SHA incorporated.** Confirm the PR's `headRefOid` is in the tested

- state (`gh pr view <PR> --json headRefOid`). For companion PRs tested

- together, local HEAD may be a merge commit of several PR heads: check the

- target head is in the ancestry

- (`git merge-base --is-ancestor <headRefOid> HEAD`) and record the composite

- SHA. Evidence must come from the code the reviewer is looking at.

- - **CI existence.** Check that at least one workflow run exists for the head

- commit (`gh run list --commit <headRefOid>`). If the repo has CI and no run

- registered, something is wrong (path filters, a conflicting state, a

- workflow syntax error). Note it as a finding.

- - **Tools alive.** Verify every tool the run needs before the first long flow:

- authenticated CLIs, running services, connectors, test-mode keys. A flow

- that dies at step 7 on a missing login wastes the whole run.

+ Validate as much of a PR as possible with local services, browser automation, CLIs, logs, and product connectors before the user does final manual testing. Treat this as a first-pass QA workflow: prove what works with evidence, identify what still needs a human, and preserve a reproducible trail.

## Workflow

1. Confirm the test target:

- - Identify PR numbers, branches, worktrees, related companion PRs, and

- whether the user allowed rebasing or syncing.

- - Check `git status`, the current branch, remotes, and any unrelated local

- changes.

- - Read PR descriptions and review notes that define required manual flows.

+ - Identify PR numbers, branches, worktrees, related companion PRs, and whether the user allowed rebasing/syncing.

+ - Check `git status`, current branch, remotes, and whether unrelated local changes exist.

+ - Inspect PR descriptions/review notes when they define required manual flows.

2. Check tools and authentication up front:

- - Verify required CLIs and connectors before long tests: `gh auth status`,

- `stripe --version` and active listener state, Docker status, PostHog,

- GitHub, or Gmail connectors, cloud CLIs, and app-specific CLIs. Confirm

- credentials are current.

- - Look for verification tools before deciding a check needs a human. If

- Composio is available, use `composio search` to find inbox, SMS or phone,

- payment, CRM, support, or provider-log tools, then inspect schemas with

- `--get-schema` before executing.

- - Prefer connected app tools for verifying product data.

- - Use test-mode accounts, test keys, local containers, and staging-safe

- endpoints unless the user explicitly asks for production verification.

+ - Verify required CLIs and connectors before starting long tests: `gh auth status`, `stripe --version` / active listener state, Docker status, PostHog/GitHub/Gmail connectors, cloud CLIs, or app-specific CLIs.

+ - Discover verification tools before assuming a manual check is required. If Composio is available, use `composio search` to find candidate inbox, SMS/phone, payment, CRM, support, or provider-log tools, then inspect schemas with `--get-schema` before executing.

+ - Prefer connected app tools for product data verification. Do not assume credentials are current.

+ - Use test-mode accounts, test keys, local containers, and staging-safe endpoints unless the user explicitly asks for production verification.

3. Prepare the environment:

- - Install dependencies only where needed, and report anything that changes

- lockfiles.

- - Start required dev servers, or confirm existing sessions, ports, and

- mounted worktrees.

- - For companion PRs, test the combined state in the worktree or container

- that actually serves the code.

- - Keep one instance of each background listener or server. List and clean

- up only the processes you started for the test.

+ - Install dependencies only where needed and report anything that changes lockfiles.

+ - Start required dev servers or confirm existing sessions, ports, and mounted worktrees.

+ - For companion PRs, test the combined state in the worktree/container that actually serves the code.

+ - Avoid leaving duplicate background listeners or servers. List and clean up only processes started for the test.

4. Build the automated test path:

- - Use Playwright when browser behavior matters. If the repo lacks it,

- install it in a temporary directory outside the repo.

- - Use stable, user-visible selectors first: labels, placeholders, button

- text, URLs, and route state.

- - Generate unique short test identities and attribution markers such as

- `agent-e2e-<timestamp>`.

- - When UI changes are in scope, map each touched surface and user journey

- to screenshots in an easy-to-find temporary folder such as

- `tmp/pr-<number>-qa/` or `tmp/<branch>-qa/`. Name files in journey order,

- such as `01-signup-account.png` and `02-dropdown-expanded.png`.

- - Capture the meaningful states along the way: empty or default, filled or

- selected, expanded menus, modals, validation errors, loading and success,

- and at least one narrow viewport when responsive layout may be affected.

- - Use the same screenshot pattern for local and dev validation and, when the

- user asks for post-merge production verification, for production. Keep

- local and production artifacts in separate folders or filenames.

- - When a scriptable browser driver runs a journey, record it as a video

- alongside the stills:

- - One video per journey, recorded at the driver level (for example

- Playwright's `recordVideo`), so it comes free with the drive.

- - Keep the driver's native format: WebM from a browser, MP4 from a

- simulator.

- - Videos add to the stills. Per-step captures stay the frame-addressable

- evidence; the video is the continuity check.

- - Where ffmpeg is available, scan for blank-frame bands

- (`ffprobe -f lavfi "movie=<video>,fps=5,signalstats" -show_entries frame=pts_time -show_entries frame_tags=lavfi.signalstats.YAVG`;

- YAVG ~235 is blank white). Report layout jumps, white flashes, and dead

- time as findings with timestamp ranges.

- - Prefer the app's built-in test or simulation path for external effects:

- local inboxes, Mailhog-style UIs, fake SMS numbers, test OTP logs, sandbox

- payment modes, webhook listeners, or provider test keys.

- - Parse email or SMS verification links and codes from container logs when

- the local environment emits them.

- - Add small human-paced waits around analytics and step transitions, so

- effects and batched events fire in the order a user would see them.

+ - Use Playwright when browser behavior matters. If the repo lacks Playwright, install it in a temporary directory rather than polluting the repo.

+ - Use stable, user-visible selectors first: labels, placeholders, button text, URLs, and route state.

+ - Generate unique short test identities and attribution markers such as `agent-e2e-<timestamp>`.

+ - When UI changes are in scope, map each touched surface area and user journey to screenshots in a temporary, easy-to-observe folder such as `tmp/pr-<number>-qa/` or `tmp/<branch>-qa/`. Use ordered filenames that describe the journey step, such as `01-signup-account.png` and `02-dropdown-expanded.png`.

+ - Capture meaningful UI states, not only final pages: empty/default, filled/selected, expanded menus, modals, validation errors, loading/success states, and at least one narrow viewport when responsive layout is likely affected.

+ - Reuse the same screenshot artifact pattern for local/dev validation and, when the user asks for post-merge production verification, for production paths. Keep local and production artifacts separated by folder or filename.

+ - When a journey is driven by a scriptable browser driver, record it as a video alongside the stills. One video per journey, recorded at the driver level (e.g. Playwright's `recordVideo`) so it is a free byproduct of the drive, not a second pass. Keep the driver's native format (WebM from browser, MP4 from simulator). Videos complement stills, never replace them: per-step captures remain the frame-addressable evidence, the video is the continuity check. Where ffmpeg is available, scan for blank-frame bands (`ffprobe -f lavfi "movie=<video>,fps=5,signalstats" -show_entries frame=pts_time -show_entries frame_tags=lavfi.signalstats.YAVG`; YAVG ~235 is blank white) and report layout jumps, white flashes, or dead time as findings with timestamp ranges.

+ - Prefer the app's built-in test/simulation path for external effects: local inboxes, Mailhog-style UIs, fake SMS numbers, test OTP logs, sandbox payment modes, webhook listeners, or provider test keys.

+ - Parse local email/SMS verification links or codes from container logs when the local environment emits them.

+ - Add small human-paced waits around analytics or step-transition tests so effects and batched events have time to fire in the same order a user would experience.

- 5. Verify externally as well as locally:

- - Network requests prove the browser tried to send data. Connector or API

- queries prove the product received it.

- - When simulation is unavailable, read back from the recipient or provider:

- - email: Gmail, Outlook, IMAP, or email-service activity

- - SMS and voice: Twilio, Dialpad, OpenPhone, Google Voice, test-number

- services, or provider logs

- - payments: Stripe or provider dashboards

- - Query by the unique marker, test email, phone number, org ID, subscription

- ID, webhook event ID, request ID, or another stable test value.

- - For webhooks, confirm both the CLI or listener output and the backend

- logs, then verify downstream data.

- - For analytics dashboards, query the exact project and state the date range

- and filters used.

+ 5. Verify externally, not just locally:

+ - Network requests prove the browser tried to send data; connector/API queries prove the product received it.

+ - When simulation is unavailable, use connected recipient/provider readback: Gmail/Outlook/IMAP or email-service activity for email; Twilio/Dialpad/OpenPhone/Google Voice/test-number services or provider logs for SMS/voice; Stripe/provider dashboards for payments.

+ - Query by the unique marker, test email, phone number, org ID, subscription ID, webhook event ID, request ID, or other stable test value.

+ - For webhooks, confirm both CLI/listener output and backend logs, then verify downstream data.

+ - For analytics dashboards, query the exact project and call out the date range and filters used.

- 6. Report results. Return what passed, what is still uncertain, the next check,

- and up to three improvements grounded in testing friction. Open with a

- verdict, then the evidence:

+ 6. Report results:

+ - Return what passed, remaining uncertainty, the next check, and up to three improvements grounded in testing friction.

+ Open with a verdict, then the evidence. The structure:

```

Verdict: <all-proven | partial | blocked-env | blocked-auth | product-bug-found>

| Journey / Check | Result | Evidence |

|-----------------|--------|----------|

| <flow or check> | Pass / Fail / Blocked / Left to human | <quoted output, screenshot ref, connector readback> |

Skipped (with rationale):

- <item>: <why it was skipped, not just "skipped">

Cleanup disposition:

|---------|--------|--------|-------------|

```

- A pass needs quoted evidence. "Blocked" is terminal: when a check can't be

- exercised (missing env, service down, no credentials), stop on it and

- report it blocked. Only the real test route produces evidence; an

- improvised workaround proves nothing.

+ A pass without quoted evidence is not a pass. "Blocked" is a terminal

+ verdict: when a check cannot be exercised (missing env, service down,

+ no credentials), stop trying rather than improvising a workaround.

+ Improvised test routes are not evidence.

- Terminal states and the human's next step:

+ Terminal states and what the human should do next:

- `all-proven`: every check passed with evidence. Ready for human review.

- - `partial`: some checks passed and others are left to the human. List the

- gaps.

- - `blocked-env`: an environment issue (service down, container missing).

- Name the blocker.

- - `blocked-auth`: missing credentials or connector scopes. Name what is

- needed.

- - `blocked-timeout`: the wall clock expired first. List what finished and

- what remains.

- - `product-bug-found`: a check failed in the product. Describe the bug with

- evidence.

+ - `partial`: some checks passed, others left to human. List the gaps.

+ - `blocked-env`: environment issue (service down, container missing). Name the blocker.

+ - `blocked-auth`: missing credentials or connector scopes. Name what is needed.

+ - `blocked-timeout`: wall clock expired before all checks completed. List what finished and what remains.

+ - `product-bug-found`: a check failed and the failure is in the product. Describe the bug with evidence.

- Beyond the verdict, include:

- - the exact test identity or marker used for external queries

- - screenshot paths for changed UI, grouped by journey step, with the

- surface, environment, and UI state each covers

- - video paths for journey recordings, with duration and what each shows

- - connector and query evidence: event names, identifiers, timestamps, and

- key properties

- - each intentionally skipped item and why, one by one

- - artifacts of the test harness (mocked browser properties, prevented

- navigation, masked automation signals)

+ Beyond the verdict, the report must include:

+ - The exact test identity/marker used for external queries.

+ - Screenshot paths for changed UI, organized by journey step with the surface area, environment, and UI state each covers.

+ - Video paths for journey recordings, with duration and what each evidences.

+ - Connector/query evidence with event names, identifiers, timestamps, and important properties.

+ - What was intentionally skipped and why (per item, not a blanket "some things were skipped").

+ - Artifacts caused by the test harness (mocked browser properties, prevented navigation, masked automation signals).

- Publish shareable screenshots and update the PR as described in

- [Durable PR QA](#durable-pr-qa-descriptions-comments-and-screenshots).

+ When screenshots are safe to share, publish them on a repository-owned durable asset surface. For GitHub PRs, prefer an existing long-lived release such as `pr-assets`; do not use an arbitrary temporary host when a suitable repository release is available. For GitHub PR targets where the user asked for PR testing, update the PR description with a concise QA summary where reviewers look first. Use a marked section so reruns replace the latest QA summary without overwriting the author-written description. Use a separate marked QA comment for long evidence, logs, and screenshot galleries when the description would become unwieldy.

- ## PostHog and browser analytics

+ ## PostHog And Browser Analytics

- PostHog JavaScript drops capture events from likely bots. Headless Playwright

- can still fetch PostHog config and run `identify`, while `capture` events are

- silently dropped because `navigator.webdriver` is `true`, the user agent looks

- automated, or `navigator.userAgentData.brands` includes `HeadlessChrome`.

+ PostHog JavaScript drops capture events from likely bots. Headless Playwright can still fetch PostHog config and run `identify`, while `capture` events are silently dropped because `navigator.webdriver` is `true`, the user agent looks automated, or `navigator.userAgentData.brands` includes `HeadlessChrome`.

When the explicit goal is to validate product analytics in local automation:

- Use a normal browser user agent.

- - Mask only the automation signals, for this test context. Override

- `navigator.userAgentData` too; setting `userAgent` alone leaves headless

- Chrome visible there:

+ - Mask only the automation bot signals for the test context. Setting `userAgent` is not enough if `navigator.userAgentData` still exposes headless Chrome:

```js

const context = await browser.newContext({

userAgent:

'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +

'(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36',

})

await context.addInitScript(() => {

const brands = [

{ brand: 'Chromium', version: '125' },

{ brand: 'Google Chrome', version: '125' },

{ brand: 'Not.A/Brand', version: '24' },

]

const fullVersionList = brands.map((brand) => ({

...brand,

version: `${brand.version}.0.0.0`,

}))

Object.defineProperty(navigator, 'webdriver', { get: () => undefined })

Object.defineProperty(navigator, 'userAgentData', {

get: () => ({

brands,

mobile: false,

platform: 'Windows',

getHighEntropyValues: async () => ({

brands,

mobile: false,

platform: 'Windows',

architecture: 'x86',

bitness: '64',

model: '',

uaFullVersion: '125.0.0.0',

fullVersionList,

}),

toJSON: () => ({ brands, mobile: false, platform: 'Windows' }),

}),

})

```

- - If the UI branches on OS or desktop versus mobile, set the platform signal

- on purpose and disclose it:

+ - If the UI branches on OS or desktop/mobile, set the relevant platform signal intentionally and disclose it:

```js

await context.addInitScript(() => {

Object.defineProperty(navigator, 'platform', { get: () => 'Win32' })

})

```

- - Keep the page open long enough for PostHog batching, or trigger an unload

- only after waiting.

- - Query PostHog for the unique marker. Ingested capture events are the

- evidence; `flags` and config requests only show the SDK loaded.

- - If event order looks wrong under automation, rerun with human-paced waits

- before calling it a product bug.

- ## Analytics identity verification

- When a PR touches analytics, signup, login, or session handling, verify person

- stitching as well as event ingestion:

- - Group verification queries by `person_id`. Event-time `person.properties.*`

- differ per row, so an email-grouped check can pass while several QA users

- have merged into one person.

- - When stitching looks wrong, inspect the raw `distinct_id` per event. It

- names the identity that captured the event and usually points straight at

- the merge vector.

- - When the PR touches identity stitching itself (aliasing, identify calls,

- distinct-id or session-identity plumbing), or the product targets shared

- devices, run a **multi-user same-browser pass**:

- - several signups and login switches in one browser profile

- - assert each user resolved to a separate person

- - assert functional session state (websocket auth, cookies) followed each

- switch

- Single-user passes can't see shared-machine bugs such as identity merges or

- stale-socket auth. The pass is expensive, so reserve it for changes where

- that failure mode is in play.

- - When a fix's re-verification still fails, suspect stacked causes: fix one

- vector, re-run the proof, and let the raw `distinct_id` data name the next.

- - Test events fired just before hard navigations (checkout redirects, external

- scheduling links). SDK batching drops them on unload; they need per-capture

- `sendBeacon` transport. Prove it in the warehouse, where a dropped event is

- absent.

+ - Keep the page open long enough for PostHog batching, or trigger an unload only after waiting.

+ - Query PostHog for the unique marker. Do not treat `flags` or config requests as evidence that capture events were ingested.

+ - If event order looks wrong under automation, rerun with human-paced waits before treating it as a product bug.

- ## Multi-surface attribution flows

+ ## Multi-Surface Attribution Flows

- Some PRs only work when a marketing site, API route, installer, desktop app,

- or mobile client is tested as one product flow. When attribution or install

- and download analytics cross those boundaries:

+ Some PRs only work when a marketing site, API route, installer, desktop app, or mobile client is tested as one product flow. When attribution or install/download analytics cross those boundaries:

- - Test companion PRs together in the worktree or preview environment that

- actually serves each surface.

- - Validate route-level behavior directly before browser testing. For install

- and download flows, assert that fresh tokens are accepted, stale or

- malformed tokens are dropped, invalid files or inputs are rejected, and

- crawler-facing routes are excluded when needed.

- - In browser clipboard tests, compare the visible UI text with the clipboard

- text. The visible command or link often stays clean while the copied value

- carries a hidden `ref`, `utm`, or attribution token.

- - If client-side analytics must create a distinct ID but production capture

- is out of scope, use a dummy public key and intercept the analytics

- endpoints. If production verification is requested, use a unique marker and

- query the analytics project afterward.

- - Use temporary app data directories for native app tests, so config

- migrations, attribution files, cookies, and local databases leave the

- user's real profile alone.

- - In Electron, Tauri, React Native, or similar native-shell mocks, event

- subscription APIs must return cleanup functions. Promise-returning mocks for

- `on*` or `subscribe*` APIs cause false crashes that look like product

- regressions.

- - Captures that fire before an analytics SDK finishes initializing need

- explicit host, token, and distinct-ID assertions. A request that falls back

- to the SDK vendor's default host before app config loads is usually a

- product bug.

- - Cover the happy path and one negative path: accepted or refreshed

- attribution, stale or malformed attribution, user opt-in or opt-out, and any

- server-side invalid-input analytics.

+ - Test companion PRs together in the worktree or preview environment that actually serves each surface.

+ - Validate route-level behavior directly before browser testing. For install/download flows, assert fresh tokens are accepted, stale or malformed tokens are dropped, invalid files or inputs are rejected, and crawler-facing routes are excluded when needed.

+ - Browser clipboard tests should compare visible UI text with clipboard text. It is common for the visible command/link to stay clean while the copied value includes a hidden `ref`, `utm`, or attribution token.

+ - If client-side analytics must create a distinct id but production capture is not part of the test, use a dummy public key and intercept analytics endpoints. If production verification is requested, use a unique marker and query the analytics project afterward.

+ - Use temporary app data directories for native app tests so config migrations, attribution files, cookies, and local databases do not touch the user's real profile.

+ - For Electron, Tauri, React Native, or similar native-shell mocks, event subscription APIs must return cleanup functions. Promise-returning mocks for `on*` or `subscribe*` APIs can create false crashes that look like product regressions.

+ - Direct captures that fire before an analytics SDK is fully initialized need explicit host, token, and distinct-id assertions. A request falling back to the SDK vendor default host before app config loads is usually a product bug, not enough evidence that the event will reach the intended project.

+ - Capture both the happy path and one negative path: accepted/refreshed attribution, stale or malformed attribution, user opt-in or opt-out, and any server-side invalid-input analytics.

- ## External integrations

+ ## External Integrations

For payment, email, SMS, analytics, and other third-party integrations:

- - Confirm the account, project, and mode before running tests.

- - Prefer test-mode objects and fake or test cards.

- - When the PR's behavior depends on delivery, ingestion, webhook receipt, or

- downstream processing, get recipient- or provider-side evidence. A 200 from

- the app or provider only shows the send.

- - Use plus-addresses, reserved fake phone numbers, sandbox identities,

- metadata, notes, UTM values, or request IDs, so every external artifact can

- be found without ambiguity.

- - Respect production stop boundaries. Never bypass MFA, consume one-time

- tokens, send real calls or SMS, create paid subscriptions, charge cards, or

- mutate customer data without the user's explicit approval for that

- production action.

- - Check for an existing listener before starting a webhook listener.

- - Record the IDs that let the user or a future agent find the test again:

- email, phone number, org key, customer ID, subscription ID, message ID,

- webhook event type, dashboard URL, event marker, or screenshot path.

- - If a provider key lacks read scopes, try another non-destructive readback:

- a connected mailbox, a recipient-side tool, a provider dashboard export, an

- app database row, a webhook table, logs, or an analytics event. Report the

- scope limitation as an environment limit.

- - Never expose secrets in the final answer. Public analytics tokens differ

- from private API keys, but describe them carefully too.

- ## Durable PR QA descriptions, comments, and screenshots

- When testing an open PR, put the result where reviewers look first.

- ### Collect and classify

+ - Confirm the account/project/mode before running tests.

+ - Prefer test-mode objects and fake/test cards.

+ - Prefer recipient/provider-side evidence over send-side success. A 200 response from the app or provider is useful but not enough when the PR's behavior depends on actual delivery, ingestion, webhook receipt, or downstream processing.

+ - Use plus-addresses, reserved fake phone numbers, sandbox identities, metadata, notes, UTM values, or request IDs so every external artifact can be found without ambiguity.

+ - Respect production stop boundaries. Do not bypass MFA, consume one-time tokens, send real calls/SMS, create paid subscriptions, charge cards, or mutate customer data unless the user explicitly approved that production action.

+ - Check for duplicate listeners before starting a new webhook listener.

+ - Record IDs that let the user or future agent find the test again: email, phone number, org key, customer ID, subscription ID, message ID, webhook event type, dashboard URL, event marker, or screenshot path.

+ - If a provider key lacks read scopes, try another non-destructive readback source such as a connected mailbox, recipient-side tool, provider dashboard export, app database row, webhook table, logs, or analytics event. Report the scope limitation rather than treating it as product failure.

+ - Never expose secrets in the final answer. Public analytics tokens are not the same as private API keys, but still describe them carefully.

- - Create a local artifact folder such as `tmp/pr-<number>-qa/` with raw

- screenshots, scripts, the exact PR Markdown, and `pr-assets-manifest.json`.

- - Classify every image before upload. Upload only what every PR reader may

- see. Keep local anything with PHI, secrets, private customer data, real

- inbox contents, payment details, MFA codes, or production admin data.

- Redact a copy only when you can verify the redaction visually.

+ ## Durable PR QA Descriptions, Comments, And Screenshots

- ### Publish to a repository release

+ When testing an open PR, preserve the result where reviewers will look first:

- - Use a repository-owned durable surface. For GitHub PRs, find and reuse a

- published, mutable, long-lived release such as `pr-assets` or the

- repository's documented equivalent. One release serves every PR. Use it in

- preference to any arbitrary temporary host:

+ - Create a local artifact folder such as `tmp/pr-<number>-qa/` containing raw screenshots, scripts, the exact PR Markdown, and `pr-assets-manifest.json`.

+ - Classify every image before upload. Do not upload PHI, secrets, private customer data, real inbox contents, payment details, MFA codes, production admin data, or anything inappropriate for every person who can read the PR. Keep sensitive images local and redact a copy only when the redaction can be verified visually.

+ - Prefer a repository-owned durable surface. For GitHub PRs, discover and reuse a published, mutable, long-lived release such as `pr-assets` or the repository's documented equivalent:

```bash

repo="$(gh repo view --json nameWithOwner --jq .nameWithOwner)"

default_branch="$(gh repo view --json defaultBranchRef --jq .defaultBranchRef.name)"

gh release list -R "$repo" --limit 100 \

--json tagName,name,isDraft,isPrerelease

gh release view pr-assets -R "$repo" \

--json tagName,isDraft,isPrerelease,isImmutable,url,assets

```

- - If no suitable release exists, creating one long-lived `pr-assets` release

- is a separate hard stop. It needs an exact grant such as

- `{"action":"create_release","repo":"owner/name","tag":"pr-assets"}`; general

- GitHub, PR, comment, or upload authorization does not cover it. Target the

- default branch and keep it out of Latest-release semantics:

+ Do not create a release per PR. Do not use an arbitrary temporary host when a suitable repository release exists.

+ - If no suitable release exists, creating one dedicated long-lived `pr-assets` release is a separate hard stop requiring an exact grant such as `{"action":"create_release","repo":"owner/name","tag":"pr-assets"}`. Generic GitHub, PR, comment, or asset-upload authorization does not grant creation. Use the default branch as its target and keep it out of Latest-release semantics:

```bash

gh release create pr-assets -R "$repo" --title "PR assets" \

--notes "Long-lived image assets for pull requests and QA evidence." \

--latest=false --target "$default_branch"

```

- - If release creation or upload is not authorized, skip temporary hosts too.

- Write the intended filenames, the manifest, the exact `gh release create`

- and `gh release upload` commands, and ready-to-paste marked PR Markdown into

- the artifact folder, and report durable publication as blocked.

- - Compute each source file's SHA-256 before upload. Name each asset with

- stable context plus content identity, for example

- `pr-<number>-<head-short-sha>-<content-sha12>-<step>.png`. Use a branch slug

- when there is no PR number yet. Sanitize names to portable lowercase ASCII.

- - Make reruns idempotent. Inspect the release assets before uploading:

- - If the exact name exists and its GitHub digest matches the local file

- (or, with no digest, a downloaded byte-for-byte hash does), reuse its URL.

- - If the content differs, extend the hash or add a deterministic suffix and

- upload a new asset. Leave the existing asset untouched (no

- `gh release upload --clobber`), so older PRs keep their images.

- - Upload with `gh release upload <tag> <path> -R "$repo"`, then read back the

- release and asset metadata. Require the intended tag, a non-draft release,

- uploaded asset state, the expected filename and size, the SHA-256 digest

- when GitHub supplies one, and `browser_download_url`.

- - Download the uploaded bytes with a direct GET (authenticated through GitHub

- for private repositories); a HEAD request is insufficient. Compare the

- downloaded SHA-256 and size with the local source and check the decoded file

- type or image magic. An HTML login or error page behind a misleading status

- is a failure. Record the verification time and result.

- - Keep `pr-assets-manifest.json` across reruns. For each asset record the

- repository, release tag and URL, PR number, head commit, source path,

- semantic step, asset name, local SHA-256 and size, asset API URL, browser

- download URL, upload-or-reuse status, timestamp, and verification result.

- Keep tokens, cookies, and sensitive test data out of it.

- ### Update the PR description

- The PR description is the primary review surface. Append or replace only the

- section between `` and

- ``, leaving the human-written summary

- as it is. If a legacy `` section

- exists, migrate it once. Keep the section compact:

- - current QA status

- - test account, org, and marker identifiers

- - user journeys and surfaces tested

- - key external evidence IDs, such as Stripe subscription IDs, email IDs,

- PostHog event names, webhook IDs, or database readback

- - key screenshot previews, when UI review is central and the set is small

- enough to skim

- - a link to the detailed QA comment or local artifacts when the evidence is

- long

- - what remains for human review and what was intentionally skipped

- ### Post the detail comment

- When evidence, logs, or screenshot galleries are too large for the

- description, post or update one PR comment whose owned content sits between

- `` and

- ``. Recognize the legacy

- `` marker so reruns update the older comment.

- Include:

- - a summary of the automated manual QA outcome

- - test account, org, and marker identifiers

- - user journeys and surfaces tested

- - screenshot previews

- - connector and provider evidence such as PostHog, Stripe, email, SMS, logs,

- or database readback

- - what remains for human review and what was intentionally skipped

- If PR commenting is not authorized or a connector is unavailable, write the

- exact Markdown comment body into the artifact folder and report the path.

- ### Render screenshots inline

- When UI changed, render safe uploaded screenshots inline so reviewers can skim

- without opening each link. Use grouped preview galleries:

- - one `<details open>` section per user journey or touched UI surface when

- there are many screenshots

- - screenshots in chronological order, each labeled with the journey step and

- the state it proves

- - one sentence per screenshot: what surface and state it shows, and what the

- reviewer should notice

- - a two-column Markdown or HTML table when there are more than four

- screenshots

- - direct image URLs in Markdown image syntax or `<img>` tags; with HTML, keep

- width around `360`-`480` pixels so the PR stays readable

- Keep unsafe screenshots local and say why: payment card entry screens, PHI,

- secrets, private customer data, real inbox contents, MFA codes, or production

- admin data. Give their local paths without rendering or uploading them.

- When updating an existing marked summary or comment, replace dead, expiring,

- temporary, or local-only image references with verified durable URLs and

- inline previews in the same update. Preserve all author-written text outside

- the markers. If an image URL outside a marker is broken, change only that URL,

- after verifying the replacement.

- Example compact preview block:

+ If release creation or upload is not authorized, do not fall back to a temporary host. Write the intended filenames, manifest, exact `gh release create` / `gh release upload` commands, and ready-to-paste marked PR Markdown into the artifact folder; report that durable publication is blocked.

+ - Compute the source SHA-256 before upload. Name every asset with stable context plus content identity, for example `pr-<number>-<head-short-sha>-<content-sha12>-<step>.png`. Use a branch slug when the PR number does not exist yet. Sanitize names to portable lowercase ASCII.

+ - Make reruns idempotent. Inspect release assets before uploading. If the exact name exists and its GitHub digest (or a downloaded byte-for-byte hash when no digest is present) matches the local file, reuse its URL. If the content differs, do not overwrite or use `gh release upload --clobber`; extend the hash or add a deterministic suffix and upload a new asset so an older PR never changes underneath reviewers.

+ - Upload with `gh release upload <tag> <path> -R "$repo"`, then read back the release and asset metadata. Require the intended tag, a non-draft release, uploaded asset state, expected filename, size, SHA-256 digest when GitHub supplies it, and `browser_download_url`.

+ - Perform a direct GET of the uploaded bytes (authenticated through GitHub for private repositories), not only a HEAD request. Compare the downloaded SHA-256 and size with the local source and verify the decoded file type or image magic; an HTML login/error page with a misleading status is a failure. Record the verification timestamp and result.

+ - Maintain `pr-assets-manifest.json` across reruns. For each asset record the repository, release tag and URL, PR number, head commit, source path, semantic step, asset name, local SHA-256 and size, asset API URL, browser download URL, upload-or-reuse status, timestamp, and content-verification result. Never put tokens, cookies, or sensitive test data in the manifest.

+ - Treat the PR description as the primary review surface. Append or replace only the section between `` and `` without rewriting the human-authored PR summary. If a legacy `` section exists, migrate that section once instead of duplicating it. Keep the PR description QA section compact and include:

+ - current QA status;

+ - test account/org/marker identifiers;

+ - user journeys and surface areas tested;

+ - key external evidence IDs, such as Stripe subscription IDs, email IDs, PostHog event names, webhook IDs, or database readback;

+ - key screenshot previews when UI review is central and the set is small enough to skim;

+ - a link to the detailed QA comment or local artifacts when the full evidence is long;

+ - what remains for human review and what was intentionally skipped.

+ - Post or update one PR comment whose owned content is bounded by `` and `` when detailed evidence, logs, or screenshot galleries are too large for the PR description. Recognize the legacy `` marker so reruns update rather than duplicate an older comment. Include:

+ - summary of automated manual QA outcome;

+ - test account/org/marker identifiers;

+ - user journeys and surface areas tested;

+ - screenshot previews, not just screenshot links;

+ - connector/provider evidence such as PostHog, Stripe, email, SMS, logs, or database readback;

+ - what remains for human review and what was intentionally skipped.

+ - Render safe uploaded screenshots inline so reviewers can skim without opening every link. Do not leave the PR description or QA comment as a plain list of screenshot URLs when UI changed.

+ - Prefer grouped preview galleries:

+ - Use one `<details open>` section per user journey or touched UI surface when there are many screenshots.

+ - Put screenshots in chronological order and label each one with the journey step and state it proves.

+ - Add a one-sentence explanation for each screenshot that answers: what surface/state is this, and what should the reviewer notice?

+ - Use a two-column Markdown/HTML table for compact skimming when there are more than four screenshots.

+ - Use direct image URLs in Markdown image syntax or HTML `<img>` tags. If using HTML, constrain width around `360`-`480` pixels so the PR remains readable.

+ - Keep unsafe screenshots local only and say why. Examples: payment card entry screens, PHI, secrets, private customer data, real inbox contents, MFA codes, or production admin data. Mention their local paths without rendering or uploading them.

+ - When updating an existing marked QA summary or comment, replace dead, expiring, temporary, or local-only image references with verified durable URLs and inline previews during the same update instead of adding a second comment. Preserve all author-written text outside the markers. If an image URL outside a marker is broken, change only that URL after verifying the intended replacement; do not rewrite the surrounding prose.

+ - Example compact preview block:

```markdown

<summary>Signup journey screenshots</summary>

| Step | Preview |

| --- | --- |

| Account details | Shows the default account form before submission; reviewer should check required fields and spacing.<br><img src="https://example.test/01-account.png" width="420" alt="Account details form"> |

| Validation error | Shows the blocked submit state; reviewer should check copy, focus, and error placement.<br><img src="https://example.test/02-validation.png" width="420" alt="Validation error state"> |

</details>

```

+ - If PR commenting is not authorized or a connector is unavailable, write the exact Markdown comment body into the artifact folder and report the path.

- ## Fix-verify loop hygiene

+ ## Stop Conditions

- - Before driving a browser proof of a just-committed fix, confirm the served

- bundle contains it: fetch the bundle URL and grep for a distinctive marker,

- or compare the hash in page or script URLs. Dev-server rebuild races cost

- whole proof rounds and look like "the fix didn't work".

- - Wait about 45-60 s before querying an analytics warehouse for just-captured

- events. An empty result inside that window proves nothing.

+ Stop and ask the user before:

- ## Browser extension and vendor interference

+ - Running real production payments or destructive production mutations.

+ - Rebasing, force-pushing, or modifying an open PR if the user has not authorized it.

+ - Posting comments, sending emails, toggling flags, or changing dashboards unless the user asked for that action.

- - Password-manager extensions (1Password) pull focus into extension frames on

- credential-like fields. After that, all automation on the tab fails with

- "Cannot access a chrome-extension:// URL". To avoid and recover:

- - set form values by element reference in preference to click-and-type

- - dismiss popovers by clicking a neutral page area (Escape may feed the

- popover)

- - recover a wedged tab by opening a fresh one (hosted checkout URLs resume

- by URL)

- - Export GIF recordings before closing their tab. Recordings die with the tab

- group.

- - Vendor sandboxes rate-limit. For example, the Dropbox Sign test API

- throttles after about 6 signature requests a day, stalling embeds for about

- 10 minutes. Budget signature-heavy passes and report throttling as an

- environment limit.

+ Otherwise, keep going through setup, execution, verification, cleanup, and a concise result summary.

- ## Stop conditions

+ ## Run Bounds

- Stop and ask the user before:

+ Set a 90-minute wall clock for the entire run. Per-journey, allow at most 2

+ retry attempts before marking the journey failed or blocked. No combination

+ of retries extends the wall clock. If the wall clock expires mid-journey,

+ finalize evidence for what completed, mark in-progress items as

+ `blocked-timeout`, and report.

- - running real production payments or destructive production mutations

- - rebasing, force-pushing, or modifying an open PR without authorization

- - posting comments, sending emails, toggling flags, or changing dashboards

- the user didn't ask for

+ ## Cleanup Discipline

- Otherwise keep going through setup, execution, verification, cleanup, and a

- concise result summary.

+ The run's unique marker names real things that now exist in external systems:

+ accounts, organizations, records, subscriptions, analytics events. Disclosure

+ alone is not disposal.

- ## Run bounds

+ Delete in-run only when all three conditions hold:

- - Set a 90-minute wall clock for the whole run. Retries count against it.

- - Allow at most 2 retries per journey before marking it failed or blocked.

- - If the wall clock expires mid-journey, finalize evidence for what completed,

- mark in-progress items `blocked-timeout`, and report.

+ 1. The surface is **non-production**, established from the environment the drive

+ actually reached (the ingestion target, the key, the project id), not

+ assumed from the stack you launched.

+ 2. The deletion is **scoped by this run's unique marker**, not a broader query.

+ 3. The drive **already holds** the credentials that perform it.

- ## Cleanup

+ If any condition fails or is uncertain, do not delete. Register the item:

+ name the marker, the system, and what remains, precisely enough that a

+ repo-side reaper can find it by marker alone. Production analytics and

+ live-mode billing are register-only by default.

- The run's unique marker names real things that now exist in external systems:

- accounts, organizations, records, subscriptions, analytics events. Each one

- gets a disposition: deleted or registered.

+ Either way the disposition appears in the report's cleanup table. "None

+ created" is a disposition too, not an excuse to omit the table.

- Delete in-run only when all three hold:

+ ## Analytics Identity Verification

- 1. The surface is **non-production**, established from the environment the

- drive actually reached (the ingestion target, the key, the project ID),

- and not assumed from the stack you launched.

- 2. The deletion is **scoped by this run's unique marker** alone.

- 3. The drive **already holds** the credentials that perform it.

+ Event ingestion alone is not enough. Verify PERSON STITCHING whenever a PR touches

+ analytics, signup, login, or session handling:

- Otherwise, or when any condition is uncertain, register the item: name the

- marker, the system, and what remains, precisely enough that a repo-side reaper

- can find it by marker alone. Production analytics and live-mode billing are

- register-only by default.

+ - Group verification queries by `person_id`, never by `person.properties.*`. Event-time

+ person properties differ per row and can make N merged users each look like "one clean

+ person". Six merged QA users passed an email-grouped check; a `person_id`-grouped check

+ exposed they were all a single person.

+ - Inspect raw `distinct_id` per event when stitching looks wrong. It names the exact

+ identity that captured the event and usually identifies the merge vector directly.

+ - When the PR touches identity stitching itself (aliasing, identify calls, distinct-id or

+ session-identity plumbing), or the product targets shared devices, run a

+ **multi-user same-browser pass**: several signups and login switches in one browser

+ profile, then assert each user resolved to a separate person AND that functional session

+ state (websocket auth, cookies) followed the switch. Shared-machine bugs (identity

+ merges, stale-socket auth) are invisible to single-user passes; the pass is expensive,

+ so reserve it for changes where that failure mode is actually in play.

+ - Suspect STACKED causes when a fix's re-verification still fails: fix one vector, re-run

+ the proof, and let the raw distinct_id data name the next vector. Do not assume the fix

+ simply "didn't work".

+ - Test events fired immediately before hard navigations (checkout redirects, external

+ scheduling links): SDK batching silently drops them on unload; they need per-capture

+ `sendBeacon` transport. Absence in the warehouse (not the network tab) is the proof.

- Always include the cleanup table in the report. "None created" is a valid

- disposition.

+ ## Fix-Verify Loop Hygiene

+ - Before driving a browser proof of a just-committed fix, verify the SERVED bundle

+ contains it (fetch the bundle URL and grep a distinctive marker, or compare the hash in

+ page/script URLs). Dev-server rebuild races cost entire proof rounds and mimic "fix

+ didn't work".

+ - Wait ~45-60s before querying an analytics warehouse for just-captured events; an empty

+ result inside that window proves nothing.

+ ## Browser-Extension and Vendor Interference

+ - Password-manager extensions (1Password) steal focus into extension frames on

+ credential-like fields; afterwards ALL automation on the tab fails with

+ "Cannot access a chrome-extension:// URL". Prefer setting form values by element

+ reference over click+type, dismiss popovers by clicking neutral page areas (Escape may

+ feed the popover), and recover a wedged tab only by opening a fresh one (hosted

+ checkout URLs resume by URL).

+ - Export GIF recordings BEFORE closing their tab. Recordings die with the tab group.

+ - Vendor sandboxes rate-limit (e.g. Dropbox Sign test API throttles after ~6 signature

+ requests/day, stalling embeds ~10 min). Budget signature-heavy passes and report

+ throttling as an environment limit, not a product bug.

+ ## Before You Start: Preflight

+ QA evidence is current-head evidence and the Manual tests checklist is the

+ body's contract, so anything that would change either runs first. On a large

+ PR (over 10 files or 300 hand-written lines) that has not had a `refactor`

+ pass, say so and offer it before driving anything. A refactor landed after

+ QA means this whole pass runs again. Likewise `cold-read` on the PR body

+ comes before QA, so the checklist you execute is the one the reader will see.

+ Before spending a long QA pass, verify the PR is in a testable state. Pass

+ the identified PR number or URL to every `gh pr view` call (bare `gh pr view`

+ defaults to the current branch's PR, which may differ from the test target):

+ - **Mergeability.** Check `gh pr view <PR> --json mergeable,mergeStateStatus`.

+ A conflicting PR may get no gating CI run at all, and QA evidence against a

+ conflicting head is evidence against code that will change on merge. If

+ conflicting, report blocked rather than driving.

+ - **Head SHA incorporated.** Confirm the PR's `headRefOid` is part of the

+ tested state (`gh pr view <PR> --json headRefOid`). When testing companion

+ PRs together, the local HEAD may be a merge commit that incorporates

+ multiple PR heads; verify the target PR's head is in the ancestry

+ (`git merge-base --is-ancestor <headRefOid> HEAD`) and record the composite

+ SHA rather than requiring an exact match. A checkout that does not contain

+ the PR head produces evidence for code the reviewer is not looking at.

+ - **CI existence.** Check whether at least one workflow run exists for the

+ PR's head commit (`gh run list --commit <headRefOid>`). If the repo has CI

+ and no run registered, something is wrong (path filters, a conflicting

+ state, a workflow syntax error). Note it; do not assume the code is healthy.

+ - **Tools alive.** Verify every tool the run will need before the first long

+ flow: authenticated CLIs, running services, connectors, test-mode keys.

+ A flow that dies at step 7 for a missing login wastes the entire run.