git:20260505.a05ff22 to git:20260526.af84f57

120 added, 853 removed. Audit A to A.

---

name: shipwright-test

description: "Profile-aware test runner. Runs unit tests, smoke tests, Playwright E2E, and optional security scans.\nTRIGGER when: user wants to run tests, execute test suite, check if tests pass, run unit tests, run E2E tests, run integration tests, verify test results, check design fidelity, visual comparison, compare UI with mockup, verify against design, run visual tests, or fix failing tests.\nDO NOT TRIGGER when: user asks to write new code or implement a section (/shipwright-build), fix a bug by changing code (/shipwright-iterate), deploy (/shipwright-deploy), create requirements (/shipwright-project), plan implementation (/shipwright-plan), or design UI (/shipwright-design)."

license: MIT

compatibility: Requires uv (Python 3.11+). Optional: Playwright.

---

# Shipwright Test Skill

Profile-aware test execution across all test layers.

+ > **How invoked:** directly via `/shipwright-test [--fix|--e2e-only|--design-fidelity|--report-boundary-coverage]`, or by `/shipwright-run` (orchestrator).

+ ## Step Index — Where the prose lives

+ | Section | Reference |

+ |---|---|

+ | First Actions (A. Banner / B. Detect Profile / B2. Mode / B3. Context / B4. Prereqs / C. Strategy) | [first-actions](references/first-actions.md) · [prerequisite-self-healing](references/prerequisite-self-healing.md) |

+ | Step 0: Phase Session Context Recovery | [step-0-phase-session](references/step-0-phase-session.md) |

+ | Step 1: Run Unit Tests | [step-1-unit-tests](references/step-1-unit-tests.md) |

+ | Step 1.5: Run Integration Tests | [step-1.5-integration-tests](references/step-1.5-integration-tests.md) |

+ | Step 1.6: Run pgTAP Database Tests | [step-1.6-pgtap-tests](references/step-1.6-pgtap-tests.md) |

+ | Step 2: Run Smoke Test | [step-2-smoke-test](references/step-2-smoke-test.md) |

+ | Step 2.5: Generate E2E Specs from Plan | [step-2.5-e2e-spec-generation](references/step-2.5-e2e-spec-generation.md) |

+ | Step 3: Run Playwright E2E | [step-3-playwright-e2e](references/step-3-playwright-e2e.md) |

+ | Step 3.5: E2E Results Verification | [step-3.5-e2e-verification](references/step-3.5-e2e-verification.md) |

+ | Step 3.6: Cross-Page UI Consistency Check | [step-3.6-ui-consistency](references/step-3.6-ui-consistency.md) |

+ | Step 3.7: Design Fidelity Verification | [step-3.7-design-fidelity](references/step-3.7-design-fidelity.md) |

+ | Step 3.8: Performance Budget Check | [step-3.8-performance-budget](references/step-3.8-performance-budget.md) |

+ | Step 3.9: Stop Dev Server (always — finally-block) | [step-3.9-stop-dev-server](references/step-3.9-stop-dev-server.md) |

+ | Step 3.95: Boundary Coverage Report | [step-3.95-boundary-coverage](references/step-3.95-boundary-coverage.md) |

+ | Step 4: Security Scan -> /shipwright-security | [step-4-security](references/step-4-security.md) |

+ | Step 5: Report Results | [step-5-report-results](references/step-5-report-results.md) · [results-enforcement](references/results-enforcement.md) · [completion-gate](references/completion-gate.md) |

+ | Anti-Rationalization | [anti-rationalization](references/anti-rationalization.md) |

+ | Test layers strategy (legacy) | [test-layers](references/test-layers.md) |

+ | Reflection | [reflection](references/reflection.md) |

---

## CRITICAL: First Actions

**Governing rules:** Read and follow `shared/constitution.md` (ALWAYS / ASK FIRST / NEVER boundaries).

### A. Print Intro Banner

```

================================================================================

SHIPWRIGHT-TEST: Test Runner

================================================================================

Runs tests across all layers based on stack profile.

Usage: /shipwright-test

or: /shipwright-test --fix (auto-fix failures, max 3 retries)

or: /shipwright-test --e2e-only (only Playwright E2E)

or: /shipwright-test --design-fidelity (only design fidelity check)

or: /shipwright-test --report-boundary-coverage (audit Affected Boundaries declarations across iterate-specs, ADR-027)

or: Invoked by /shipwright-run (orchestrator)

Test layers:

1. Unit tests (Vitest / pytest)

1.5 Integration tests (if profile has testing.integration)

1.6 pgTAP database tests (if supabase/tests/database/ exists)

2. Smoke test (HTTP 200 on DEV URL)

3. Playwright E2E (if UI project + DEV URL available)

3.6 Cross-page consistency (if .shipwright/designs/visual-guidelines.md exists)

3.7 Design fidelity (if .shipwright/designs/screen-routes.json exists)

3.8 Performance budget (if profile has testing.performance.enabled)

3.9 Stop dev server (always, finally-block — runs even if 3.7 / 3.8 failed)

4. Security scan → out-of-band; see /shipwright-security (not auto-invoked)

================================================================================

```

### B. Detect Profile

- Read `shipwright_project_config.json` from project root:

- ```json

- {

- "profile": "supabase-nextjs",

- ...

- }

- ```

- Load profile from `{plugin_root}/../../shared/profiles/{profile}.json`.

- If no config: detect from package.json / pyproject.toml.

+ Read `shipwright_project_config.json` from project root and load profile from

+ `{plugin_root}/../../shared/profiles/{profile}.json`. If no config, detect from

+ `package.json` / `pyproject.toml`. Full detection rules: see

+ [first-actions](references/first-actions.md).

### B2. Detect Invocation Mode

- Determine if running within the pipeline or standalone:

- 1. Read `shipwright_run_config.json` (if exists)

- 2. **Pipeline mode**: `status == "in_progress"` AND `current_step == "test"`

- - Full pipeline integration (update orchestrator state, enforce gates)

- 3. **Standalone mode**: file missing OR `status == "complete"` OR `current_step != "test"`

- - Skip pipeline state updates (no `orchestrator.py update-step` calls)

- - Skip upstream completion checks

- - Still produce all artifacts (`shipwright_test_results.json`, event log)

- - **Mark artifacts**: When writing `shipwright_test_results.json`, add `"mode": "standalone"` at the top level. This tells the pipeline validator to ignore standalone results and require a fresh pipeline test run.

- - Print: `"Running in standalone mode — pipeline state will not be updated."`

- 4. If `status == "in_progress"` AND `current_step != "test"`:

- - Warn: `"Pipeline is in progress at step {current_step}. Running /shipwright-test out of sequence may cause issues."`

- - Ask user before continuing.

- Store the detected mode in a variable `invocation_mode` = `"pipeline"` | `"standalone"` for use in later steps.

+ Pipeline mode (`shipwright_run_config.json.status == "in_progress"` AND

+ `current_step == "test"`) vs standalone. Standalone runs add

+ `"mode": "standalone"` to `shipwright_test_results.json` and skip pipeline

+ state updates. Out-of-sequence pipeline state requires ASK before continuing.

+ Full rules: see [first-actions](references/first-actions.md).

### B3. Load Project Context

- Read these files for app context before running tests:

- 1. `.shipwright/agent_docs/architecture.md` — app structure (understand what to test)

- 2. `shipwright_test_results.json` — previous test state (if exists, for comparison)

- If a file does not exist, skip it silently.

+ Read `.shipwright/agent_docs/architecture.md` and previous

+ `shipwright_test_results.json`. Missing files: skip silently.

### B4. Prerequisite Self-Healing

- Before determining test strategy, check for missing artifacts and auto-generate where possible.

- This follows the constitution rule: **never silently skip a test layer due to missing prerequisites.**

- **1. `dev_url` missing from `shipwright_build_config.json`?**

- - Search `CLAUDE.md` for `PORT=` references or port numbers

- - Search `package.json` scripts for `--port` flags

- - If found: add `"dev_url": "http://localhost:{port}"` to `shipwright_build_config.json`

- - If not found: **ASK** user "What port does your dev server run on?"

- **2. `.shipwright/designs/visual-guidelines.md` missing but `.shipwright/designs/screens/` has HTML files?**

- - Read CSS `:root` variables from the first mockup HTML file

- - Extract color, typography, spacing, radius, and shadow tokens

- - Generate `.shipwright/designs/visual-guidelines.md` using the template format

- - Commit: `chore(test): auto-generate visual-guidelines.md from mockup CSS`

- **3. `.shipwright/designs/screen-routes.json` missing but mockups + router exist?**

- - List mockup HTML files in `.shipwright/designs/screens/`

- - Read router config (`src/router.tsx`, `src/App.tsx`, or framework equivalent)

- - Generate `.shipwright/designs/screen-routes.json` mapping each mockup to its route

- - Commit: `chore(test): auto-generate screen-routes.json from mockups + router`

- **4. `.shipwright/planning/claude-plan-e2e.md` missing but `.shipwright/designs/screen-routes.json` exists?**

- - Generate a minimal E2E test plan with one flow per major screen/route

- - Include page object model suggestions and test data structure

- - Commit: `chore(test): auto-generate E2E test plan from screen routes`

- **5. `playwright.config.ts` missing?**

- - Run `playwright_setup.py` (creates config, installs browser)

- - If `dev_url` was resolved in step 1: substitute the correct port in the config

- **Print diagnostic summary:**

- ```

- PREREQUISITE CHECK:

- dev_url: ✓ http://localhost:3847 (from CLAUDE.md)

- visual-guidelines: ✓ auto-generated from mockup CSS

- screen-routes.json: ✓ auto-generated (11 screens → 5 routes)

- E2E test plan: ✓ auto-generated (6 flows)

- playwright.config: ✓ created with port 3847

- ```

- Only **ASK** the user if auto-generation is not possible (no source data to derive from).

- Each auto-generated artifact gets its own commit.

+ Auto-generate missing `dev_url`, `.shipwright/designs/visual-guidelines.md`,

+ `.shipwright/designs/screen-routes.json`, `.shipwright/planning/claude-plan-e2e.md`,

+ and `playwright.config.ts` where source data exists. Full rules and per-artifact

+ commit hints: see [prerequisite-self-healing](references/prerequisite-self-healing.md).

### C. Determine Test Strategy

Based on profile:

|---------|------------|-----|-------------------|

---

## Step 0: Phase Session Context Recovery

- If your context contains a `=== SHIPWRIGHT-PIPELINE-CONTEXT ===` block (injected

- by the SessionStart hook), you are part of an active `/shipwright-run` pipeline.

- Parse `phaseTaskId` from that block and run as your very first action:

- ```bash

- uv run "${SHIPWRIGHT_PLUGIN_ROOT}/../../shared/scripts/tools/get_phase_context.py" \

- --phase-task-id <phaseTaskId-from-context>

- ```

- The tool prints structured JSON with `runId`, `phase`, `splitId`, `prerequisites`,

- `runConditions`, and a `skill_artifacts_to_read` list. Read those artifacts

- before proceeding so this phase session has full context for what came before.

- If NO `PIPELINE-CONTEXT` block is present, this is a standalone invocation —

- continue with Step 1 below as normal.

- ---

+ If your context contains `=== SHIPWRIGHT-PIPELINE-CONTEXT ===`, parse

+ `phaseTaskId` and run `get_phase_context.py` as the first action.

+ Full body: see [step-0-phase-session](references/step-0-phase-session.md).

## Step 1: Run Unit Tests

```bash

- uv run "{plugin_root}/scripts/lib/test_runner.py" \

- --profile "{profile}" \

- --layer unit

+ uv run "{plugin_root}/scripts/lib/test_runner.py" --profile "{profile}" --layer unit

```

- **Expected output:** Test results with pass/fail counts.

- **Autonomous mode** (check `autonomy` in `shipwright_run_config.json`):

- If tests fail, automatically apply --fix behavior (structured debugging,

- up to 3 retries) without requiring the explicit --fix flag.

- **Guided mode:** Only attempt fixes if `--fix` flag was explicitly passed.

- **Fix behavior** (max 3 retries, structured debugging):

- 1. **Root cause:** Read error output, identify what's failing and why

- 2. **Pattern check:** Same root cause as previous attempt? → Change approach, don't retry same fix

- 3. **Hypothesis:** State what you'll change and why before editing

- 4. Attempt targeted fix based on hypothesis

- 5. Re-run tests

- 6. After 3 retries (or 2 with same root cause): report remaining failures with diagnosis

- ---

+ Autonomous mode auto-applies `--fix` behavior. Max 3 retries, structured

+ debugging (root cause -> pattern check -> hypothesis -> fix -> re-run).

+ Full body: see [step-1-unit-tests](references/step-1-unit-tests.md).

## Step 1.5: Run Integration Tests

- **Skip if:** Profile has no `testing.integration` config, OR `tests/integration/` directory does not exist.

- **Check prerequisites:**

- 1. Read profile `testing.integration` block

- 2. Verify env vars from `.env.test`: `NEXT_PUBLIC_SUPABASE_URL`, `NEXT_PUBLIC_SUPABASE_ANON_KEY`, `SUPABASE_SERVICE_ROLE_KEY`

- 3. Verify URL is localhost/127.0.0.1 (safety check)

- 4. **In CI:** Missing env vars = FAIL (not skip). **Locally:** Missing env vars = skip with warning.

- **Run integration tests:**

- ```bash

- npx vitest run --config vitest.integration.config.ts

- ```

- Or via runner script:

- ```bash

- uv run "{plugin_root}/scripts/lib/test_runner.py" \

- --profile "{profile}" \

- --layer integration \

- --cwd {project_root} \

- --skip-if-missing

- ```

- **Autofix behavior:** Same structured debugging as unit tests (root cause → hypothesis → fix → re-run), max 3 retries.

- **Fast-fail rules:**

- - If error matches `ECONNREFUSED`, `ETIMEDOUT`, `connect ENOENT` → skip autofix, fail immediately with infrastructure diagnosis

- - If >50% of integration tests fail simultaneously → skip autofix, fail with diagnosis (likely global issue, not individual test bugs)

- **Never auto-fix by:**

- - Weakening RLS policies

- - Switching test assertions to use service-role client

- - Disabling URL safety checks

- **Common auto-fixable patterns:**

- | Error Pattern | Diagnosis | Auto-fix |

- |---|---|---|

- | `relation "x" does not exist` | Migration not applied | Run `supabase db push --linked` |

- | `permission denied for table` | RLS policy issue | Check auth context setup in test |

- | `null value in column "x"` | Test data setup incomplete | Fix `beforeAll` / seed data |

- | `duplicate key value` | Previous cleanup failed | Fix `afterAll` cleanup |

- | Auth sign-in failure | Test user not provisioned | Create test user or check credentials |

- **Record results:**

- - `integration_passed`: number of passing tests

- - `integration_total`: total tests

- - `integration_duration_s`: duration in seconds

- - If skipped: `integration_skipped: true`, `integration_skip_reason: "..."`

- ---

+ Skip if no `testing.integration` profile config OR `tests/integration/` missing.

+ CI: missing env vars = FAIL; locally: skip with warning. Fast-fail on

+ `ECONNREFUSED`/`ETIMEDOUT` and on >50% simultaneous failure. Never auto-fix by

+ weakening RLS, swapping to service-role, or disabling URL safety.

+ Full body: see [step-1.5-integration-tests](references/step-1.5-integration-tests.md).

## Step 1.6: Run pgTAP Database Tests

- **Skip if:** `supabase/tests/database/` directory does not exist.

- **Run pgTAP tests:**

- ```bash

- supabase test db

- ```

- Or via runner script:

- ```bash

- uv run "{plugin_root}/scripts/lib/test_runner.py" \

- --profile "{profile}" \

- --layer pgtap \

- --cwd {project_root} \

- --skip-if-missing

- ```

- **Autofix:** Same as integration tests (structured debugging, max 3 retries).

- **Record results:**

- - `pgtap_passed` / `pgtap_total` / `pgtap_duration_s`

- - If skipped: `pgtap_skipped: true`, `pgtap_skip_reason: "no supabase/tests/database/ directory"`

- ---

+ Skip if `supabase/tests/database/` missing. Run via `supabase test db` or the

+ shared `test_runner.py --layer pgtap`. Autofix as for integration.

+ Full body: see [step-1.6-pgtap-tests](references/step-1.6-pgtap-tests.md).

## Step 2: Run Smoke Test (if DEV URL available)

```bash

- uv run "{shared_root}/scripts/smoke_test.py" \

- --url "{dev_url}" \

- --timeout 10 \

- --health-path "/api/health"

+ uv run "{shared_root}/scripts/smoke_test.py" --url "{dev_url}" --timeout 10 --health-path "/api/health"

```

- **DEV URL sources (in order):**

- 1. `shipwright_build_config.json` → `dev_url`

- 2. Environment variable `SHIPWRIGHT_DEV_URL`

- 3. Default: `http://localhost:3000`

- If no DEV URL and app not running: skip with note.

- **Note:** The smoke test script is a **shared plugin script** (`{shared_root}/scripts/smoke_test.py`),

- not a project file. Do NOT search for smoke test files in the project directory.

- **If smoke test fails (any reason) — diagnose before skipping:**

- 1. **Diagnose:** Read error output, identify root cause

- 2. **Attempt autonomous fix** based on diagnosis. Examples:

- - Connection refused → check if dev server is running, start/restart it

- - HTTP error (non-200) → check app logs, report to user (real app bug)

- - Timeout → increase timeout, retry

- - Process hung → kill stale process, restart

- 3. **Retry** smoke test after fix

- 4. After 2 failed fix attempts: **ASK user** how to proceed

- ---

+ DEV URL sources (in order): `shipwright_build_config.json.dev_url` ->

+ `SHIPWRIGHT_DEV_URL` env -> `http://localhost:3000` default.

+ The smoke-test script is shared (`{shared_root}/scripts/smoke_test.py`), not a

+ project file. On failure: diagnose, autonomous fix, retry, then ASK after 2.

+ Full body: see [step-2-smoke-test](references/step-2-smoke-test.md).

## Step 2.5: Generate E2E Specs from Plan (if missing)

- **Goal:** Implement Playwright E2E test specs from the E2E test plans generated by `/shipwright-plan`.

- **Skip this step if:**

- - `e2e/` directory already contains `.spec.ts` files (specs exist from previous run or manual creation)

- - Profile has no UI (backend-only, CLI tool, library)

- **Flow:**

- 1. **Check for existing specs:**

- ```bash

- find e2e/ -name "*.spec.ts" -type f 2>/dev/null | head -1

- ```

- If any `.spec.ts` file found → skip with note: "E2E specs already exist."

- 2. **Find E2E test plans:**

- ```bash

- find .shipwright/planning/ -name "claude-plan-e2e.md" -type f 2>/dev/null

- ```

- If no plans found → skip with note: "No E2E test plans found in .shipwright/planning/."

- 3. **Generate specs from plans:**

- For each `claude-plan-e2e.md` found:

- - Read the plan file (contains user flows, page objects, test data)

- - Generate Playwright test specs following the plan's scenarios

- - Place specs in `e2e/flows/NN-flow-name.spec.ts`

- - Generate Page Object Models in `e2e/pages/*.page.ts`

- - Generate test fixtures/seed data in `e2e/fixtures/test-data.ts`

- - Create `e2e/fixtures/auth.setup.ts` for authenticated test state (if auth flows in plan)

- 4. **Structure:**

- ```

- e2e/

- flows/

- 01-auth.spec.ts # Auth flows (login, signup, logout)

- 02-courses.spec.ts # Course browsing and enrollment

- 03-downloads.spec.ts # Download hub flows

- 04-payments.spec.ts # Purchase and billing flows

- ...

- pages/

- home.page.ts # Page Object: Home page

- course-detail.page.ts # Page Object: Course detail

- login.page.ts # Page Object: Login page

- ...

- fixtures/

- test-data.ts # Seed data for E2E tests

- auth.setup.ts # Playwright auth state setup

- browser-verify.ts # Existing browser verify (unchanged)

- ```

- 5. **Guidelines for spec generation:**

- - Each flow from the plan maps to one `.spec.ts` file

- - Use Playwright `test.describe()` to group related flows

- - Use Page Object Model pattern for element selectors

- - Use `test.beforeAll()` for auth setup where needed

- - Respect `playwright.config.ts` settings (base URL, browser, timeouts)

- - Tests must be runnable against the dev server (localhost)

- - Use `test.skip()` for flows that require external services (e.g., Stripe Checkout redirect)

- - If `.shipwright/designs/visual-guidelines.md` exists, generate basic visual assertion tests in `e2e/flows/00-visual.spec.ts`:

- - Check brand colors on key elements (header, CTA buttons, links)

- - Check font-family on body/headings

- - Check page background color

- **Checkpoint:** `e2e/flows/` contains at least one `.spec.ts` file.

- ---

+ Skip if `e2e/` already has `.spec.ts` files, or profile has no UI. Otherwise

+ scan `.shipwright/planning/` for `claude-plan-e2e.md` and generate

+ `e2e/flows/*.spec.ts`, `e2e/pages/*.page.ts`, `e2e/fixtures/*`, and

+ `e2e/fixtures/auth.setup.ts`. Full body and structure: see

+ [step-2.5-e2e-spec-generation](references/step-2.5-e2e-spec-generation.md).

## Step 3: Run Playwright E2E (if applicable)

- **Prerequisites:**

- - Profile has UI (e.g., `supabase-nextjs`)

- - DEV URL is accessible (smoke test passed)

- **Flow:**

- 1. **Ensure Playwright is set up:**

- ```bash

- uv run "{plugin_root}/../../shared/scripts/playwright_setup.py" --cwd {project_root}

- ```

- 2. **Ensure dev server is running:**

- ```bash

- uv run "{plugin_root}/../../shared/scripts/dev_server.py" start --profile {profile} --cwd {project_root}

- ```

- 3. **Run E2E tests:**

- ```bash

- uv run "{plugin_root}/scripts/lib/playwright_runner.py" --cwd {project_root}

- ```

- 4. **Evaluate results:**

- - Parse the JSON output for pass/fail/skip counts

- - If all passed: continue to Step 4

- - If failures and `--fix` flag active: invoke auto-fix

- 5. **Auto-fix mode** (autonomous mode or `--fix` flag):

- **Group failures by root cause** before fixing:

- - Auth/login failures (e.g., loginAs timeout, missing credentials)

- - Selector mismatches (e.g., getByRole not found, wrong label)

- - Missing data (e.g., no test courses, empty tables)

- - Connection/timeout issues

- - Other

- **For each root-cause group:**

- a. Diagnose the group's common root cause

- b. If screenshots exist for failed tests, read them for visual context

- c. Apply a single fix addressing the group (e.g., fix loginAs helper → fixes all auth tests)

- d. Re-run only the tests in this group: `npx playwright test --grep "{pattern}"`

- e. If fix works: **commit the fix**, move to next group

- f. If same root cause persists after 3 attempts on this group: **park it**, move to next group

- **After all groups attempted:**

- - Report summary: which groups fixed, which parked, with diagnosis per parked group

- - **ASK user** for direction on parked groups (one consolidated dialog, not per group)

- **Commit between fix rounds** — each successful group fix gets its own commit

- to prevent losing progress on session interruption.

- 6. Dev server remains running for Design Fidelity (Step 3.7). Stopped there.

- If no Playwright config exists and setup fails: skip with note.

- **If E2E tests fail — diagnose before reporting:**

- 1. **Diagnose:** Read error output, identify root cause pattern across failures

- 2. **Attempt autonomous fix** based on diagnosis. Examples:

- - Auth/login failures → run auth setup to seed test users (`npx playwright test --project=setup`)

- - Missing page elements → verify pages render, check selectors against actual DOM

- - Connection issues → verify dev server is running, check base URL config

- - Timeout → increase test timeout, check for slow-loading pages

- - Missing test data → check fixtures, seed required data

- 3. **Re-run failed tests only:** `npx playwright test --grep "{test_title}"`

- 4. After 2 failed fix attempts with same root cause: **ASK user** for direction

- ---

+ Prereqs: UI profile + smoke test passed. Run `playwright_setup.py`,

+ `dev_server.py start`, then `playwright_runner.py --cwd {project_root}`.

+ Auto-fix groups failures by root cause (auth / selector / data / network /

+ other), fixes per group with commits, parks unresolvable groups after 3

+ attempts and ASKs the user.

+ Full body: see [step-3-playwright-e2e](references/step-3-playwright-e2e.md).

## Step 3.5: E2E Results Verification

- After E2E tests complete, verify consistency across all result sources. This catches

- miscounts from setup projects, retries, or skipped tests.

- 1. Read `e2e-results.json` (Playwright's authoritative output):

- - Parse `stats.expected`, `stats.unexpected`, `stats.skipped`, `stats.flaky`

- - Filter: only count tests from the `chromium` project (exclude `setup` project tests)

- 2. Compare against `shipwright_test_results.json` e2e counts:

- - `e2e.total` should match Playwright's `expected + unexpected + skipped` (chromium only)

- - `e2e.passed` should match `expected`

- 3. **If numbers diverge:**

- - Analyze WHY: setup project counted? Retries inflating counts? Skipped tests miscounted?

- - Fix `shipwright_test_results.json` to match Playwright's authoritative numbers

- - Add `"e2e_verification_note"` field documenting the discrepancy and resolution

- 4. **If numbers match:** proceed silently

- 5. Verify `playwright-report/index.html` exists — note path for compliance linking

- ---

+ Read `e2e-results.json` (Playwright authoritative). Filter to the `chromium`

+ project (exclude `setup`). Reconcile `shipwright_test_results.json.e2e` with

+ Playwright's `expected + unexpected + skipped`, write an

+ `e2e_verification_note` if divergent, and verify `playwright-report/index.html`.

+ Full body: see [step-3.5-e2e-verification](references/step-3.5-e2e-verification.md).

## Step 3.6: Cross-Page UI Consistency Check (if applicable)

- **Condition:** Runs if `.shipwright/designs/visual-guidelines.md` exists AND profile has UI config (`component_library` set). Also runs standalone via `--consistency` flag or alongside `--visual`.

- **Purpose:** Detect cross-page UI inconsistencies that per-page mockup comparison cannot catch (e.g., mixed heading sizes, inconsistent spacing, different table wrappers across pages). Non-blocking (WARNING level).

- **1. Run consistency analysis:**

- ```bash

- uv run "{plugin_root}/scripts/lib/ui_consistency_check.py" \

- --cwd "{project_root}" \

- --guidelines ".shipwright/designs/visual-guidelines.md"

- ```

- Parse JSON output: `passed`, `total`, `skipped`, `categories`, `root_cause_groups`.

- **2. If all categories CONSISTENT:** Log result, proceed to Step 3.7.

- **3. If INCONSISTENT categories found:**

- Print outlier summary grouped by root cause (same taxonomy as design fidelity):

- | Root Cause | Categories | Fix Scope |

- |------------|-----------|-----------|

- | **Spacing** | heading_hierarchy, spacing_patterns | Tailwind classes on page headings/containers |

- | **Components** | component_patterns, form_patterns, interactive_patterns | Component imports, wrapper replacements |

- | **Colors** | token_usage | Replace hardcoded colors with semantic tokens |

- **Fix loop per root-cause group** (max 3 retries per group):

- a. Read outlier details (file, line, found vs. expected)

- b. Apply targeted fix (e.g., change `text-3xl` → `text-2xl`)

- c. Re-run consistency check for that category: `--category {name}`

- d. If fix works: commit with `fix(ui-consistency): normalize {category} across pages`

- e. If same issue persists after 3 attempts: park with diagnosis

- **4. Record results** in `shipwright_test_results.json`:

- ```json

- {

- "consistency": {

- "passed": 4,

- "total": 6,

- "skipped": false,

- "skip_reason": "",

- "categories": { ... },

- "root_cause_groups": { ... }

- }

- ```

- **5. Non-blocking:** Consistency issues produce WARNINGs, never hard-fail the pipeline.

- ---

+ Runs if `.shipwright/designs/visual-guidelines.md` exists AND profile has UI

+ (`component_library` set), or standalone via `--consistency`. Non-blocking

+ (WARNING). Outliers are grouped by Spacing / Components / Colors root cause and

+ fixed with per-group commits, max 3 retries per category.

+ Full body: see [step-3.6-ui-consistency](references/step-3.6-ui-consistency.md).

## Step 3.7: Design Fidelity Verification — Regressions-Only (if applicable)

- **Condition:** Runs if `.shipwright/designs/screen-routes.json` exists in the project root. Also runs standalone via `--design-fidelity` flag.

- **Purpose:** Detect and fix design fidelity regressions introduced by later build sections (cross-section side effects). Build handles the bulk of fidelity checks per-section; Test is the safety net. Non-blocking (WARNING level) — fidelity differences don't fail the pipeline.

- **1. Run structural extraction:**

- ```bash

- uv run "{plugin_root}/scripts/lib/design_fidelity_check.py" \

- --cwd "{project_root}"

- ```

- Parse the JSON output:

- - `skipped: true` → no screen-routes.json, skip to step 6

- - All screens `status: "pass"` → PASS, skip to step 5

- - Some screens `status: "needs_review"` → proceed to triage

- **2. Read build fidelity results:**

- Read `design-fidelity-report.json` from the project root. Build a screen-status map: `{screen: status}`.

- - If file is missing or fails to parse → **fallback**: treat all screens as unchecked (full analysis, backward-compatible).

- - If `build_complete: false` → log WARNING ("Build may still be in progress"), proceed with triage anyway.

- **3. Triage against build results:**

- For each screen with `status: "needs_review"`, determine its category using `design-fidelity-report.json`:

- |----------|-----------|----------|--------|

- **4. Agent deep review (for flagged screens):**

- For each flagged screen:

- a. Read the mockup HTML source at `{mockup_path}`

- b. Read the implementation TSX source at `{implementation_files[0]}`

- c. Compare against 5 dimensions: Layout Structure, Component Order, Component Types, Card Patterns, shadcn Rules

- d. If mismatches found: fix implementation, run unit tests, commit: `fix(test-fidelity): {description}`

- e. Re-run `design_fidelity_check.py --screen {screen}` to verify fix

- f. Max 3 retries per screen; if unresolvable: park with diagnosis

- **After all screens attempted:**

- - Report summary: which screens fixed, which parked, with diagnosis per parked screen

- - **ASK user** for direction on parked screens (one consolidated dialog)

- - **Commit between fix rounds** — each fix gets its own commit

- **5. Record results** in `shipwright_test_results.json`:

- ```json

- {

- "design_fidelity": {

- "passed": N,

- "total": N,

- "skipped": false,

- "screens": [

- {"mockup": "01-login.html", "route": "/login", "status": "pass"},

- {"mockup": "08-dashboard.html", "route": "/dashboard", "status": "needs_review"}

- ],

- "triage": {

- "regressions": 1,

- "persistent_failures": 0,

- "unchecked": 0,

- "resolved": 2

- }

- ```

- **6. Continue to Step 3.8 (Performance Budget) without stopping dev server.**

- The dev server is stopped in Step 3.9 below as a `finally` step (runs even if

- 3.7 / 3.8 fail), so Lighthouse in 3.8 can audit the live URL.

- ---

+ Runs if `.shipwright/designs/screen-routes.json` exists, or standalone via

+ `--design-fidelity`. Non-blocking (WARNING). Triages `needs_review` screens

+ against `design-fidelity-report.json` into Resolved / Regression / Persistent

+ Failure / Unchecked. Deep-reviews flagged screens against 5 dimensions, max

+ 3 retries each.

+ Full body: see [step-3.7-design-fidelity](references/step-3.7-design-fidelity.md).

## Step 3.8: Performance Budget Check (if applicable)

- **Condition:** Runs if the profile has `testing.performance.enabled: true`.

- The check is also activated when the project's `shipwright_test_config.json`

- sets `performance.enabled: true`. If neither opts in, the runner emits a

- structured `skipped: profile opts out` result (no silent omission — the

- result still appears in `shipwright_test_results.json.performance` with

- `skipped: true`).

- **Note for the bundle sub-check:** the bundle measurement requires a prior

- production build (`npm run build` or equivalent — the artifact directory

- declared in `profile.testing.performance.bundle.build_output_dir`). If no

- build has been produced, the bundle sub-check is `skipped` with reason —

- the Lighthouse sub-check still runs.

- **Tooling:** Lighthouse runs via the project's existing Playwright Chromium

- through a small Node script (`{plugin_root}/scripts/perf/lighthouse-runner.mjs`).

- No extra browser install required. The plugin pins `lighthouse@13.x` in

- `{plugin_root}/scripts/perf/package.json`; the Python wrapper lazy-installs

- that dep on first invocation.

- **1. Run the performance check:**

- ```bash

- uv run "{plugin_root}/scripts/lib/performance_check.py" \

- --cwd "{project_root}" \

- --profile-path "{shared_root}/profiles/{profile}.json" \

- --dev-url "{dev_url}" \

- --gate inherit

- ```

- `--gate inherit` (default) honors the profile / `shipwright_test_config.json`

- value (`warn` or `block`). Pass `--gate warn` or `--gate block` to override

- for a one-off run.

- **2. Parse the JSON output** — single object on stdout matching:

- ```json

- {

- "success": true,

- "skipped": false,

- "gate": "warn",

- "lighthouse": {

- "ran": true, "skipped": false,

- "score": 92, "score_budget": 85, "score_passed": true,

- "lcp_ms": 1840, "lcp_budget_ms": 2500, "lcp_passed": true

- },

- "bundle": {

- "ran": true, "skipped": false,

- "total_kb_gz": 187.4, "budget_kb_gz": 250, "passed": true,

- "files_measured": 14,

- "measurement_semantic": "gzipped *.js + *.css payload — excludes assets, source-maps, precompressed siblings"

- },

- "duration_seconds": 12.7

- }

- ```

- **Bundle measurement semantic:** the `total_kb_gz` figure is the gzip-compressed

- size of `*.js` + `*.css` files in `build_output_dir` (excludes source maps,

- precompressed siblings). It is the **JavaScript + CSS payload only**, not the

- full transfer size of the page (fonts, images, etc., are not counted).

- Static-asset budgets are out of scope for v1.

- **3. Record results** in `shipwright_test_results.json`:

- ```json

- {

- "performance": {

- "ran": true,

- "passed_layers": 2,

- "total_layers": 2,

- "gate": "warn",

- "lighthouse": { ... mirrors stdout ... },

- "bundle": { ... mirrors stdout ... },

- "skipped": false,

- "skip_reason": ""

- }

- ```

- **4. Gate behavior:**

- - `gate: "warn"` (default): result `success: true` even on failures.

- Failures are logged but do NOT fail the test phase.

- - `gate: "block"` (opt-in): non-skipped layer failures set `success: false`

- and exit code 1. Test phase fails.

- **5. Skip semantics** (these all return `success: true` and are valid

- completion states):

- - `profile opts out` — `testing.performance.enabled: false`

- - `no dev_url available` — Lighthouse can't run; bundle still measured

- - `no build artifacts found` — bundle skipped; Lighthouse still runs

- - `no bundle assets matched` — `build_output_dir` exists but no `*.js`/`*.css`

- - `lighthouse_unavailable` — Node/Playwright/Chromium not reachable

- **Override mechanism (precedence, most-specific wins):**

- 1. `--gate` CLI flag

- 2. `shipwright_test_config.json` → `performance.{lighthouse,bundle,gate}` (deep-merged)

- 3. Profile `testing.performance.{...}`

- 4. Built-in defaults (Lighthouse 85, bundle 250 KB gz, LCP 2500 ms, gate `warn`)

- The deep-merge means a project that sets only

- `shipwright_test_config.json.performance.lighthouse.min_score: 95` inherits

- `lcp_max_ms` from the profile.

- ---

+ Runs if `profile.testing.performance.enabled: true` OR

+ `shipwright_test_config.json.performance.enabled: true`. Lighthouse via

+ Playwright Chromium; bundle measurement = gzip-compressed `*.js`/`*.css` in

+ `build_output_dir`. Gate `warn` (default) vs `block` (opt-in).

+ Override precedence: `--gate` -> `shipwright_test_config.json.performance` ->

+ profile -> defaults (LH 85, bundle 250 KB gz, LCP 2500 ms, gate warn).

+ Full body: see [step-3.8-performance-budget](references/step-3.8-performance-budget.md).

## Step 3.9: Stop Dev Server (always — finally-block semantics)

**Run unconditionally as a cleanup pass — even if Step 3.7 or Step 3.8

raised or failed.** A blocked test phase is recoverable; a zombie dev server

- bound to the dev port is not. Practitioners executing SKILL.md by hand and

- CI runners parsing the step list MUST treat this section as a `finally`

- clause analogous to a shell `trap`.

+ bound to the dev port is not.

```bash

uv run "{plugin_root}/../../shared/scripts/dev_server.py" stop --cwd {project_root}

```

- ---

+ Full body: see [step-3.9-stop-dev-server](references/step-3.9-stop-dev-server.md).

## Step 3.95: Boundary Coverage Report (if --report-boundary-coverage)

- **Condition:** Runs only when invoked with `--report-boundary-coverage`,

- OR when `shipwright_test_config.json` sets `boundary_coverage.enabled: true`.

- **Purpose:** Audit hook for Sub-Iterate A's `touches_io_boundary`

- discipline (ADR-024). Scans `.shipwright/planning/iterate/**/*.md`

- for `## Affected Boundaries` sections, correlates each spec with

- commits in `shipwright_events.jsonl`, and emits a coverage report:

- - **Coverage Summary**: boundaries touched / round-trip-tested / probed

- - **Per-iterate breakdown**: one row per iterate spec

- - **Drift Signals**: iterates whose commits or spec text touched

- IO-boundary files but did NOT declare an `## Affected Boundaries`

- section — the audit hook for catching skipped boundary declarations

- **1. Run the report:**

- ```bash

- uv run "{plugin_root}/scripts/tools/boundary_coverage_report.py" \

- --project-root "{project_root}" \

- --output-markdown ".shipwright/test-reports/boundary-coverage-{YYYY-MM-DD}.md" \

- --output-json ".shipwright/test-reports/boundary-coverage-{YYYY-MM-DD}.json"

- ```

- **2. Merge into `shipwright_test_results.json`:**

- After the report runs, embed the JSON output under

- `shipwright_test_results.json#boundary_coverage_report` so downstream

- compliance tooling can ingest it without re-scanning. Two equivalent

- ways to do this (E spec HIGH-4 — pick one):

- **Option A — single command** (preferred when you don't need a

- standalone JSON artefact):

- ```bash

- uv run "{plugin_root}/scripts/tools/boundary_coverage_report.py" \

- --project-root "{project_root}" \

- --output-markdown ".shipwright/test-reports/boundary-coverage-{YYYY-MM-DD}.md" \

- --merge-into "{project_root}/shipwright_test_results.json"

- ```

- **Option B — two-step flow** (when you also want the standalone

- `.shipwright/test-reports/*.json` artefact alongside the merge):

- ```bash

- # (Step 1 above already wrote the standalone JSON.)

- uv run "{plugin_root}/scripts/tools/merge_boundary_coverage.py" \

- --input ".shipwright/test-reports/boundary-coverage-{YYYY-MM-DD}.json" \

- --target "{project_root}/shipwright_test_results.json"

- ```

- Both write atomically (`tmp.replace(target)`) and preserve any other

- top-level keys already present in the target.

- Resulting shape:

- ```json

- {

- "boundary_coverage_report": {

- "summary": {

- "specs_scanned": 47,

- "specs_with_boundaries": 12,

- "total_boundaries": 31,

- "round_trip_tested": 18,

- "round_trip_unknown": 2,

- "drift_signals": 3

- },

- "rows": [...]

- }

- ```

- The `round_trip_unknown` counter (E spec HIGH-5) tracks boundaries

- where the matched commit's events lacked `changed_files`, so the

- heuristic could not distinguish "no test added" from "we don't know".

- **3. Non-blocking:** Drift signals produce WARNINGs in the test summary,

- never hard-fail the pipeline. The drift signals are a discipline

- indicator, not a correctness gate.

- See `tools/boundary_coverage_report.py` source for parser semantics,

- including the 8-probe coverage on the markdown table parser

- (BOM, CRLF, non-ASCII, etc.) and the round-trip test heuristic.

- ---

+ Runs only when invoked with `--report-boundary-coverage` OR

+ `shipwright_test_config.json.boundary_coverage.enabled: true`. Audit hook for

+ ADR-024. Scans `.shipwright/planning/iterate/**/*.md` for `## Affected

+ Boundaries`, correlates against `shipwright_events.jsonl`, writes coverage

+ summary + drift signals. Non-blocking (WARNINGs in test summary).

+ Full body: see [step-3.95-boundary-coverage](references/step-3.95-boundary-coverage.md).

## Step 4: Security Scan → /shipwright-security

- Security scanning is handled by the dedicated `/shipwright-security` plugin and runs **out-of-band** — `/shipwright-run` does NOT auto-invoke it (decoupled in iterate `sec-report-and-orchestrator-decouple`, 2026-04). The previous `AIKIDO_CLIENT_ID`-gated auto-insert is gone.

- - Standalone: `/shipwright-security` (manual, typically after test)

- - CI: `.github/workflows/security.yml` (dormant by default; activate `pull_request` / `schedule` triggers at Phase B)

+ Security scanning runs **out-of-band**; `/shipwright-run` does NOT auto-invoke

+ it (decoupled in iterate `sec-report-and-orchestrator-decouple`, 2026-04).

This step is a no-op in shipwright-test.

- ---

+ Full body: see [step-4-security](references/step-4-security.md).

## Step 5: Report Results

- **Print Summary:**

- ```

- ================================================================================

- SHIPWRIGHT-TEST RESULTS

- ================================================================================

- Unit tests: {passed}/{total} passed ({duration}s)

- Integration: {passed}/{total} passed ({duration}s) | SKIP: {reason}

- pgTAP: {passed}/{total} passed ({duration}s) | SKIP: {reason}

- Smoke test: {PASS | FAIL | SKIP} ({url}, {response_time}ms)

- E2E tests: {passed}/{total} passed | SKIP

- Consistency: {passed}/{total} categories consistent | SKIP

- Design fidelity: {passed}/{total} checked | SKIP

- Performance: LH {score}/100 (budget {budget}), bundle {size}KB (budget {budget}KB), gate {warn|block} | SKIP: {reason}

- Security: {via /shipwright-security | not run}

- Overall: {PASS | FAIL}

- {Failed tests: list if any}

- ================================================================================

- ```

- **If profile has UI** (component_library set, or client-side framework detected):

- ```

- ================================================================================

- Verify visually: /shipwright-preview

- Preview URL: {dev_url from shipwright_build_config.json}

- ================================================================================

- ```

- If `--fix` was used:

- ```

- Auto-fix attempts: {N}

- Fixed: {list of fixed tests}

- Remaining failures: {list}

- ```

- ### Results Enforcement

- Test results determine pipeline continuation:

- | Layer | On FAIL | Rationale |

- |-------|---------|-----------|

- | **Unit tests** | **Pipeline stops** (blocking) | Unit tests are deterministic — failure = real bug |

- | **Integration tests** | Autofix (3 retries, fast-fail for infra), then blocking | Deterministic against real DB |

- | **pgTAP tests** | Autofix (3 retries), then blocking | Schema-level verification |

- | **Smoke test** | **Pipeline stops** (blocking) | App not running = can't deploy |

- | **E2E tests** | **Warning only** (non-blocking) | E2E can be flaky; log failures but continue |

- | **Consistency** | **Warning only** (non-blocking) | Cross-page cosmetic issues don't gate deployment |

- | **Design fidelity** | **Warning only** (non-blocking) | Fidelity divergence ≠ broken functionality |

- | **Performance** | **Warn (default)** / **Block (opt-in via profile or test_config)** | Discipline gate: `warn` ships honest signal without breaking flow; `block` once project has calibrated budgets |

- If unit tests, integration tests, pgTAP tests, or smoke test FAIL: set phase status to `FAIL` and inform user. Do NOT proceed to deploy.

- ### Completion Gate

- Before marking the test phase complete, ALL test layers must have an explicit result:

- | Layer | Required Result |

- |-------|----------------|

- | Unit tests | `pass` or `fail` (always required) |

- | Integration tests | `pass`, `fail`, or `skipped: {reason}` |

- | pgTAP tests | `pass`, `fail`, or `skipped: {reason}` |

- | Smoke test | `pass`, `fail`, or `skipped: {reason}` |

- | E2E tests | `pass`, `fail`, or `skipped: {reason}` |

- | Consistency | `pass`, `warning`, or `skipped: {reason}` |

- | Design fidelity | `pass`, `fail`, or `skipped: {reason}` |

- | Performance | `pass`, `warning` (gate=warn), `fail` (gate=block), or `skipped: {reason}` |

- If any layer has NO result (was never executed and has no skip reason):

- - **Do NOT mark test phase as complete**

- - Print warning: "Test layer {layer} has no result. Run it or document skip reason."

- - Set phase status to `incomplete`

- Valid skip reasons:

- - `skipped: no testing.integration config in profile` (Integration)

- - `skipped: tests/integration/ directory does not exist` (Integration)

- - `skipped: missing integration test env vars` (Integration, local only)

- - `skipped: no supabase/tests/database/ directory` (pgTAP)

- - `skipped: no DEV URL available` (Smoke + E2E)

- - `skipped: no Playwright config` (E2E)

- - `skipped: profile has no UI` (E2E)

- - `skipped: smoke test failed` (E2E, because prerequisite not met)

- - `skipped: no .shipwright/designs/visual-guidelines.md` (Consistency)

- - `skipped: profile has no UI` (Consistency)

- - `skipped: no screen-routes.json` (Design fidelity)

- - `skipped: profile opts out (testing.performance.enabled is false)` (Performance)

- - `skipped: no dev_url available` (Performance — Lighthouse sub-check only)

- - `skipped: no build artifacts found` (Performance — bundle sub-check only)

- - `skipped: no bundle assets matched` (Performance — bundle sub-check, build_output_dir empty of *.js / *.css)

- - `skipped: lighthouse_unavailable` (Performance — Node/Playwright/Chromium failure, see captured stderr)

- **Reflection — Capture Test Learnings** (before marking phase complete):

- If test failures required investigation or fixes:

- 1. Flaky test patterns worth documenting?

- 2. Infrastructure quirks (timing, ports, browser drivers)?

- 3. Test strategy insights (missing coverage, better approaches)?

- If learnings exist:

- - **Observations** → append to `.shipwright/agent_docs/conventions.md` under `## Learnings`

- Format: `- ({YYYY-MM-DD}) test — {summary}`

- - **Cross-project insights** → save Claude Code feedback/project Memory

- If none: skip.

- **Record test_run event** (always, even on failure — captures layer results):

- ```bash

- uv run "{shared_root}/scripts/tools/record_event.py" \

- --project-root "$(pwd)" \

- --type test_run \

- --trigger "pipeline" \

- --unit-passed {unit_passed} \

- --unit-total {unit_total} \

- --e2e-passed {e2e_passed} \

- --e2e-total {e2e_total} \

- --smoke-status "{pass|fail|skip}"

- ```

- Where `{shared_root}` = `{plugin_root}/../../shared`.

- Omit `--e2e-passed`/`--e2e-total` if E2E was skipped. Omit `--smoke-status` if smoke was skipped.

- Use `--trigger "iterate"` when invoked by `/shipwright-iterate`, `"manual"` when invoked standalone.

- **Phase complete — update pipeline state** (only if Completion Gate passes):

- Iterate 12.4 wires the test plugin into the Minimum Phase Completion

- Canon at C1/C2/C3 only. **C4 is skipped by policy** — test runs are

- events, not architectural decisions (both LLM reviewers flagged this

- as CRITICAL). **C5 is also skipped** — test results live in

- `shipwright_test_results.json`, not CHANGELOG.

- ```bash

- # Derive a run id if the orchestrator didn't set one.

- : "${SHIPWRIGHT_RUN_ID:=test-$(date +%Y%m%d-%H%M%S)}"

- export SHIPWRIGHT_RUN_ID

- # C1 — test_run event already recorded above.

- # (The event-type is `test_run`, not `phase_completed`, but also emit

- # a phase_completed event so the generic C1 verifier matches uniformly.)

- uv run "{shared_root}/scripts/tools/record_event.py" \

- --project-root "$(pwd)" --type phase_completed --phase test \

- --detail "{unit_passed}/{unit_total} unit, {e2e_passed}/{e2e_total} e2e"

- # C2 — delivery dashboard

- uv run "{shared_root}/scripts/tools/update_build_dashboard.py" \

- --project-root "$(pwd)" --phase test --detail "{passed}/{total} passing" \

- --session-id "{SHIPWRIGHT_SESSION_ID}"

- # C3 (NEW 12.4) — canon-marker handoff

- uv run "{shared_root}/scripts/tools/generate_session_handoff.py" \

- --project-root "$(pwd)" --canon-marker --phase test \

- --reason "test complete: {unit_passed}/{unit_total} unit, {e2e_passed}/{e2e_total} e2e, smoke {smoke_status}"

- # C4 — SKIPPED by policy (test is not a decision-taking phase).

- # C5 — SKIPPED by policy (test results belong in shipwright_test_results.json,

- # not CHANGELOG).

- # phase_history (NEW 12.4) — audit trail

- uv run "{shared_root}/scripts/tools/append_phase_history.py" \

- --project-root "$(pwd)" --phase test --run-id "$SHIPWRIGHT_RUN_ID" \

- --entry-json '{"unit":"{unit_passed}/{unit_total}","e2e":"{e2e_passed}/{e2e_total}","smoke":"{smoke_status}","outcome":"passed"}'

+ Print the SHIPWRIGHT-TEST RESULTS summary (unit / integration / pgTAP / smoke /

+ e2e / consistency / design fidelity / performance / security / overall) and the

+ preview banner when the profile has UI. Capture learnings into

+ `conventions.md#Learnings`. Record `test_run` + `phase_completed` events,

+ update build dashboard, write canon-marker handoff, append `phase_history`,

+ then mark the test step complete via the orchestrator.

- # Mark test phase complete (triggers compliance update automatically).

- # _validate_test() now runs the modular test_checks verifier (canon

- # C1/C2/C3 + phase_history) in addition to the existing results-layer

- # completion gate.

- uv run "{plugin_root}/../../plugins/shipwright-run/scripts/lib/orchestrator.py" \

- update-step --project-root "$(pwd)" --step test --status complete

- ```

+ Per-layer enforcement: see [results-enforcement](references/results-enforcement.md).

+ Completion gate (every layer must have an explicit result): see

+ [completion-gate](references/completion-gate.md).

+ Full body: see [step-5-report-results](references/step-5-report-results.md).

---

## Anti-Rationalization

- Before accepting test results as sufficient, resist these justifications:

- | Rationalization | Reality |

- |---|---|

- | "Tests pass, so the code is correct" | Tests are necessary but insufficient — they can't catch what they don't cover |

- | "E2E tests are flaky, ignore failures" | Flaky tests hide real regressions. Fix the flakiness or investigate each failure |

- | "We'll add more tests later" | Test debt compounds faster than code debt. Cover it now |

- | "Manual testing is enough" | Manual testing doesn't run on every commit. Automated tests do |

- | "100% coverage means no bugs" | Coverage measures execution, not correctness. A test that asserts nothing has coverage |

- ---

- ## Reference Documents

- - [test-layers.md](references/test-layers.md) — Test layer overview and strategy

+ See [anti-rationalization](references/anti-rationalization.md) — the table of

+ common rationalizations ("Tests pass = correct", "E2E flaky = ignore",

+ "We'll add more later", etc.) and the reality check for each.