integration-e2e-testing · diff
git:20260728.312fef9 to git:20260802.aa3bfed
48 added, 166 removed. Audit A to A.
---
name: integration-e2e-testing
- description: "Integration and E2E test design principles, value-based selection, test skeleton specification, and review criteria. Use when: designing integration tests, E2E tests, generating test skeletons, or reviewing test quality."
+ description: "Selects and specifies only integration/E2E tests whose observable boundary cannot be proven more cheaply."
---
- # Integration and E2E Testing Principles
-
- ## References
-
- **E2E test design**: See [references/e2e-design.md](references/e2e-design.md) for UI Spec-driven E2E test candidate selection and browser test architecture. Playwright is the default browser harness example; use the project's standard when different.
-
- ## Test Type Definition and Limits [MANDATORY]
-
- | Test Type | Purpose | Scope | External Deps | Limit per Feature | Implementation Timing |
- |-----------|---------|-------|---------------|-------------------|----------------------|
- | Integration | Verify component interactions in-process | Partial system integration | Project-local dependencies | MAX 3 | Created alongside implementation |
- | fixture-e2e | Verify browser/user journey with controlled state | Browser UI + mocked backend or fixtures | No live stack required | MAX 3 | Created alongside UI implementation |
- | service-integration-e2e | Verify live-stack cross-service correctness | Full local stack | Local services, DB, queues, stubs | MAX 1-2 | Executed in final phase only |
-
- **ENFORCEMENT**: Exceeding test limits requires explicit justification
-
- ## Behavior-First Principle [MANDATORY]
-
- ### MUST Include (High Value)
- - Business logic correctness (calculations, state transitions, data transformations)
- - Data integrity and persistence behavior
- - User-visible functionality completeness
- - Error handling behavior (what user sees/experiences)
-
- ### MUST Exclude (Low Value in CI/CD)
- - External service real connections — use contract/interface verification instead
- - Performance metrics — non-deterministic, defer to load testing
- - Implementation details — test observable behavior only
- - UI layout specifics — test information availability instead
-
- **ENFORCEMENT**: Test = User-observable behavior verifiable in isolated CI environment
-
- ## Value and Selection Model
-
- ```
- Value Score = (Business Value x User Frequency) + (Legal Requirement x 10) + Defect Detection
- ```
-
- Score each factor as follows:
-
- - **Business Value**: `0` for no material outcome impact, `5` for a material secondary outcome, and `10` for a critical or core outcome; interpolate `1-4` and `6-9`
- - **User Frequency**: `0` below 1% of eligible journeys, `1-9` for the corresponding 10%-90% band, and `10` for virtually every eligible journey
- - **Legal Requirement**: `1` when legally required, otherwise `0`
- - **Defect Detection**: `0` when existing tests already prove the same failure mode, `5` for a material coverage gap, and `10` when this is the primary detector of a critical regression; interpolate `1-4` and `6-9`
-
- Use `Value Score` for ranking candidates of the same test type. Handle E2E cost through budget limits and reserved-slot rules instead of cost-division scoring.
-
- ### E2E Lane Thresholds
-
- - `fixture-e2e threshold = Value Score >= 20` for non-reserved candidates
- - `service-integration-e2e threshold = Value Score > 50` for non-reserved candidates
- - Reserved-slot eligibility overrides the threshold when the candidate is the highest-value user-facing multi-step journey
-
- The fixture-e2e threshold is lower because this lane uses mocked backend or fixture-driven state, avoids live-stack setup, and has a higher per-feature budget. The service-integration-e2e threshold stays higher because live-stack tests are slower, more brittle, and more expensive to maintain.
-
- ### Selection Rules
-
- | Test Type | Ranking Basis | Selection Rule |
- |-----------|---------------|----------------|
- | Integration | Highest `Value Score` among integration candidates | Select up to budget |
- | fixture-e2e | Highest `Value Score` among fixture-e2e candidates | Select reserved user-facing journey or candidates with `Value Score >= 20` |
- | service-integration-e2e | Highest `Value Score` among service-integration-e2e candidates | Select reserved cross-service journey or candidates with `Value Score > 50` |
-
- ### E2E Candidate Rules
-
- - Treat integration and E2E as complementary coverage layers
- - Default browser-level user journeys to `fixture-e2e` when mocked backend or fixture-driven state can verify the behavior
- - Promote to `service-integration-e2e` only when correctness depends on real cross-service behavior such as DB persistence, queue/event delivery, transactional consistency, or external service contract payloads
- - Retain an E2E candidate when it validates a user-facing multi-step journey, even if integration tests partially cover the behavior
- - Distinguish user-facing journeys from service-internal chains; reserved fixture-e2e coverage applies only to user-facing journeys
+ # Integration and E2E Testing
- ### Reserved E2E Slot
+ ## Reference
- Reserve 1 fixture-e2e slot for the highest-value user-facing multi-step journey when such a journey exists, even if it does not satisfy `Value Score >= 20`.
+ Read [references/e2e-design.md](references/e2e-design.md) only when a selected browser-level claim needs UI Spec mapping or browser harness guidance. Use the repository's browser harness when it differs from the examples.
- Reserve 1 service-integration-e2e slot only when that journey requires real cross-service verification that fixture-e2e cannot prove.
+ ## Selection Gate
- ### E2E Absence Contract
+ Select an integration or E2E test only when all conditions hold:
- When no E2E test is generated, downstream artifacts must treat that as an explicit decision, not an error. Carry:
- - `generatedFiles.fixtureE2e: null`
- - `generatedFiles.serviceE2e: null`
- - `e2eAbsenceReason.fixtureE2e`: one of `no_user_facing_multi_step_journey`, `all_e2e_candidates_below_threshold`, `covered_by_existing_e2e`, `budget_not_justified`
- - `e2eAbsenceReason.serviceE2e`: one of the fixture reasons plus `no_real_service_dependency`
+ 1. a confirmed AC, preserved behavior, or task Verification Focus names an observable claim;
+ 2. correctness depends on a component, persistence, process, browser, or service boundary that a local/unit proof cannot exercise;
+ 3. existing tests do not already prove the same failure mode at that boundary;
+ 4. the expected regression-detection value justifies the fixture, environment, runtime, and maintenance cost.
- ### E2E Selection Decision Table
+ When any condition fails, leave the claim to its focused local or task verification. An empty integration/E2E selection is a successful result and needs no absence artifact.
- | Condition | Result |
- |-----------|--------|
- | At least one user-facing multi-step journey exists | Reserve 1 fixture-e2e slot for the highest-value such journey |
- | Journey correctness requires live cross-service behavior | Reserve or consider service-integration-e2e |
- | Remaining fixture-e2e candidate has `Value Score >= 20` | Eligible for non-reserved fixture-e2e selection |
- | Remaining service-integration-e2e candidate has `Value Score > 50` | Eligible for non-reserved service-integration-e2e selection |
- | Existing E2E already covers the same journey | Exclude and use `covered_by_existing_e2e` if no lane remains |
+ ## Lanes and Ceilings
- ## Test Skeleton Specification [MANDATORY]
+ | Lane | Use when | Ceiling per outcome |
+ |---|---|---|
+ | `integration` | In-process component, persistence, or contract interaction must stay real | 3 |
+ | `fixture-e2e` | Browser-visible interaction needs the real UI but controlled backend/fixture state is sufficient | 3 |
+ | `service-integration-e2e` | The claim specifically depends on a running local cross-service boundary that other lanes cannot prove | 2 |
- ### Required Comment Patterns
+ Ceilings are limits, not targets or reserved slots. Prefer the lowest-cost lane that proves the named claim. Real external production services are outside these lanes; verify their repository-owned contract instead.
- Each test MUST include the following annotations:
+ ## Candidate Selection
- ```
- // AC: [Original acceptance criteria text]
- // Behavior: [Trigger] -> [Process] -> [Observable Result]
- // @category: core-functionality | integration | edge-case | fixture-e2e | service-integration-e2e
- // @lane: integration | fixture-e2e | service-integration-e2e
- // @dependency: none | [component names] | full-ui (mocked backend) | full-system
- // @real-dependency: [component names] (optional)
- // @complexity: low | medium | high
- // Value Score: [score]
- ```
+ For each boundary-dependent claim:
- Adapt comment syntax to the project's language when generating or reviewing test skeletons.
+ 1. state the material failure that could remain falsely green;
+ 2. identify the boundary that must remain real and what may be controlled or mocked;
+ 3. compare with existing tests and lower-cost proof;
+ 4. select the candidate only when it adds distinct material detection value;
+ 5. combine claims that share setup, boundary, and observable outcome when one test can prove them clearly.
- ### Verification Items (Optional)
+ Do not invent revenue, frequency, legal, or probability scores. Use supplied product evidence and observed repository/test cost; when that evidence is weak, prefer no new wider-lane test unless the boundary is the only proof of a binding claim.
- When verification points need explicit enumeration:
- ```
- // Verification items:
- // - [Item 1]
- // - [Item 2]
- ```
+ ## Skeleton Contract
- ### E2E Preconditions (Optional but Recommended)
+ A generated skeleton is a non-runnable design artifact containing comments only until its implementation task makes it executable. Adapt comment syntax and filename conventions to the repository.
- When an E2E test requires environment setup, seed data, login state, or external dependency control, annotate it explicitly:
+ Each selected case records:
```text
- // Preconditions:
- // - Seeded user with active subscription
- // - Authenticated browser session
- // - Payment provider mocked or available in local environment
+ AC: [binding observable claim]
+ Behavior: [trigger] -> [boundary exercised] -> [observable result]
+ @lane: integration | fixture-e2e | service-integration-e2e
+ @dependency: [boundary components]
+ @real-dependency: [dependencies that must remain real]
+ Primary failure mode: [regression this test must detect]
+ Proof obligation: [assertions and permitted controlled boundaries]
```
- These annotations allow work-planner to create prerequisite tasks before E2E execution.
-
- ## EARS Format Mapping
-
- | EARS Keyword | Test Type | Generation Approach |
- |--------------|-----------|---------------------|
- | **When** | Event-driven | Trigger event -> verify outcome |
- | **While** | State condition | Setup state -> verify behavior |
- | **If-then** | Branch coverage | Both condition paths verified |
- | (none) | Basic functionality | Direct invocation -> verify result |
-
- ## Test File Naming Convention
-
- - Integration tests: `*.int.test.*` or `*.integration.test.*`
- - fixture-e2e tests: `*.fixture.e2e.test.*`
- - service-integration-e2e tests: `*.service.e2e.test.*`
- - legacy E2E tests: `*.e2e.test.*`
-
- The test runner or framework in the project determines the appropriate file extension.
+ Add verification items only when the proof cannot be stated clearly in the obligation. Do not add complexity scores, category taxonomies, or metadata without a downstream consumer.
## Review Criteria
- ### Skeleton and Implementation Consistency
-
- | Check | Failure Condition |
- |-------|-------------------|
- | Behavior Verification | No assertion for "observable result" in the implemented test |
- | Verification Item Coverage | Listed items not all covered by assertions |
- | Mock Boundary | Real dependencies from `@real-dependency` are isolated away or internal components are mocked without rationale |
-
- ### Implementation Quality
-
- | Check | Failure Condition |
- |-------|-------------------|
- | AAA Structure | Arrange/Act/Assert separation unclear |
- | Independence | State sharing between tests, order dependency |
- | Reproducibility | Date/random dependency, varying results |
- | Readability | Test name doesn't match verification content |
-
- ### Route Parity for Shared Mutations
-
- When multiple routes reach the same mutation, compare validation, classification, resource bounds, and read/parse/mutation/reporting order. Each difference requires an authoritative requirement, design contract, or explicit test claim; otherwise require a test exposing the bypass or inconsistent outcome.
+ A changed test passes review when it:
- ## Quality Standards [MANDATORY]
+ - exercises the selected real boundary rather than a substitute path;
+ - contains substantive assertions for the named observable result and primary failure;
+ - controls only dependencies permitted by the proof obligation;
+ - is deterministic and isolated enough for the repository's normal execution model.
- ### REQUIRED
- - Each test MUST verify one behavior
- - Clear AAA (Arrange-Act-Assert) structure
- - No test interdependencies
- - Deterministic execution
+ AAA comments, naming style, extra assertions, and generic readability improvements are non-blocking unless they obscure or invalidate the proof.
- ### PROHIBITED
- - Testing implementation details — test observable behavior only
- - Multiple behaviors per test — split into separate tests
- - Shared mutable state — each test creates its own state
- - Time-dependent assertions without mocking — use deterministic time
+ ## Completion Check
- **ENFORCEMENT**: Tests violating ANY standard MUST be rewritten before merge
+ - Every selected case passed the Selection Gate.
+ - The selected lane is the lowest-cost boundary that proves the claim.
+ - The set stays below the lane ceilings and may be empty.
+ - Each skeleton carries enough proof information for implementation and review without copying the Design Doc.