testing-strategy · diff
git:20251231.cb53cf6 to git:20260829.876a214
40 added, 212 removed. Audit A to A.
---
name: testing-strategy
- description: "Prioritizes tests based on ROI and critical user journeys. Use when: deciding test coverage, selecting test types, or budgeting effort."
+ description: "Selects the narrowest sufficient test boundary from requirements, repository evidence, and maintenance cost. Use when deciding integration or E2E coverage."
---
- # Test Strategy: ROI-Based Selection
-
- ## Core Principle: Maximum Coverage, Minimum Tests
-
- **Philosophy**: 10 reliable tests > 100 unmaintained tests
-
- Quality over quantity - focus resources on high-value tests that provide maximum coverage with minimum maintenance burden.
-
- ## ROI Calculation Framework
-
- ### ROI Formula
-
- ```
- ROI Score = (Business Value × User Frequency + Legal Requirement × 10 + Defect Detection)
- / (Creation Cost + Execution Cost + Maintenance Cost)
- ```
-
- ### Value Components
-
- **Business Value** (0-10 scale):
- - 10: Revenue-critical (payment processing, checkout)
- - 8-9: Core business features (user registration, data persistence)
- - 5-7: Important secondary features (search, filtering)
- - 2-4: Nice-to-have features (UI enhancements)
- - 0-1: Cosmetic features
-
- **User Frequency** (0-10 scale):
- - 10: Every user, every session (authentication)
- - 8-9: >80% of users regularly
- - 5-7: 50-80% of users occasionally
- - 2-4: <50% of users rarely
- - 0-1: Edge case users only
-
- **Legal Requirement** (boolean → 0 or 1):
- - 1: Legally mandated (GDPR compliance, data protection)
- - 0: Not legally required
-
- **Defect Detection** (0-10 scale):
- - 10: High likelihood of catching critical bugs
- - 5-7: Moderate likelihood of catching bugs
- - 0-4: Low likelihood (simple logic, well-tested patterns)
-
- ### Cost Components
-
- **Test Level Cost Table**:
-
- | Test Level | Creation Cost | Execution Cost | Maintenance Cost | Total Cost |
- |-------------|---------------|----------------|------------------|------------|
- | Unit | 1 | 1 | 1 | 3 |
- | Integration | 3 | 5 | 3 | 11 |
- | E2E | 10 | 20 | 8 | 38 |
-
- **Cost Rationale**:
- - **Unit Tests**: Fast to write, fast to run, rarely break from refactoring
- - **Integration Tests**: Moderate setup, slower execution, moderate maintenance
- - **E2E Tests**: Complex setup, very slow execution, high brittleness (12x more expensive than unit tests)
-
- ### ROI Calculation Examples
-
- **Example 1: Payment Processing Integration Test**
- ```
- Business Value: 10 (revenue-critical)
- User Frequency: 9 (90% of users)
- Legal Requirement: 0
- Defect Detection: 8 (high complexity)
-
- ROI = (10 × 9 + 0 + 8) / 11 = 98 / 11 = 8.9
- Decision: HIGH ROI → Generate this test
- ```
-
- **Example 2: UI Theme Toggle E2E Test**
- ```
- Business Value: 2 (cosmetic feature)
- User Frequency: 5 (50% of users)
- Legal Requirement: 0
- Defect Detection: 3 (simple logic)
-
- ROI = (2 × 5 + 0 + 3) / 38 = 13 / 38 = 0.34
- Decision: LOW ROI → Skip this E2E test (consider unit test instead)
- ```
-
- **Example 3: GDPR Data Deletion E2E Test**
- ```
- Business Value: 8 (critical compliance)
- User Frequency: 1 (rare user action)
- Legal Requirement: 1 (legally mandated)
- Defect Detection: 9 (high consequences if broken)
-
- ROI = (8 × 1 + 1 × 10 + 9) / 38 = 27 / 38 = 0.71
- Decision: MEDIUM ROI → Generate (legal requirement justifies cost)
- ```
-
- ## Critical User Journey Definition
-
- Tests with HIGH priority regardless of strict ROI calculation:
-
- ### Mandatory Coverage Areas
-
- 1. **Revenue-Impacting Flows**
- - Payment processing end-to-end
- - Checkout and order completion
- - Subscription management
- - Purchase confirmation and receipts
+ # Test Strategy
- 2. **Legally Required Flows**
- - GDPR data deletion/export
- - User consent management
- - Data protection compliance
- - Regulatory audit trails
+ ## Purpose
- 3. **High-Frequency Core Functionality**
- - User authentication/authorization (>80% of users)
- - Core CRUD operations for primary entities
- - Critical business workflows
- - Data integrity for primary data models
+ Select tests that prove an observable requirement at the lowest sufficient cost. Candidate discovery does not create an obligation to implement every technically valid test.
- **Budget Exception**: Critical User Journeys may exceed standard budget limits with explicit justification.
+ ## Selection Process
- ## Test Selection Guidelines
+ For each acceptance criterion that may require integration or E2E proof:
- ### Selection Thresholds
+ 1. **Name the observable boundary**: Identify the interaction, persisted effect, process, or user journey that must be proven.
+ 2. **Check existing proof**: Reuse or extend an established test when it already observes the required behavior.
+ 3. **Select the narrowest level**:
+ - use a unit or local check for isolated behavior;
+ - use integration when the requirement depends on a named component, persistence, or process boundary;
+ - use E2E only when the complete journey is itself the requirement.
+ 4. **Compare cost**: Consider setup, execution time, brittleness, maintenance, and diagnostic value using repository evidence rather than fixed universal scores.
+ 5. **Retain necessary proof**: Select a test only when a cheaper boundary cannot establish the required behavior or protect an accepted contract.
- **Integration Tests**:
- - ROI > 3.0: Strong candidate
- - ROI 1.5-3.0: Consider based on available budget
- - ROI < 1.5: Skip or convert to unit test
+ No new integration or E2E test is a valid conclusion when existing or cheaper proof is sufficient.
- **E2E Tests**:
- - ROI > 2.0: Strong candidate
- - ROI 1.0-2.0: Consider if Critical User Journey
- - ROI < 1.0: Skip (too expensive relative to value)
+ ## Priority
- ### Push-Down Analysis
+ Prioritize proof for:
- Before generating higher-level test, ask:
+ - core user or consumer journeys;
+ - shared or public contracts;
+ - persisted state transitions;
+ - security, authorization, compliance, or irreversible effects;
+ - integrations with demonstrated failure risk.
- 1. **Can this be unit-tested?**
- - YES → Generate unit test instead
- - NO → Continue to integration test consideration
+ Priority does not determine test level. Use the narrowest boundary that can observe the prioritized behavior.
- 2. **Already covered by integration test?**
- - YES → Don't create E2E version
- - NO → Consider E2E test if ROI justifies
+ ## Deduplication
- **Example**:
- - "Tax calculation accuracy" → Unit test (pure logic)
- - "Tax applied to order total" → Integration test (multiple components)
- - "User sees correct tax in checkout flow" → E2E test only if Critical User Journey
+ Before adding a test:
- ## Deduplication Strategy
+ - search the existing suite for equivalent behavior;
+ - identify overlapping proof at other levels;
+ - extend existing coverage when that remains clear and maintainable;
+ - keep one representative proof when several candidates observe the same risk.
- Before generating any test:
+ ## Selection Record
- 1. **Search existing test suite** for similar coverage
- 2. **Check for overlapping scenarios** at different test levels
- 3. **Identify redundant verifications** already covered elsewhere
+ Record only decisions needed by planning or review:
- **Decision Matrix**:
- ```
- Existing coverage found?
- → Full coverage: Skip new test
- → Partial coverage: Extend existing test
- → No coverage: Generate new test
+ ```text
+ Acceptance criterion: [identifier or behavior]
+ Selected proof: existing | local | integration | E2E | none
+ Boundary observed: [observable interaction or result]
+ Reason: [why this is the narrowest sufficient proof]
```
- ## Application in Test Generation
-
- ### Phase 1: Candidate Enumeration
- - List all possible test scenarios
- - Assign ROI metadata to each candidate
-
- ### Phase 2: ROI-Based Selection
- 1. Calculate ROI for each candidate
- 2. Apply deduplication checks
- 3. Apply push-down analysis
- 4. Sort by ROI (descending)
-
- ### Phase 3: Budget Enforcement
- - Select top N tests within budget limits
- - Document budget usage
- - Report selection rationale
-
- **See**: `.agents/tasks/acceptance-test-generation.md` for detailed implementation process
-
- ## Continuous Improvement
-
- ### Metrics to Track
-
- 1. **Selection Rate**: Tests generated / Total candidates
- - Target: 25-35% (indicates effective filtering)
-
- 2. **Average ROI**: Average ROI of generated tests
- - Target: >3.0 for integration, >1.5 for E2E
-
- 3. **Budget Utilization**: Actual tests / Budget limit
- - Target: 80-100% (full utilization of valuable test slots)
-
- 4. **Defect Detection Rate**: Bugs caught / Total tests
- - Track over time to validate ROI predictions
-
- ### Calibration
-
- Periodically review:
- - Are high-ROI tests actually catching bugs?
- - Are cost estimates accurate?
- - Do business value ratings align with stakeholder priorities?
-
- **Adjust formula weights based on empirical data**
-
- ## Anti-Patterns to Avoid
-
- ❌ **Gaming the System**:
- - Inflating business value scores to justify favorite tests
- - Ignoring ROI when it contradicts intuition
- - Cherry-picking ROI calculation only for preferred tests
-
- ✅ **Proper Usage**:
- - Apply ROI calculation consistently to all candidates
- - Document justification when overriding ROI decisions
- - Use empirical data to calibrate scores over time
-
- ❌ **Analysis Paralysis**:
- - Spending excessive time on precise ROI calculations
- - Debating single-point differences in scores
- - Treating ROI as exact science rather than decision aid
+ ## Completion Check
- ✅ **Practical Application**:
- - Use ROI for relative prioritization, not absolute precision
- - Focus on order-of-magnitude differences (8.9 vs 0.34)
- - Make quick decisions for obvious high/low ROI cases
+ - Every selected test traces to an observable requirement or accepted contract.
+ - A cheaper sufficient boundary was considered.
+ - Existing proof was reused where sufficient.