test-strategy · diff

v1.0 to v2.0

159 added, 271 removed. Audit A to A.

---

name: test-strategy

description: >-

Produce a multi-quarter QA strategy document. Covers scope, risk-based

prioritization, test levels (unit/integration/E2E), pyramid analysis, entry/exit

- criteria, quality KPIs, tool selection rationale, and timeline planning. Output

- is an actionable strategy document, not a shelf document. Use when: "test

- strategy," "QA strategy doc," "testing approach," "QA roadmap." Not for: a

- single-sprint or single-release plan — use `test-planning`. Not for: identifying

- which areas carry the most risk — use `risk-based-testing` first.

- Related: risk-based-testing, qa-metrics, release-readiness, test-planning.

+ criteria, quality KPIs, tool selection rationale, CI scaling levers, and timeline

+ planning. Output is an actionable strategy document, not a shelf document. Use when:

+ "test strategy," "QA strategy doc," "testing approach," "QA roadmap,"

+ "multi-quarter QA direction." Not for: a single-sprint or single-release plan —

+ use test-planning. Not for: identifying which areas carry the most risk — use

+ risk-based-testing first.

+ Related: risk-based-testing, qa-metrics, release-readiness, test-planning,

+ test-reliability.

license: MIT

metadata:

author: kindlmann

- version: "1.0"

+ version: "2.0"

category: strategy

---

- Generate a comprehensive, actionable QA strategy document tailored to the product, team, and risk profile. The output should be a living document that drives daily testing decisions, not a compliance artifact that collects dust.

+ Generate an actionable QA strategy tailored to the product, team, and risk profile — a document that drives daily testing decisions, not a compliance artifact that collects dust. A team with 150 E2E tests and a 52-minute pipeline thinks it has good coverage; this skill diagnoses the inverted pyramid, prescribes the rebalance, and ties every element to a measurable KPI.

</objective>

---

## Discovery Questions

- Before writing a single line of strategy, gather context. Check `.agents/qa-project-context.md` first -- if it exists, use it as the foundation and skip questions already answered there.

+ Before writing a single line of strategy, gather context. Check `.agents/qa-project-context.md` first — if it exists, use it as the foundation and skip questions already answered there.

### Product & Business Context

- What is the product? (SaaS, e-commerce, API platform, mobile app, content site)

- Who are the users? (consumers, enterprise, internal, developers)

- - What are the business-critical flows? (signup, checkout, payment, data export, etc.)

+ - What are the business-critical flows? (signup, checkout, payment, data export)

- What is the release cadence? (continuous, weekly, bi-weekly, quarterly)

- - What regulatory or compliance requirements exist? (SOC2, HIPAA, PCI-DSS, GDPR)

+ - What compliance requirements exist? (SOC2, HIPAA, PCI-DSS, GDPR, EU AI Act)

### Current Testing State

- - What test levels exist today? (unit, integration, E2E, manual, none)

- - What is the current test count at each level?

+ - What test levels exist today, and the current count at each level?

- What frameworks and tools are in use?

- - What is the current code coverage percentage?

- - What is the target coverage, if any?

+ - Current code coverage, and the target if any?

- How long does the CI pipeline take end-to-end?

- What is the current flakiness rate?

### Pain Points & Goals

- - What are the biggest quality pain points? (regressions, slow feedback, flaky tests, gaps)

+ - Biggest quality pain points? (regressions, slow feedback, flaky tests, gaps)

- What broke in the last 3 releases? What escaped to production?

- What does "good enough quality" look like for this team?

- - What is the team's appetite for investment in test infrastructure?

+ - Appetite for investment in test infrastructure?

### Team & Constraints

- Team size and composition (devs, QA, SDET, manual testers)

- Skill levels with automation tools

- Budget constraints for tooling

- - Timeline pressure -- is there a deadline driving this strategy?

+ - Timeline pressure — is there a deadline driving this strategy?

---

- > **Calibrate to your team maturity** (set `team_maturity` in `.agents/qa-project-context.md`):

- > - **startup** — Focus on a minimal test pyramid: unit tests + a handful of critical E2E paths. Skip contract testing and formal metrics until you have CI running reliably.

+ > **Calibrate to team maturity** (set `team_maturity` in `.agents/qa-project-context.md`):

+ > - **startup** — Minimal pyramid: unit tests + a handful of critical E2E paths. Skip contract testing and formal metrics until CI runs reliably. Phase 1 under 4 weeks.

> - **growing** — Full pyramid with defined coverage targets, flakiness thresholds, and CI quality gates. Add risk-based prioritization.

- > - **established** — Comprehensive strategy with SLA-backed quality gates, multi-environment coverage, advanced tooling (contract testing, chaos, observability), and formal review cadence.

+ > - **established** — SLA-backed quality gates, multi-environment coverage, advanced tooling (contract testing, chaos, observability), and formal review cadence.

---

## Core Principles

- Every strategy produced by this skill adheres to these five principles:

- ### 1. Risk-Based Prioritization Over Exhaustive Coverage

- Not all code is equal. A payment processing bug costs 1000x more than a tooltip typo. Allocate testing effort proportional to business risk, not code volume. The risk assessment matrix (below) drives where to invest.

+ 1. **Risk-based prioritization over exhaustive coverage.** Not all code is equal — a payment bug costs 1000x more than a tooltip typo. Allocate testing effort proportional to business risk, not code volume. The risk matrix drives where to invest; run `risk-based-testing` first if no matrix exists yet.

- ### 2. Test Pyramid Health

- A healthy test suite follows the pyramid shape: many fast unit tests, fewer integration tests, fewest E2E tests. When the shape inverts (ice cream cone), feedback is slow, maintenance is high, and confidence is paradoxically low. Diagnose the current shape and prescribe corrections.

+ 2. **Test pyramid health is the leading indicator.** A healthy suite is many fast unit tests, fewer integration, fewest E2E. When the shape inverts (ice cream cone) feedback is slow, maintenance is high, and confidence is paradoxically low. Diagnose the current shape before prescribing anything.

- ### 3. Shift-Left: Catch Defects Earlier

- Every defect found later costs exponentially more. Strategy should push validation earlier: static analysis before tests, unit tests before integration, contract tests before E2E. Design reviews catch architecture bugs that no test can find.

+ 3. **Shift-left: catch defects earlier.** Every defect found later costs exponentially more. Push validation earlier — static analysis before tests, unit before integration, contract before E2E. Design reviews catch architecture bugs no test can find.

- ### 4. Measurable: Every Strategy Element Has a KPI

- If you cannot measure it, you cannot improve it. Every section of the strategy must define what success looks like in numbers: coverage targets, flakiness thresholds, defect escape rate goals, MTTR limits.

+ 4. **Every strategy element has a KPI.** If you cannot measure it, you cannot improve it. Coverage targets, flakiness thresholds, escape-rate goals, MTTR limits — each section names a number and a tracking cadence.

- ### 5. Living Document: Strategy Evolves With the Product

- The strategy is reviewed quarterly at minimum. It includes a revision history, owners for each section, and triggers for re-evaluation (new product area, team change, major incident).

+ 5. **Living document, not a shelf document.** Reviewed quarterly at minimum. It carries a revision history, a named owner per section, and explicit re-evaluation triggers (new product area, team change, major incident, defect escape).

---

## Strategy Document Template

- Walk through each section below to produce the final strategy document. Tailor depth to the product's complexity -- a 5-person startup needs 5 pages, not 50.

- ---

+ Walk through each section to produce the final document. Tailor depth to complexity — a 5-person startup needs 5 pages, not 50. The final document follows a 13-section structure (Executive Summary through Revision History); see `references/diagrams-and-worksheets.md` for the copy-paste markdown skeleton, and `references/strategy-templates.md` for four fully worked examples (SaaS, e-commerce, API-first, media).

### 1. Scope & Objectives

- Define the boundaries clearly. Ambiguity here causes gaps and wasted effort downstream.

- **In Scope:**

- - List every product area, service, and integration that this strategy covers

- - Include both functional and non-functional testing types

- - Specify platforms and browsers/devices

- **Out of Scope:**

- - Explicitly state what is NOT covered and why

- - Third-party services tested only at the contract level

- - Legacy systems scheduled for deprecation

- **Objectives:**

- - State 3-5 measurable objectives with timelines

- - Example: "Reduce defect escape rate from 12% to under 5% within two quarters"

- - Example: "Achieve 80% unit test coverage on all services launched after Q1 2026"

+ Define boundaries clearly. Ambiguity here causes gaps and wasted effort downstream.

- ---

+ - **In scope:** every product area, service, and integration this strategy covers; functional and non-functional types; platforms and browsers/devices.

+ - **Out of scope:** state what is NOT covered and why — third-party services tested only at the contract level, legacy systems slated for deprecation.

+ - **Objectives:** 3-5 measurable objectives with timelines, e.g. "Reduce defect escape rate from 12% to under 5% within two quarters," "Achieve 80% unit coverage on all services launched after Q1 2026."

### 2. Test Levels & Types

- Define each test level, what it covers, who owns it, and the expected volume.

+ Define each level, what it covers, who owns it, and expected volume.

|-------|-------------------|-------|-----------|-------------|---------------|

- Adjust this table based on what the product actually needs. Not every product needs visual regression testing. Every product needs unit and integration tests.

- ---

+ Adjust to what the product actually needs. Not every product needs visual regression. Every product needs unit and integration tests.

### 3. Test Pyramid Analysis

- Diagnose the current shape of the test suite and define the target state.

- #### Shapes and What They Mean

- The suite takes one of four shapes — healthy pyramid (many unit, few E2E), ice cream cone (inverted), diamond (integration-heavy), or hourglass (missing middle layer). Each signals a different feedback/maintenance trade-off. See `references/diagrams-and-worksheets.md` for the side-by-side ASCII diagram of all four shapes and what each one means.

- #### Current State Assessment

- Count the current tests at each level, compute the percentage split, and identify the shape. Then capture CI duration, flaky rate, and pass rate. See `references/diagrams-and-worksheets.md` for the fill-in Current State Assessment Worksheet.

+ Diagnose the current shape, then define the target.

- #### Target State

+ **Shapes.** The suite takes one of four shapes — healthy pyramid (many unit, few E2E), ice cream cone (inverted, E2E-heavy), diamond (integration-heavy), or hourglass (unit-heavy and E2E-sparse with a missing integration middle). Each signals a different feedback/maintenance trade-off. See `references/diagrams-and-worksheets.md` for the side-by-side ASCII diagram.

- Define target ratios (70-80% unit, 15-20% integration, 5-10% E2E) with concrete counts, plus target CI duration and flaky rate. See `references/diagrams-and-worksheets.md` for the Target State Worksheet.

+ **Current state.** Count tests at each level, compute the percentage split, identify the shape, then capture CI duration, flaky rate, and pass rate. See the Current State Assessment Worksheet in the reference file.

- #### Action Plan to Shift Toward Healthy Pyramid

+ **Target state.** Define target ratios (70-80% unit, 15-20% integration, 5-10% E2E) with concrete counts, plus target CI duration and flaky rate. See the Target State Worksheet.

- If ice cream cone or diamond:

- 1. **Freeze E2E growth** -- no new E2E tests unless covering a net-new critical path

- 2. **Decompose existing E2E tests** -- identify E2E tests that validate logic testable at unit level, rewrite them

- 3. **Add unit test requirements to PR checklist** -- every PR touching business logic must include unit tests

- 4. **Set CI gates** -- fail PRs where unit:E2E ratio drops below threshold

+ **Action plan — if ice cream cone or diamond:**

+ 1. **Freeze E2E growth** — no new E2E tests unless covering a net-new critical path.

+ 2. **Decompose existing E2E** — find E2E tests validating logic testable at unit level (a checkout test asserting tax math becomes a unit test on the tax function), rewrite them down a level.

+ 3. **Add unit requirements to the PR checklist** — every PR touching business logic ships unit tests.

+ 4. **Set CI gates** — fail PRs where the unit:E2E ratio drops below threshold.

- If hourglass:

- 1. **Invest in integration test infrastructure** -- database fixtures, service stubs, contract tests

- 2. **Identify service boundaries** -- each boundary needs integration tests for happy path + error cases

- 3. **Use contract testing** (Pact or similar) for inter-service communication

+ Before rebalancing, separate genuinely flaky E2E tests from ones exposing real bugs — quarantining a flaky test that hides a race condition is how the regression escapes. For flake root-cause triage and quarantine mechanics, see `test-reliability`.

- ---

+ **Action plan — if hourglass:**

+ 1. **Invest in integration infrastructure** — DB fixtures, service stubs, contract tests.

+ 2. **Identify service boundaries** — each boundary needs integration tests for happy path + error cases.

+ 3. **Use contract testing** (Pact) for inter-service communication.

### 4. Risk Assessment Matrix

- Map features to risk levels. This directly determines testing depth.

- #### 5x5 Risk Matrix

- Score each feature as Impact (1 Negligible → 5 Catastrophic) × Likelihood (1 Rare → 5 Almost Certain); the product (1-25) maps to LOW/MED/HIGH/CRIT bands. See `references/diagrams-and-worksheets.md` for the full 5x5 matrix with every cell labeled.

- #### Risk-to-Testing Action Map

+ Map features to risk levels — this directly determines testing depth. Score each feature as Impact (1 Negligible → 5 Catastrophic) × Likelihood (1 Rare → 5 Almost Certain); the product (1-25) maps to LOW/MED/HIGH/CRIT bands. See `references/diagrams-and-worksheets.md` for the full 5x5 matrix with every cell labeled.

|------------|---------------|------------|------------|

- #### Example Risk Assessment

+ **Example mapping:**

- |-------------|--------|------------|------------|-----------------|

+ |-------------|--------|------------|-------|-----------------|

- ---

### 5. Environment Strategy

- Define which environments exist and what testing happens in each.

|------------|---------|------------|------|---------------|

- Key decisions to document:

- - How is test data managed in each environment?

- - Are environments ephemeral (preview deployments) or long-lived?

- - Who has access to each environment?

- - How are environment-specific configurations managed?

- ---

+ Document: how test data is managed per environment, whether environments are ephemeral (preview deployments) or long-lived, who has access, and how environment-specific config is managed.

### 6. Tool Selection Rationale

- Do not pick tools first. Understand needs first, then select tools that fit.

- #### Decision Matrix Template

+ Do not pick tools first. Understand needs, then select tools that fit. Score each candidate against weighted criteria.

|-------------------|--------|--------|--------|

- Score each 1-5, multiply by weight, sum for weighted total.

- #### Total Cost of Ownership

- Beyond license fees, account for:

- - **Setup time:** How long to configure CI, write first tests, train team

- - **Writing time:** How long to write a typical test (measure this -- time 5 tests)

- - **Maintenance time:** How often do tests break due to framework updates

- - **Debug time:** When a test fails, how long to diagnose (good error messages matter)

- - **Infrastructure cost:** Browser farms, parallel execution, cloud runners

+ Score each 1-5, multiply by weight, sum for the weighted total. Beyond license fees, account for **total cost of ownership**: setup time (configure CI, write first tests, train team), writing time (time 5 real tests to measure), maintenance time (how often tests break on framework updates), debug time (good error messages cut this), and infrastructure cost (browser farms, parallel runners).

- #### Common Stack Recommendations

+ **Common stack starting points** — document why you chose or deviated:

|-------------|------|-------------|-----|-----|--------|

- These are starting points, not mandates. Document why you chose or deviated. For AI/LLM features, also add explicit risk testing for hallucinations, bias, prompt injection, and privacy — see `ai-system-testing` and `compliance-testing` (EU AI Act).

- #### Reference Frameworks

- - **CTAL-AT v2.0** (ISTQB, released 2026-05-06) — first major Advanced Agile Tester syllabus update since 2014. Directly covers test strategy and approach challenges, whole-team approach, shift-left, end-to-end testing, test smells, exploratory + assisted testing. Replaces CTFL-AT.

- - **CT-GenAI v1.1** (ISTQB, released 2026-04-27) — formalizes LLM-powered test infrastructure as a discipline. Defines AI-specific risk classes (hallucinations, reasoning errors, bias, privacy, AI regulations).

- - **CTFL v4.0** (ISTQB) — foundational vocabulary; useful when aligning across teams that come from different testing traditions.

- - **HTSM v6.3** (Bach, last updated Dec 2024) — Heuristic Test Strategy Model; emphasizes state-based testing and boundary heuristics. Lightweight alternative to ISTQB framing.

- - **World Quality Report 2025-26** (Capgemini, 17th edition, "Adapting to Emerging Worlds") — strategy data points: 43% of orgs experimenting with Gen AI in QA, 15% scaled. Useful for benchmarking your AI-adoption stage during strategy planning.

+ For AI/LLM features, add explicit risk testing for hallucinations, bias, prompt injection, and privacy — see `ai-system-testing` and `compliance-testing` (EU AI Act).

- ---

+ **Reference frameworks:**

+ - **CTAL-AT v2.0** (ISTQB, Advanced Agile Tester, 2026) — a new Advanced-level certification requiring CTFL v4.0, not an update of an Advanced predecessor; it supersedes the retired Foundation-level CTFL-AT Agile extension (already absorbed into CTFL v4.0). Covers test strategy and approach, whole-team approach, shift-left, end-to-end testing, test smells, exploratory + AI-assisted testing.

+ - **CT-GenAI v1.1** (ISTQB, released 2026-04-27) — formalizes LLM-powered test infrastructure as a discipline; defines AI-specific risk classes (hallucinations, reasoning errors, bias, privacy, AI regulations).

+ - **CTFL v4.0** (ISTQB) — foundational vocabulary; useful when aligning teams from different testing traditions.

+ - **HTSM v6.3** (Bach) — Heuristic Test Strategy Model; emphasizes state-based testing and boundary heuristics. Lightweight alternative to ISTQB framing.

+ - **World Quality Report 2025-26** (Capgemini, 17th edition) — benchmark data: 43% of orgs experimenting with Gen AI in QA, 15% scaled. Useful for placing your AI-adoption stage during planning.

- ### 7. Entry/Exit Criteria

+ ### 7. CI Scaling Levers

- Define what must be true before testing starts (entry) and what must be true before testing is considered done (exit) at each level.

+ When the suite or team grows, CI wall-clock time is the constraint that breaks the strategy. Pull these levers before deleting tests:

- #### Unit Testing

- - **Entry:** Code compiles, function has a clear contract (inputs/outputs documented)

- - **Exit:** All branches covered, edge cases tested, no skipped tests, coverage target met

+ - **Sharding** — split the suite across N parallel runners (Playwright `--shard=1/4`, Jest `--shard`, pytest-xdist, Cypress parallelization). Linear speedup until per-shard fixed costs (install, build) dominate.

+ - **Test impact analysis** — run only tests affected by the diff instead of the whole suite on every PR. Driven by a dependency graph (Nx affected, Vitest `--changed`, Bazel) or coverage-to-file maps. Keep the full suite on a nightly/merge gate so nothing rots.

+ - **Caching** — cache dependencies, build artifacts, and browser binaries between runs.

+ - **Selective E2E on PR** — run smoke E2E on PRs, full E2E on merge/nightly.

- #### Integration Testing

- - **Entry:** Unit tests pass, dependent services available (or stubbed), test data seeded

- - **Exit:** All service boundaries tested, error paths validated, no flaky tests

+ Measure the payoff, do not assume it. **Parallel efficiency = summed test-run time ÷ wall-clock time**; target a value approaching the shard count (e.g. >3x on 4 shards). A low value means fixed setup costs or a long-pole test are eating the speedup. Track **CI-minutes-per-PR** to catch parallelization that cuts wall-clock time but balloons billed compute.

- #### E2E Testing

- - **Entry:** Integration tests pass, staging deployed, test accounts provisioned

- - **Exit:** All critical user journeys pass, no P0/P1 defects open, performance within SLA

+ ### 8. Entry/Exit Criteria

- #### Release

- - **Entry:** All test levels pass, no CRITICAL/HIGH defects open, release notes drafted

- - **Exit:** Smoke tests pass in production, monitoring shows no anomalies for 30 min, rollback plan verified

+ Define what must be true before testing starts (entry) and before it is done (exit) at each level.

- ---

+ **Unit** — Entry: code compiles, function has a documented contract (inputs/outputs). Exit: all branches covered, edge cases tested, no skipped tests, coverage target met.

- ### 8. Quality Gates & Definition of Done

+ **Integration** — Entry: unit tests pass, dependent services available or stubbed, test data seeded. Exit: all service boundaries tested, error paths validated, no flaky tests.

- Define automated gates that prevent bad code from moving forward.

+ **E2E** — Entry: integration tests pass, staging deployed, test accounts provisioned. Exit: all critical user journeys pass, no P0/P1 defects open, performance within SLA.

- #### PR Gate (runs on every pull request)

- - [ ] All unit tests pass

- - [ ] All integration tests pass

- - [ ] Code coverage does not decrease (or meets minimum threshold)

- - [ ] No new linting errors

- - [ ] SAST scan passes (no new high/critical findings)

- - [ ] Bundle size does not increase beyond threshold

- - [ ] At least one approval from code reviewer

+ **Release** — Entry: all test levels pass, no CRITICAL/HIGH defects open, release notes drafted. Exit: smoke tests pass in production, monitoring shows no anomalies for an agreed bake window (30 min is a reasonable default — tune to your deploy frequency and alert latency), rollback plan verified.

- #### Merge Gate (runs on merge to main)

- - [ ] All PR gate checks pass

- - [ ] E2E smoke suite passes against preview deployment

- - [ ] No merge conflicts

- - [ ] Branch is up to date with main

+ ### 9. Quality Gates & Definition of Done

- #### Deploy Gate (runs before production deployment)

- - [ ] Full E2E suite passes on staging

- - [ ] Performance benchmarks within acceptable range

- - [ ] Security scan passes

- - [ ] Feature flags configured correctly

- - [ ] Rollback plan documented and tested

+ Automated gates that prevent bad code from moving forward.

- #### Nightly Gate (runs on schedule)

- - [ ] Full E2E suite including edge cases

- - [ ] Visual regression tests

- - [ ] Performance/load tests

- - [ ] Accessibility scan

- - [ ] Dependency vulnerability scan

- - Results reviewed by QA lead next morning

+ **PR gate** (every PR): unit tests pass; integration tests pass; coverage does not decrease (or meets minimum); no new lint errors; SAST scan passes (no new high/critical); bundle size within threshold; at least one reviewer approval.

- ---

+ **Merge gate** (merge to main): all PR-gate checks pass; E2E smoke suite passes against preview deployment; no merge conflicts; branch up to date with main.

- ### 9. Metrics & KPIs

+ **Deploy gate** (before production): full E2E suite passes on staging; performance benchmarks within range; security scan passes; feature flags configured; rollback plan documented and tested.

- Track these metrics to know if the strategy is working.

+ **Nightly gate** (scheduled): full E2E including edge cases; visual regression; performance/load tests; accessibility scan; dependency vulnerability scan. Results reviewed by QA lead next morning.

- |--------|-----------|--------|-----------------|

+ Every gate names a concrete pass/fail threshold and is enforced in CI — a gate that can be clicked past is documentation, not a gate.

- #### How to Use These Metrics

+ ### 10. Metrics & KPIs

- - **Do not** use metrics to punish teams. Use them to identify systemic issues.

- - **Do** track trends over time, not absolute numbers. A team going from 30% to 60% coverage is doing great.

- - **Do** set realistic targets based on current state. Jumping from 20% to 90% coverage in one quarter is not a plan, it is a fantasy.

- - **Do** review metrics quarterly with engineering leadership. Celebrate improvements.

- - **Do** investigate spikes. A sudden increase in flakiness signals an infrastructure problem, not a laziness problem.

+ |--------|-----------|--------|---------|

- ---

+ **Using metrics:** track trends over time, not absolute numbers — a team going 30%→60% coverage is doing great. Set realistic targets from current state (20%→90% in one quarter is a fantasy, not a plan). Review quarterly with leadership; celebrate improvements. Investigate spikes — a sudden flakiness jump signals infrastructure, not laziness. Never use metrics to punish teams. See `qa-metrics` for full KPI definitions, DORA metrics, and dashboards.

- ### 10. Timeline & Milestones

+ ### 11. Timeline & Milestones

- Roll out the strategy in phases. Trying to do everything at once guarantees nothing gets done well.

+ Roll out in phases. Doing everything at once guarantees nothing gets done well.

- #### Phase 1: Foundation (Weeks 1-4)

- - [ ] Complete risk assessment for all product areas

- - [ ] Set up CI pipeline with unit test gate

- - [ ] Establish baseline metrics (current coverage, flakiness, pipeline time)

- - [ ] Write unit tests for top 5 highest-risk areas

- - [ ] Select and configure E2E framework

- - **Exit criteria:** CI runs unit tests on every PR, baseline metrics documented

+ **Phase 1 — Foundation (Weeks 1-4):** risk assessment for all product areas; CI pipeline with unit-test gate; baseline metrics (coverage, flakiness, pipeline time); unit tests for top 5 highest-risk areas; select and configure E2E framework. *Exit: CI runs unit tests on every PR, baseline metrics documented.*

- #### Phase 2: Coverage Expansion (Weeks 5-10)

- - [ ] Add integration tests for all service boundaries

- - [ ] Write E2E tests for top 10 critical user journeys

- - [ ] Implement visual regression testing for key pages

- - [ ] Set up test data management

- - [ ] Configure nightly test runs

- - **Exit criteria:** All critical paths have E2E coverage, integration tests cover all APIs

+ **Phase 2 — Coverage Expansion (Weeks 5-10):** integration tests for all service boundaries; E2E for top 10 critical journeys; visual regression for key pages; test data management; nightly runs. *Exit: all critical paths have E2E coverage, integration tests cover all APIs.*

- #### Phase 3: Quality Gates (Weeks 11-14)

- - [ ] Enable coverage gates on PRs (no decrease allowed)

- - [ ] Add performance benchmarks to CI

- - [ ] Implement security scanning in pipeline

- - [ ] Set up monitoring dashboards for all KPIs

- - **Exit criteria:** All four gates (PR, merge, deploy, nightly) are active and enforced

+ **Phase 3 — Quality Gates (Weeks 11-14):** coverage gates on PRs (no decrease); performance benchmarks in CI; security scanning; monitoring dashboards for all KPIs. *Exit: all four gates (PR, merge, deploy, nightly) active and enforced.*

- #### Phase 4: Optimization (Weeks 15-20)

- - [ ] Identify and fix or quarantine flaky tests

- - [ ] Optimize CI pipeline for speed (parallelization, caching)

- - [ ] Implement test impact analysis (run only affected tests)

- - [ ] Set up synthetic monitoring in production

- - [ ] First quarterly strategy review

- - **Exit criteria:** CI under 15 min, flakiness under 2%, first strategy revision published

+ **Phase 4 — Optimization (Weeks 15-20):** fix or quarantine flaky tests; CI scaling levers (sharding, caching, test impact analysis); synthetic monitoring in production; first quarterly strategy review. *Exit: CI under 15 min, flakiness under 2%, first strategy revision published.*

- #### Ongoing

- - Quarterly strategy review and revision

- - Monthly metrics review with team

- - Continuous test maintenance (refactor, de-flake, retire)

+ **Ongoing:** quarterly strategy review and revision; monthly metrics review; continuous maintenance (refactor, de-flake, retire).

---

## Anti-Patterns

- Watch for these common failures. If you spot them, call them out explicitly in the strategy document.

- ### 100% Coverage Targets

- Diminishing returns kick in hard past 80%. The last 20% requires testing getters, setters, and trivial code while ignoring the integration gaps where real bugs live. Set coverage targets per module based on risk, not a blanket number.

+ **100% coverage targets.** Diminishing returns past 80%. The last 20% means testing getters and trivial code while ignoring integration gaps where real bugs live. Set coverage per module by risk, not a blanket number.

- ### Ice Cream Cone (Inverted Pyramid)

- Too many E2E tests, too few unit tests. Symptoms: CI takes 45+ minutes, tests break constantly due to UI changes, nobody trusts the test suite. Fix by freezing E2E growth and decomposing existing E2E tests into lower levels.

+ **Ice cream cone (inverted pyramid).** Too many E2E, too few unit. Symptoms: CI 45+ minutes, tests break on every UI change, nobody trusts the suite. Fix by freezing E2E growth and decomposing existing E2E into lower levels.

- ### Strategy as One-Time Document

- A strategy written once and never updated is worse than no strategy, because it gives false confidence. Build in review triggers: quarterly calendar review, post-incident review, new product area launch, team composition change.

+ **Strategy as a one-time document.** Written once and never updated is worse than none — it gives false confidence. Build in review triggers: quarterly calendar review, post-incident, new product area, team composition change.

- ### Tool-First Thinking

- "We should use Playwright" is not a strategy. It is a tool choice masquerading as a plan. Start with what you need to validate, then pick tools that fit. The strategy document should justify tool choices, not lead with them.

+ **Tool-first thinking.** "We should use Playwright" is a tool choice masquerading as a plan. Start from what you need to validate, then pick tools that fit. The document justifies tool choices, never leads with them.

- ### No Metrics = No Accountability

- A strategy without measurable targets is a wish list. Every section should connect to a KPI. If you cannot define what success looks like for a strategy element, question whether it belongs in the strategy.

+ **No metrics = no accountability.** A strategy without measurable targets is a wish list. Every section connects to a KPI. If you cannot define success in numbers, question whether the element belongs.

- ### Testing in Isolation

- QA strategy that lives only in the QA team's wiki is invisible to developers. The strategy must be integrated into the development workflow: PR templates, CI gates, Definition of Done. If developers do not see it daily, it does not exist.

+ **Testing in isolation.** A strategy living only in the QA wiki is invisible to developers. It must live in PR templates, CI gates, and the Definition of Done. If developers do not see it daily, it does not exist.

- ### Copy-Paste Strategy

- Taking another company's strategy verbatim ignores your product's unique risk profile, team skills, and constraints. Use templates as starting points, but every section must be tailored to your specific context.

+ **Copy-paste strategy.** Taking another company's strategy verbatim ignores your risk profile, team skills, and constraints. Templates are starting points; every section is tailored.

- ### Automating Everything Immediately

- Manual exploratory testing has enormous value, especially early in a product's life. Automate regression, keep exploration manual. The strategy should specify what stays manual and why.

+ **Automating everything immediately.** Manual exploratory testing has enormous value, especially early. Automate regression, keep exploration manual. The strategy specifies what stays manual and why.

---

- ## Output Format

+ ## Verification

- The final strategy document follows a 13-section structure: Executive Summary, Scope & Objectives, Test Levels & Types, Test Pyramid Analysis, Risk Assessment, Environment Strategy, Tool Selection, Entry/Exit Criteria, Quality Gates, Metrics & KPIs, Timeline & Milestones, Risks to the Strategy Itself, and Revision History. See `references/diagrams-and-worksheets.md` for the copy-paste markdown skeleton, and `references/strategy-templates.md` for four fully worked examples (SaaS, e-commerce, API-first, media).

+ Prove the produced document is complete before calling it done. Run against the saved strategy file:

+ ```bash

+ DOC=docs/qa-strategy.md

+ # 1. All 13 numbered section headings present (Executive Summary → Revision History)

+ grep -cE '^### [0-9]+\.' "$DOC" # expect 13

+ # 2. Every row in the Metrics & KPIs table has a non-empty Target cell

+ # (no "| | " gaps in the target column) — visually scan the table block:

+ # 3. A revision history and a named owner exist

+ grep -niE 'revision history|owner:' "$DOC"

+ ```

+ Then sanity-check the pyramid math by hand: target unit% + integration% + E2E% should sum to ~100%. If the document recommends sharding, confirm a parallel-efficiency or CI-minutes target appears in the Metrics table — an optimization with no metric is a guess.

---

## Done When

- - A strategy document exists at an agreed location with all 13 sections populated (Executive Summary through Revision History)

- - Test pyramid target ratios are defined with concrete counts and a timeline to reach them

- - Entry and exit criteria are written for each test level (unit, integration, E2E, release)

- - Tool selection decisions are documented with a scored rationale matrix, not just tool names

- - Quality gates are defined for all four stages (PR, merge, deploy, nightly) with specific pass/fail thresholds

+ - [ ] A strategy document exists at an agreed path and `grep -cE '^### [0-9]+\.'` returns 13 (all sections Executive Summary → Revision History populated).

+ - [ ] Test pyramid target ratios are defined with concrete counts and a timeline to reach them; unit+integration+E2E percentages sum to ~100%.

+ - [ ] Entry and exit criteria are written for each level (unit, integration, E2E, release).

+ - [ ] Tool selection is documented with a scored, weighted rationale matrix — not just tool names.

+ - [ ] Quality gates are defined for all four stages (PR, merge, deploy, nightly), each with a concrete pass/fail threshold.

+ - [ ] Every Metrics & KPIs row has a non-empty Target and a tracking cadence.

- ## Reference Files (in `references/`)

+ ## Related Skills

- - **diagrams-and-worksheets.md** — Test pyramid shape diagrams, current/target state worksheets, the full 5x5 risk matrix, and the 13-section strategy document output skeleton.

- - **strategy-templates.md** — Four fully worked strategy documents (SaaS, e-commerce, API-first, media), plus a step-by-step pyramid analysis worksheet and a risk-matrix feature-inventory template.

+ - **risk-based-testing** — run first; deep dive into risk-assessment methodology and AI/LLM-specific failure classes. This skill consumes its matrix.

+ - **test-planning** — go here for a single sprint or release plan; test-strategy is the multi-quarter umbrella above it.

+ - **test-reliability** — flake root-cause triage, self-healing locators, and quarantine mechanics referenced in the pyramid-rebalance step.

+ - **qa-metrics** — full KPI definitions, DORA metrics, Test Impact Analysis, dashboards, trend analysis.

+ - **release-readiness** — go/no-go checklists, canary analysis, release confidence scoring.

+ - **ci-cd-integration** — pipeline configuration, gate implementation, and smart sharding setup.

+ - **shift-left-testing** — techniques for moving validation earlier.

+ - **ai-system-testing** — when the strategy covers AI/LLM features, defines the eval-suite layer.

+ - **compliance-testing** — when the strategy serves a regulated audience (GDPR, EU AI Act, EAA, US state laws).

- ## Related Skills

+ ## Reference Files (in `references/`)

- - **risk-based-testing** -- deep dive into risk assessment methodology, including AI/LLM-specific failure classes

- - **qa-metrics** -- detailed KPI definitions, DORA metrics, Test Impact Analysis, dashboards, and trend analysis

- - **release-readiness** -- go/no-go checklists, vendor-native canary analysis, and release confidence scoring

- - **test-planning** -- sprint-level test planning and estimation (including AI-assisted authoring caveats)

- - **ci-cd-integration** -- pipeline configuration, gate implementation, and smart sharding

- - **shift-left-testing** -- techniques for moving validation earlier

- - **ai-system-testing** -- when the strategy covers AI/LLM features, this skill defines the eval-suite layer

- - **compliance-testing** -- when the strategy serves a regulated audience (GDPR, EU AI Act, EAA, US state laws)

+ - **diagrams-and-worksheets.md** — pyramid shape diagrams, current/target state worksheets, the full 5x5 risk matrix, and the 13-section output skeleton.

+ - **strategy-templates.md** — four fully worked strategy documents (SaaS, e-commerce, API-first, media), plus a step-by-step pyramid analysis worksheet and a risk-matrix feature-inventory template.