ci-cd-integration · diff

v1.0 to v2.0

196 added, 301 removed. Audit A to A.

---

name: ci-cd-integration

description: >-

- Design CI/CD pipelines for test execution. Covers GitHub Actions and GitLab CI

- pipeline templates, parallelism and sharding strategies, artifact management,

- flaky test quarantine, test result publishing, and quality gates. Includes

- copy-paste-ready workflow files for Playwright, Jest, and multi-stage pipelines.

+ Design CI/CD pipelines that run test suites. Covers GitHub Actions and GitLab CI

+ templates, parallelism and sharding, artifact management, flaky-test quarantine,

+ test-result publishing, coverage quality gates, OIDC keyless deploy, and copy-paste

+ workflows for Playwright, Jest, and multi-stage pipelines.

Use when: "CI/CD," "GitHub Actions," "pipeline," "test in CI," "GitLab CI,"

- "continuous integration," "test automation pipeline."

- Related: playwright-automation, qa-metrics, test-reliability, coverage-analysis.

+ "continuous integration," "test automation pipeline," "shard tests in CI."

+ Not for: per-test flaky healing at runtime — use test-reliability; go/no-go release

+ decisions and smoke-test checklists — use release-readiness; test-result dashboards

+ and trend reporting — use qa-metrics.

+ Related: playwright-automation, qa-metrics, test-reliability, coverage-analysis, release-readiness.

license: MIT

metadata:

author: kindlmann

- version: "1.0"

+ version: "2.0"

category: infrastructure

---

- Design CI/CD pipelines for test execution. Covers GitHub Actions and GitLab CI pipeline templates, parallelism and sharding, artifact management, flaky test quarantine, and quality gates. Use this skill over others when the question is about running tests in a pipeline rather than writing them.

+ A 20-minute serial suite on every push destroys developer velocity; a green pipeline that retries flaky tests three times hides the race condition until it ships. This skill produces CI/CD pipelines that run the right tests at the right trigger, shard them across runners, store traces and reports as evidence, quarantine flaky tests instead of masking them, and gate merges on real coverage numbers. Use this skill when the question is about running tests in a pipeline, not writing them.

</objective>

## Discovery Questions

- 1. **Which CI platform?** GitHub Actions, GitLab CI, CircleCI, Jenkins? This skill focuses on GitHub Actions and GitLab CI.

- 2. **What test types need to run?** Unit, integration, E2E, visual regression, performance? Each has different resource and timing needs.

- 3. **What is the current CI duration?** If over 10 minutes, parallelism and sharding are essential.

- 4. **How many developers push per day?** High-frequency teams need aggressive concurrency controls and caching.

- 5. **What triggers should run which tests?** Not every push needs a full E2E suite.

- 6. **Check `.agents/qa-project-context.md` first.** Respect existing CI conventions and infrastructure constraints.

+ Check `.agents/qa-project-context.md` first — if it exists, use it and skip anything already answered there (especially `team_maturity` and existing CI conventions). Then:

- ---

+ 1. **Which CI platform?** GitHub Actions, GitLab CI, CircleCI, Jenkins? This skill ships templates for GitHub Actions and GitLab CI.

+ 2. **What test types need to run?** Unit, integration, E2E, visual, performance? Each has different resource and timing needs.

+ 3. **What is the current CI duration?** Over 10 minutes means parallelism and sharding are mandatory, not optional.

+ 4. **How many developers push per day?** High-frequency teams need aggressive concurrency cancellation and caching.

+ 5. **What triggers should run which tests?** Not every push needs a full E2E suite — map triggers to suites before writing YAML.

- ## Core Principles

+ ### Calibrate to team maturity

- 1. **Fast feedback: right tests at the right time.** Unit tests on every push (under 2 min). E2E on PRs (under 10 min). Full suite on merge and nightly.

- 2. **Parallel first: shard tests across workers.** A 20-minute serial suite becomes 5 minutes across 4 shards. Always worth the runner cost.

- 3. **Artifacts are evidence.** Every CI run must store traces, screenshots, coverage reports, and HTML reports. Without artifacts, CI failures are undebuggable.

- 4. **Flaky tests need quarantine, not retries.** Retrying hides the problem. Move flaky tests to a separate non-blocking job, track them, and fix them.

- 5. **Quality gates at every stage.** Define what must pass before code moves forward. Gates get stricter as code gets closer to production.

+ Set `team_maturity` in `.agents/qa-project-context.md`; pick the matching pipeline shape:

+ - **startup** — one job: lint + unit + one E2E smoke on PR. Fast feedback over completeness.

+ - **growing** — separate jobs for unit, integration, E2E. Parallelization, artifact uploads, result publishing, flaky quarantine.

+ - **established** — full matrix: sharded E2E, multi-environment promotion gates, perf and security scans, deploy-gated checks, SLA-backed pipelines.

---

- > **Calibrate to your team maturity** (set `team_maturity` in `.agents/qa-project-context.md`):

- > - **startup** — Single pipeline job: lint + unit tests + one E2E smoke test on PR. Fast feedback over completeness.

- > - **growing** — Separate jobs for unit, integration, E2E. Parallelization, artifact uploads, test result publishing, flaky test quarantine.

- > - **established** — Full matrix: parallelized E2E sharding, multi-environment promotion gates, performance and security scans, deployment-gated quality checks, SLA-backed pipelines.

+ ## Core Principles

+ 1. **Fast feedback: right tests at the right time.** Unit tests on every push (under 2 min). E2E on PRs (under 10 min). Full suite on merge and nightly. The trigger-to-suite map below is the contract.

+ 2. **Parallel first: shard tests across workers.** A 20-minute serial suite becomes 5 minutes across 4 shards. Always worth the runner cost.

+ 3. **Artifacts are evidence.** Every run stores traces, screenshots, coverage, and HTML reports. Without artifacts, a CI failure is an undebuggable "reproduce locally" cycle.

+ 4. **Flaky tests need quarantine, not retries.** Retrying hides the problem — the test passes on retry, the report is green, the race condition persists. Move flaky tests to a non-blocking job, track them, fix the root cause.

+ 5. **Quality gates get stricter toward production.** Define what must pass at each stage; PR gate is fast and cheap, deploy gate is comprehensive.

+ 6. **Read thresholds from config, not from bash.** Let the test runner enforce coverage via its own `coverageThreshold`/`thresholds` and exit non-zero. Scraping percentages out of stdout with regex is fragile across runner versions.

---

## Pipeline Architecture

```

- Push to branch:

- ┌─────────────┐

- │ lint + types │ (30s)

- └──────┬──────┘

- │

- ┌──────▼──────┐

- │ unit tests │ (1-2 min)

- └──────┬──────┘

- │ (pass)

- ▼

- PR opened/updated:

- ┌──────────────┐

- │ integration │ (2-3 min)

- └──────┬───────┘

- │

- ┌──────▼──────────┐

- │ E2E (sharded) │ (5-8 min)

- └──────┬──────────┘

- │

- ┌──────▼──────┐

- │ merge report │

- └─────────────┘

- Merge to main:

- ┌───────────┐ ┌──────────┐ ┌──────────┐

- │ full E2E │ │ visual │ │ perf │

- └─────┬─────┘ └────┬─────┘ └────┬─────┘

- └──────────────┼─────────────┘

- ▼

- ┌─────────────┐

- │ deploy │

- └─────────────┘

- Nightly (scheduled):

- full suite + security scan + a11y audit + flaky quarantine

+ Push to branch: lint+types (30s) → unit (1-2m)

+ PR opened: + integration (2-3m) → E2E sharded (5-8m) → merge report

+ Merge to main: full E2E ∥ visual ∥ perf budget → deploy (OIDC)

+ Nightly (cron): full suite + npm audit + axe a11y + flaky quarantine

```

- ### What Runs When

+ ### What runs when

- | Trigger | Tests | Max Duration |

- |---------|-------|-------------|

+ | Trigger | Tests | Max duration |

+ |---------|-------|--------------|

| Push to branch | lint, type-check, unit | 2 min |

| PR opened/updated | + integration, E2E smoke | 10 min |

| Merge to main | + full E2E, visual, perf budget | 15 min |

| Nightly schedule | full suite, security, a11y, flaky quarantine | 30 min |

| Release tag | full suite, smoke against staging | 20 min |

---

- ## GitHub Actions Templates

+ ## GitHub Actions

- For complete, copy-paste-ready workflow files, see `references/github-actions-templates.md`.

+ For complete copy-paste workflow files (unit, sharded Playwright E2E, full pipeline, nightly, PR gate), see `references/github-actions-templates.md`.

- ### Key Concepts

+ ### Action versions (June 2026)

- **Concurrency groups** prevent wasted runs when a branch gets multiple pushes:

+ Pin to the current major and let Dependabot bump them. The `actions/*` family runs on the Node 24 runner; Node 20 is deprecated on GH-hosted runners.

+ | Action | Current major | Notes |

+ |--------|---------------|-------|

+ | `actions/checkout` | `@v6` | |

+ | `actions/setup-node` | `@v6` | v5+ auto-caches only when `packageManager` is set; use `cache: npm` to be explicit |

+ | `actions/cache` | `@v5` | new cache service v2 backend |

+ | `actions/upload-artifact` | `@v7` | v7 can upload unzipped (`archive: false`) |

+ | `actions/download-artifact` | `@v7` | pair with upload-artifact major |

+ | `dorny/test-reporter` | `@v3` | v3 requires Node 24 runner; reporter keys unchanged |

+ | `dorny/paths-filter` | `@v3` | |

+ | `marocchino/sticky-pull-request-comment` | `@v3` | |

+ | `slackapi/slack-github-action` | `@v2` | floating major; see notification note before adopting v3 |

+ For supply-chain-sensitive pipelines, pin third-party actions (dorny, marocchino, slackapi, knapsack) to a full-length commit SHA with a version comment, and let Dependabot update the SHA: `uses: dorny/test-reporter@<40-char-sha> # v3.0.0`. First-party `actions/*` are lower risk; tags are acceptable there.

+ ### Key concepts

+ **Concurrency groups** cancel wasted runs when a branch gets multiple pushes:

```yaml

concurrency:

group: tests-${{ github.ref }}

cancel-in-progress: true

```

- **Matrix strategy** for sharding tests across runners:

+ **Matrix sharding** across runners:

```yaml

strategy:

fail-fast: false

matrix:

shard: [1, 2, 3, 4]

steps:

- run: npx playwright test --shard=${{ matrix.shard }}/4

```

- **Smarter sharding** — past 10–15 shards, naïve hash-based splitting wastes runner time on uneven shards. Use a balancer:

- - **`knapsack-pro`** (timing-data based, supports Playwright/Jest/Cypress/RSpec) — distributes by historical duration.

- - **CloudBees Smart Tests** (formerly **Launchable**) — ML-based prioritization + Test Impact Analysis; runs only the tests likely to fail given the diff.

- - **Datadog Test Optimization** — TIA + flake management; shard-balancing by historical time.

- - **Trunk Flaky Tests** — flake-aware quarantine + retry budgeting.

- For self-hosted runners on Kubernetes, use **Actions Runner Controller (`gha-runner-scale-set`)** — Helm-installed, auto-scales runner pods per workflow. Replaces the deprecated `actions-runner-controller/runner-deployment` CRD; current name is `arc-runner-set`.

- > **GitHub Actions versioning (May 2026):** the `actions/*` v4 family (`checkout`, `setup-node`, `cache`, `upload-artifact`, `download-artifact`) has rolled to v5/v6/v7/v8 on the Node 24 runner; Node 20 is deprecated on GH-hosted runners. Examples below pin v4/v5 for stability — verify the current major before adopting in new pipelines and use Dependabot to keep them current.

- **Caching** to avoid reinstalling on every run:

+ **Caching** browsers so they aren't re-downloaded every run:

```yaml

- # Node modules: handled by setup-node's cache option

- - uses: actions/setup-node@v4

+ - uses: actions/setup-node@v6

with: { node-version: 22, cache: npm }

- # Playwright browsers: cache separately

- name: Cache Playwright browsers

id: playwright-cache

- uses: actions/cache@v4

+ uses: actions/cache@v5

with:

path: ~/.cache/ms-playwright

key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

- name: Install Playwright browsers

if: steps.playwright-cache.outputs.cache-hit != 'true'

run: npx playwright install --with-deps chromium

```

- **Artifact management** for reports and traces:

+ **Artifacts** for reports and traces, and **merging sharded reports** into one HTML report — see `references/github-actions-templates.md` (E2E workflow). The merge job uses `actions/download-artifact@v7` with `pattern: test-results-*` then `npx playwright merge-reports --reporter=html`.

- ```yaml

- - uses: actions/upload-artifact@v4

- if: ${{ !cancelled() }}

- with:

- name: test-results-${{ matrix.shard }}

- path: |

- test-results/

- playwright-report/

- coverage/

- retention-days: 7

- ```

+ ### Smarter sharding at scale

- **Merging sharded reports** into a single HTML report:

+ Past 10–15 shards, naïve hash-based splitting wastes runner time on uneven shards. Use a timing-aware balancer:

- ```yaml

- merge-reports:

- needs: e2e

- if: ${{ !cancelled() }}

- runs-on: ubuntu-latest

- steps:

- - uses: actions/checkout@v4

- - uses: actions/setup-node@v4

- with: { node-version: 22, cache: npm }

- - run: npm ci

- - uses: actions/download-artifact@v4

- with: { pattern: 'test-results-*', path: all-results }

- - run: npx playwright merge-reports --reporter=html all-results

- - uses: actions/upload-artifact@v4

- with: { name: playwright-report, path: playwright-report/, retention-days: 14 }

- ```

+ - **`knapsack-pro`** — timing-data based, supports Playwright/Jest/Cypress/RSpec; distributes by historical duration.

+ - **CloudBees Smart Tests** (formerly **Launchable**) — ML prioritization + Test Impact Analysis; runs only the tests likely to fail for the diff.

+ - **Datadog Test Optimization** — TIA + flake management; shard-balancing by historical time.

+ - **Trunk Flaky Tests** — flake-aware quarantine + retry budgeting.

- ### Status Checks Configuration

+ For self-hosted runners on Kubernetes, use **Actions Runner Controller** (`arc-runner-set` / `gha-runner-scale-set`) — Helm-installed, auto-scales runner pods per workflow. Replaces the deprecated `runner-deployment` CRD.

- Required status checks protect your main branch. Configure in GitHub repo settings:

+ ### Required status checks

- 1. Go to Settings > Branches > Branch protection rules

- 2. Enable "Require status checks to pass before merging"

- 3. Add these required checks: `lint`, `unit-tests`, `e2e` (all shards)

- 4. Enable "Require branches to be up to date before merging"

+ Protect main in Settings → Branches → Branch protection rules: enable "Require status checks to pass before merging," add `lint`, `unit-tests`, and `e2e` (all shards) as required checks, and enable "Require branches to be up to date."

---

- ## GitLab CI Templates

- ```yaml

- # .gitlab-ci.yml

- stages: [validate, test, e2e, deploy]

- variables:

- NODE_ENV: test

- npm_config_cache: '$CI_PROJECT_DIR/.npm'

- cache:

- key: ${CI_COMMIT_REF_SLUG}

- paths: [.npm/, node_modules/]

- lint:

- stage: validate

- image: node:22-alpine

- script: [npm ci --prefer-offline, npm run lint, npm run type-check]

- unit-tests:

- stage: test

- image: node:22-alpine

- script: [npm ci --prefer-offline, 'npm run test:ci -- --coverage']

- coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'

- artifacts:

- when: always

- paths: [coverage/]

- reports:

- junit: junit.xml

- coverage_report: { coverage_format: cobertura, path: coverage/cobertura-coverage.xml }

+ ## GitLab CI

- e2e-tests:

- stage: e2e

- image: mcr.microsoft.com/playwright:v1.59.1-noble

- parallel: 4 # GitLab provides CI_NODE_INDEX and CI_NODE_TOTAL automatically

- script:

- - npm ci --prefer-offline

- - npm run build

- - npm start &

- - npx wait-on http://localhost:3000 --timeout 60000

- - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

- artifacts:

- when: always

- paths: [test-results/, playwright-report/]

- expire_in: 7 days

- reports:

- junit: test-results/junit.xml # GitLab parses this and shows results in MR UI

- rules:

- - if: $CI_PIPELINE_SOURCE == "merge_request_event"

- - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

+ For the full pipeline, see `references/gitlab-ci-template.md`. Key points:

- deploy-staging:

- stage: deploy

- script: [./deploy.sh staging]

- rules: [{ if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' }]

- needs: [unit-tests, e2e-tests]

- ```

+ - Stages `[validate, test, e2e, deploy]`; `node:22-alpine` for lint/unit, `mcr.microsoft.com/playwright:v1.60.0-noble` for E2E (keep this pinned to your installed `@playwright/test` minor).

+ - Parallel sharding: `parallel: 4` exposes `CI_NODE_INDEX`/`CI_NODE_TOTAL`; run `npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL`.

+ - Coverage: emit a cobertura `coverage_report` artifact and a `junit` report; GitLab reads the percentage and test results from those. The legacy `coverage:` stdout regex is a fragile fallback across Jest versions — prefer the cobertura report.

---

## Advanced Patterns

- ### Test Result Publishing to PR Comments

- Post test results directly on the PR for visibility:

+ ### Test result publishing to PR comments

```yaml

- # Add after test step in GitHub Actions

- name: Publish test results

- uses: dorny/test-reporter@v1

+ uses: dorny/test-reporter@v3

if: ${{ !cancelled() }}

with:

name: Test Results

path: test-results/junit.xml

- reporter: jest-junit # or java-junit for Playwright

- - name: Comment coverage on PR

- uses: marocchino/sticky-pull-request-comment@v2

- if: github.event_name == 'pull_request'

- with:

- header: coverage

- path: coverage/coverage-summary.md

+ reporter: jest-junit # use java-junit for a Playwright JUnit report

```

- ### Conditional Test Execution

- Only test what changed to save CI time:

- ```yaml

- - name: Detect changed files

- id: changes

- uses: dorny/paths-filter@v3

- with:

- filters: |

- frontend:

- - 'src/**'

- - 'e2e/**'

- backend:

- - 'api/**'

- - 'lib/**'

- config:

- - 'package.json'

- - 'playwright.config.ts'

- - name: Run E2E tests

- if: steps.changes.outputs.frontend == 'true' || steps.changes.outputs.config == 'true'

- run: npx playwright test

- - name: Run API tests

- if: steps.changes.outputs.backend == 'true' || steps.changes.outputs.config == 'true'

- run: npm run test:api

- ```

+ For the sticky coverage PR comment (`marocchino/sticky-pull-request-comment@v3`), see `references/github-actions-templates.md` (PR Quality Gate).

- ### Test Timing Optimization

+ ### Conditional test execution

- Playwright `--shard` distributes tests by file, balancing duration from previous runs. For custom balancing, generate timing data: `npx playwright test --reporter=json | jq '[.suites[].specs[] | {file: .file, duration: .tests[].results[].duration}]'`. For Jest, use `jest-slow-test-reporter` to identify slow tests.

+ Only test what changed. Use `dorny/paths-filter@v3` to set outputs, then gate steps on them — see `references/github-actions-templates.md` (Conditional execution).

- ### Flaky Test Quarantine

+ ### Flaky test quarantine

- Separate flaky tests into a non-blocking job:

+ Separate flaky tests into a non-blocking job so they run in CI but don't block merges:

```yaml

- e2e-stable:

- runs-on: ubuntu-latest

+ e2e-stable: # required for merge

steps:

- run: npx playwright test --grep-invert @flaky

- # This job is required for merge

- e2e-quarantine:

- runs-on: ubuntu-latest

- continue-on-error: true # Non-blocking

+ e2e-quarantine: # non-blocking

+ continue-on-error: true

steps:

- run: npx playwright test --grep @flaky

- - name: Report flaky results

- if: failure()

- run: |

- echo "::warning::Quarantined tests failed. Review and fix or remove."

+ - if: failure()

+ run: echo "::warning::Quarantined tests failed. Review and fix or remove."

```

- Tag flaky tests in your test files:

+ Tag flaky tests at the source so the grep splits them:

```typescript

test('sometimes fails due to race condition @flaky', async ({ page }) => {

- // This test is quarantined -- runs in CI but doesn't block merges

+ // runs in CI but doesn't block merges

});

```

- Track flaky tests over time. If a quarantined test passes 10 consecutive runs, remove the `@flaky` tag.

+ If a quarantined test passes 10 consecutive runs, remove the `@flaky` tag. For runtime self-healing of a single flaky test (selector recovery, auto-retry policy), use `test-reliability`.

- ### Cache Strategies

+ ### Cache strategies

- | Layer | Path | Cache Key |

+ | Layer | Path | Cache key |

|-------|------|-----------|

| Node modules | (handled by `setup-node` `cache: npm`) | automatic |

| Playwright browsers | `~/.cache/ms-playwright` | `pw-{os}-{hash(package-lock.json)}` |

| Build cache (Next.js) | `.next/cache` | `nextjs-{os}-{hash(lockfile)}-{hash(src)}` |

| Test fixtures | `e2e/fixtures/.cache` | `test-data-{hash(seed.sql)}` |

- Use `actions/cache@v4` for layers 2-4. Add `restore-keys` for build caches to allow partial matches.

+ Use `actions/cache@v5` for layers 2–4; add `restore-keys` on build caches for partial matches.

- ### Slack/Teams Notification on Failure

+ ### OIDC keyless deploy

- Use `slackapi/slack-github-action@v2.0.0` with `webhook-type: incoming-webhook`. Condition on `if: failure() && github.ref == 'refs/heads/main'` so notifications only fire for main branch failures. See `references/github-actions-templates.md` (Nightly Full Suite) for a complete example.

+ Don't store a long-lived `DEPLOY_TOKEN`. Use GitHub Actions OIDC to assume a cloud role for short-lived credentials — nothing static to leak or rotate:

+ ```yaml

+ deploy:

+ permissions:

+ id-token: write # request the OIDC JWT

+ contents: read

+ steps:

+ - uses: aws-actions/configure-aws-credentials@v6

+ with:

+ role-to-assume: arn:aws:iam::123456789012:role/gha-deploy

+ aws-region: eu-central-1

+ - run: ./deploy.sh production # uses short-lived STS creds, no static secret

+ ```

+ The IAM role's trust policy pins the `sub` claim to your repo and branch. GCP (`google-github-actions/auth`) and Azure (`azure/login`) have equivalent OIDC flows.

+ ### Slack/Teams notification on failure

+ Use `slackapi/slack-github-action@v2` with `webhook-type: incoming-webhook`, gated on `if: failure() && github.ref == 'refs/heads/main'` so only main-branch failures notify. Before moving to `@v3`, note v3 changed payload handling for workflow-trigger webhooks (no longer flattened/stringified) — verify your payload against the v3 docs first. Full example in `references/github-actions-templates.md` (Nightly Full Suite).

---

## Quality Gates

- ### Gate Definitions

|------|------|-----------------|-----------|

- ### PR Gate (fast, under 3 minutes)

+ ### PR Gate (under 3 minutes)

+ Enforce the coverage floor in the test runner's config, not in bash. In `jest.config.js` (or `vitest.config.ts` `coverage.thresholds`):

+ ```javascript

+ coverageThreshold: { global: { lines: 80, statements: 80, branches: 70 } }

+ ```

+ Then `jest --coverage` exits non-zero when coverage drops, so the job fails with no extra script. If you must read the number in CI (e.g. to print it), have Jest emit `json-summary` and read the file — there is no `coverage-summary` CLI:

```yaml

- pr-gate:

- runs-on: ubuntu-latest

- steps:

- - run: npm run lint

- - run: npm run type-check

- - run: npm test -- --ci --coverage

- - run: |

- COVERAGE=$(npx coverage-summary --json | jq '.total.lines.pct')

- if (( $(echo "$COVERAGE < 80" | bc -l) )); then

- echo "::error::Coverage $COVERAGE% is below 80% threshold"

- exit 1

- fi

+ - run: npm test -- --ci --coverage # exits 1 if below coverageThreshold

+ - name: Print coverage (optional)

+ run: |

+ PCT=$(jq '.total.lines.pct' coverage/coverage-summary.json)

+ echo "Line coverage: ${PCT}%"

```

- ### Merge Gate (comprehensive, under 10 minutes)

+ (`json-summary` reporter writes `coverage/coverage-summary.json`. For nyc/c8 projects, `nyc report --reporter=text-summary`. The standalone `istanbul` CLI is deprecated — don't use `istanbul report`.)

- Requires PR Gate + E2E smoke tests. Configure as required status checks in branch protection.

+ ### Merge Gate (under 10 minutes)

- ### Deploy Gate (full confidence, under 15 minutes)

+ PR Gate + E2E smoke. Configure as required status checks in branch protection.

+ ### Deploy Gate (under 15 minutes)

+ Needs `[unit-tests, e2e-tests, visual-tests]`, then a perf budget check (`npx lhci autorun` / `lhci assert --config=lighthouserc.json`) before the OIDC deploy step above.

+ ### Nightly Gate (up to 30 minutes)

+ Full E2E across all browsers, security scan, a11y audit, flaky quarantine. Wire the security and a11y steps as real jobs, not just prose:

```yaml

- deploy-gate:

- needs: [unit-tests, e2e-tests, visual-tests]

- runs-on: ubuntu-latest

- steps:

- - name: Check performance budget

- run: |

- npx lighthouse-ci assert --config=lighthouserc.json

- - name: Deploy to production

- if: success()

- run: ./deploy.sh production

+ - run: npm audit --audit-level=high # fails on high/critical advisories

+ - run: npx playwright test --grep @a11y # specs that call @axe-core/playwright

```

- ### Nightly Gate (thorough, up to 30 minutes)

+ Where the `@a11y`-tagged specs use `@axe-core/playwright`:

- ```yaml

- on:

- schedule:

- - cron: '0 2 * * *' # 2am UTC daily

+ ```typescript

+ import AxeBuilder from '@axe-core/playwright';

+ test('home page has no a11y violations @a11y', async ({ page }) => {

+ await page.goto('/');

+ const results = await new AxeBuilder({ page }).analyze();

+ expect(results.violations).toEqual([]);

+ });

```

- Runs everything: full E2E suite across all browsers, security scan (npm audit, Snyk), accessibility audit (axe-core), and the flaky quarantine suite. Results go to Slack, not as blocking checks.

+ Results go to Slack, not as blocking checks.

---

## Anti-Patterns

### 1. Running all tests on every commit

- A 20-minute full suite on every push destroys developer velocity. Use the pipeline architecture above: fast tests on push, comprehensive tests on PR and merge.

+ A 20-minute full suite on every push destroys velocity. Use the trigger-to-suite map: fast tests on push, comprehensive on PR and merge.

### 2. No artifact storage

- When CI tests fail, developers need traces, screenshots, and logs to debug. Without artifacts, every failure requires a "reproduce locally" cycle that wastes hours.

+ Without traces, screenshots, and logs, every CI failure becomes a "reproduce locally" cycle that wastes hours. Upload artifacts on `if: ${{ !cancelled() }}`.

### 3. Retrying flaky tests without tracking them

- Adding `retries: 3` hides flakiness. The test passes on retry, the report is green, but the underlying race condition persists. Quarantine flaky tests, track them in a dashboard, and fix the root cause.

- ### 4. CI-only failures without local reproduction steps

- If a test only fails in CI, document why (e.g., different timezone, missing env var, screen resolution). Add a `Makefile` or script that replicates CI conditions locally:

+ `retries: 3` hides flakiness — the report is green but the race condition persists. Quarantine, track, fix the root cause.

- ```bash

- # Reproduce CI environment locally

- docker run --rm -v $(pwd):/work -w /work \

- mcr.microsoft.com/playwright:v1.49.0-noble \

- npx playwright test --project=chromium

- ```

+ ### 4. CI-only failures without local reproduction

+ If a test only fails in CI, document why (timezone, missing env var, screen resolution) and add a script that replicates CI locally with the **same** Playwright image you run in CI — don't pin a stale image. See `references/github-actions-templates.md` (Local repro).

### 5. Shared state between CI jobs

- Jobs that depend on files from other jobs without using artifacts or proper `needs` dependencies. Each job starts fresh. Use `actions/upload-artifact` and `actions/download-artifact` to pass data.

+ Jobs that read files from sibling jobs without artifacts or `needs`. Each job starts fresh; pass data via `upload-artifact`/`download-artifact`.

### 6. No concurrency controls

- Multiple CI runs for the same branch waste resources. Always use concurrency groups with `cancel-in-progress: true`.

+ Multiple runs for the same branch waste runners. Always use a concurrency group with `cancel-in-progress: true`.

### 7. Hardcoded secrets in workflow files

- Never put tokens, passwords, or API keys in YAML. Use GitHub Actions secrets (`${{ secrets.MY_SECRET }}`) or GitLab CI/CD variables.

+ Never put tokens, passwords, or keys in YAML. Use repo secrets (`${{ secrets.X }}`) or GitLab CI/CD variables — and prefer OIDC keyless auth over any long-lived deploy token.

### 8. Ignoring job timeouts

- A stuck test can consume a runner for hours. Always set `timeout-minutes` on jobs and `actionTimeout` / `navigationTimeout` in test configs.

+ A stuck test can hold a runner for hours. Set `timeout-minutes` on every job and `actionTimeout`/`navigationTimeout` in the Playwright config.

---

+ ## Verification

+ Prove the pipeline before relying on it. Smallest check first:

+ 1. **Lint the workflow syntax** — `actionlint .github/workflows/*.yml` catches expression, `needs`, and shell-quoting errors before they fail at runtime. Add a `yamllint .github/workflows/` pass for indentation. Run actionlint as a job too.

+ 2. **Dry-run a job locally** — `act -j unit-tests` runs the job in a container so you can iterate without pushing.

+ 3. **Confirm required checks appear** — push to a throwaway branch, open a draft PR, and verify the expected check runs (`lint`, `unit-tests`, `e2e`) show up and that the coverage gate fails when you drop coverage below the threshold.

+ 4. **Verify artifacts** — download the run's artifacts from the Actions UI (or `gh run download <id>`) and confirm `playwright-report/` and traces are present.

+ ---

## Done When

- - Pipeline runs unit, integration, and E2E tests on every PR

- - Flaky tests quarantined in a non-blocking job with a tracking ticket — not silently retried without a plan

- - Test artifacts (reports, screenshots, traces) uploaded and accessible for every CI run

- - Concurrency groups configured to prevent redundant runs on the same branch

- - Secrets managed via the CI secrets store — no tokens or passwords hardcoded in workflow files

+ - A trigger map exists: push runs lint+unit only; the PR workflow gates E2E behind `if: github.event_name == 'pull_request'` (verify the YAML, not "integration runs somewhere").

+ - `actionlint .github/workflows/*.yml` exits 0.

+ - A non-blocking quarantine job runs `--grep @flaky` with `continue-on-error: true`; the stable job runs `--grep-invert @flaky` and is in the required checks list.

+ - Test artifacts (reports, screenshots, traces) upload on `if: ${{ !cancelled() }}` with an explicit `retention-days`/`expire_in`.

+ - Concurrency groups with `cancel-in-progress: true` are set on the PR/test workflows.

+ - Coverage is enforced by the runner's `coverageThreshold`/`thresholds` (job exits non-zero below the floor) — no `coverage-summary` CLI scrape.

+ - Branch protection lists `lint`, `unit-tests`, and `e2e` as required status checks.

+ - No long-lived deploy token in YAML — deploy uses OIDC (`id-token: write` + cloud role) or, at minimum, a secrets-store reference.

---

## Related Skills

- - **playwright-automation** -- E2E test framework setup, Page Object Model, and test patterns.

- - **qa-metrics** -- Test result dashboards, coverage tracking, and flakiness monitoring.

- - **test-reliability** -- Strategies for reducing test maintenance and auto-recovering from UI changes.

- - **test-strategy** -- Overall test planning, pyramid design, and risk-based test selection.

+ - **playwright-automation** — writing the E2E tests, Page Object Model, and the `playwright.config.ts` whose sharding/timeouts this pipeline drives.

+ - **test-reliability** — runtime self-healing of one flaky test (selector recovery, retry policy); go there to fix a flaky test, come here to quarantine it in CI.

+ - **qa-metrics** — turning the JUnit/coverage artifacts this pipeline produces into dashboards and flakiness trends.

+ - **release-readiness** — the human go/no-go decision and release checklist that consumes these gate results; this skill builds the gates, that one decides on them.

+ - **coverage-analysis** — finding the coverage gaps and setting the threshold this skill's PR gate enforces.

- For complete, copy-paste-ready GitHub Actions workflow files, see `references/github-actions-templates.md`.

+ ## Reference Files (in `references/`)

+ - **github-actions-templates.md** — copy-paste unit, sharded Playwright E2E (+ report merge), full pipeline, nightly (Slack + audit + axe), and PR quality-gate workflows, plus conditional execution and local-repro snippets.

+ - **gitlab-ci-template.md** — full `.gitlab-ci.yml` with parallel sharding, cobertura coverage, and JUnit MR reporting.