cicd-pipeline · diff
v1.0.1 to v1.1.0
60 added, 170 removed. Audit A to A.
---
name: cicd-pipeline
description: >-
Configure testing in CI/CD pipelines for GitHub Actions, Jenkins, and GitLab CI: shards,
parallelization, wait-on health checks, and service containers. Use when configuring
tests in CI/CD pipelines (GitHub Actions, Jenkins, GitLab).
slug: cicd-pipeline
- version: 1.0.1
+ version: 1.1.0
displayName: cicd-pipeline
---
# CI/CD Pipeline Config Skill
You are an expert DevOps engineer specializing in CI/CD pipeline configuration for test automation. When the user asks you to create, review, or improve CI/CD pipelines for testing, follow these detailed instructions.
## Core Principles
1. **Fast feedback** -- Tests should provide feedback as quickly as possible.
2. **Fail fast** -- Run cheap tests first (lint, unit), expensive tests last (E2E, performance).
3. **Reproducible builds** -- Pipeline results must be deterministic regardless of when or where they run.
4. **Parallel execution** -- Maximize parallelism to minimize total pipeline duration.
5. **Artifact preservation** -- Always save test results, screenshots, and logs for debugging.
## Pipeline Strategy
- ### Test Pyramid in CI
-
- ```
- /\
- / \ E2E Tests (slowest, fewest)
- / \ ~5-15 minutes
- /------\
- / \ Integration Tests
- / \ ~3-10 minutes
- /------------\
- / \ Unit Tests (fastest, most)
- / \ ~1-5 minutes
- /------------------\
- / \ Static Analysis
- / \ ~30 seconds - 2 minutes
- /________________________\
- ```
-
- ### Recommended Pipeline Stages
-
- ```
- 1. Checkout & Install (~1 min)
- 2. Static Analysis (~1-2 min) -- lint, type check, format check
- 3. Unit Tests (~2-5 min) -- jest, pytest, junit
- 4. Build (~2-5 min) -- compile, bundle
- 5. Integration Tests (~3-10 min) -- API tests, database tests
- 6. E2E Tests (~5-15 min) -- browser tests, mobile tests
- 7. Performance Tests (~5-30 min) -- only on main/release branches
- 8. Security Scan (~3-10 min) -- SAST, dependency audit
- 9. Deploy to Staging (~2-5 min)
- 10. Smoke Tests (~2-3 min)
- 11. Report & Notify (~1 min)
- ```
+ Test pyramid in CI, stage ordering, and durations: moved to
+ [references/pipeline-strategy.md](references/pipeline-strategy.md). Rule that stays in
+ the body: **fail fast** -- order stages cheap-to-expensive (static analysis -> unit ->
+ build -> integration -> E2E -> performance), and keep stages independent so failures
+ diagnose to one stage.
## GitHub Actions
### Complete Testing Pipeline
```yaml
name: Test Pipeline
# Remaining pins below (action majors like @v4/@v3, postgres/redis/node tags,
# snyk @master) are channel examples: prefer the versions your project already
# uses; the refresh-managed table (rules + last-verified values + re-check
# procedure) is references/pinned-versions.md.
on:
push:
branches: [main, develop]
pull_request:
branches: [main, develop]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
env:
NODE_VERSION: '20'
CI: true
jobs:
lint-and-typecheck:
name: Lint & Type Check
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- run: npm ci
- name: ESLint
run: npx eslint . --max-warnings=0
- name: TypeScript Check
run: npx tsc --noEmit
- name: Prettier Check
run: npx prettier --check .
unit-tests:
name: Unit Tests
runs-on: ubuntu-latest
timeout-minutes: 10
needs: [lint-and-typecheck]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- run: npm ci
- name: Run Unit Tests
run: npx jest --coverage --ci --reporters=default --reporters=jest-junit
env:
JEST_JUNIT_OUTPUT_DIR: ./test-results/unit
- name: Upload Coverage
if: always()
uses: actions/upload-artifact@v4
with:
name: unit-coverage
path: coverage/
retention-days: 7
- name: Upload Test Results
if: always()
uses: actions/upload-artifact@v4
with:
name: unit-test-results
path: test-results/unit/
api-tests:
name: API Tests
runs-on: ubuntu-latest
timeout-minutes: 15
needs: [unit-tests]
services:
postgres:
image: postgres:16
env:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: testdb
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
redis:
image: redis:7
ports:
- 6379:6379
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- run: npm ci
- name: Run Database Migrations
run: npx prisma migrate deploy
env:
DATABASE_URL: postgresql://test:test@<db-host>:5432/testdb
- name: Start Application
run: npm run start:test &
env:
DATABASE_URL: postgresql://test:test@<db-host>:5432/testdb
REDIS_URL: redis://<redis-host>:6379
PORT: 3000
- name: Wait for Application
run: npx wait-on http://<app-host>:<port>/health --timeout 30000
- name: Run API Tests
run: npx playwright test --project=api
env:
API_BASE_URL: http://<app-host>:<port>
- name: Upload Results
if: always()
uses: actions/upload-artifact@v4
with:
name: api-test-results
path: test-results/
e2e-tests:
name: E2E Tests (${{ matrix.shard }})
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [unit-tests]
strategy:
fail-fast: false
matrix:
shard: [1/4, 2/4, 3/4, 4/4]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- run: npm ci
- name: Install Playwright Browsers
run: npx playwright install --with-deps chromium
- name: Start Application
run: npm run start:test &
- name: Wait for Application
run: npx wait-on http://<app-host>:<port> --timeout 30000
- name: Run E2E Tests (Shard ${{ matrix.shard }})
run: npx playwright test --shard=${{ matrix.shard }}
- name: Upload Test Results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-${{ strategy.job-index }}
path: |
test-results/
playwright-report/
merge-e2e-reports:
name: Merge E2E Reports
runs-on: ubuntu-latest
if: always()
needs: [e2e-tests]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- run: npm ci
- name: Download All Reports
uses: actions/download-artifact@v4
with:
pattern: e2e-results-*
path: all-results/
- name: Merge Reports
run: npx playwright merge-reports --reporter=html all-results/
- name: Upload Merged Report
uses: actions/upload-artifact@v4
with:
name: e2e-report-merged
path: playwright-report/
retention-days: 14
security-scan:
name: Security Scan
runs-on: ubuntu-latest
timeout-minutes: 10
needs: [lint-and-typecheck]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- run: npm ci
- name: npm audit
run: npm audit --audit-level=high
continue-on-error: true
- name: Run Snyk
uses: snyk/actions/node@master
continue-on-error: true
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
performance-tests:
name: Performance Tests
runs-on: ubuntu-latest
timeout-minutes: 30
if: github.ref == 'refs/heads/main'
needs: [api-tests, e2e-tests]
steps:
- uses: actions/checkout@v4
- name: Install k6
run: |
# <k6-key-id> = the CURRENT k6 apt archive signing key ID published in
# k6's official APT install instructions (key IDs rotate; a stale ID
# fails this step). Last-verified value and re-check procedure:
# references/pinned-versions.md.
sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys <k6-key-id>
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install k6
- name: Run Load Test
run: k6 run k6/scripts/load-test.js --out json=k6-results.json
env:
BASE_URL: ${{ secrets.STAGING_URL }}
- name: Upload Results
if: always()
uses: actions/upload-artifact@v4
with:
name: k6-results
path: k6-results.json
notify:
name: Notify
runs-on: ubuntu-latest
if: always()
needs: [unit-tests, api-tests, e2e-tests, security-scan]
steps:
- name: Slack Notification
uses: 8398a7/action-slack@v3
with:
status: ${{ job.status }}
fields: repo,message,commit,author,action,ref
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
```
## Jenkins Pipeline
- ```groovy
- pipeline {
- agent any
-
- environment {
- NODE_VERSION = '20'
- CI = 'true'
- }
-
- options {
- timeout(time: 60, unit: 'MINUTES')
- disableConcurrentBuilds()
- buildDiscarder(logRotator(numToKeepStr: '20'))
- }
-
- stages {
- stage('Checkout') {
- steps {
- checkout scm
- }
- }
-
- stage('Install') {
- steps {
- sh 'npm ci'
- }
- }
-
- stage('Static Analysis') {
- parallel {
- stage('Lint') {
- steps {
- sh 'npx eslint . --max-warnings=0'
- }
- }
- stage('Type Check') {
- steps {
- sh 'npx tsc --noEmit'
- }
- }
- }
- }
-
- stage('Unit Tests') {
- steps {
- sh 'npx jest --coverage --ci --reporters=default --reporters=jest-junit'
- }
- post {
- always {
- junit 'test-results/unit/*.xml'
- publishHTML(target: [
- reportDir: 'coverage/lcov-report',
- reportFiles: 'index.html',
- reportName: 'Coverage Report'
- ])
- }
- }
- }
-
- stage('E2E Tests') {
- steps {
- sh 'npx playwright install --with-deps chromium'
- sh 'npm run start:test &'
- sh 'npx wait-on http://<app-host>:<port> --timeout 30000'
- sh 'npx playwright test'
- }
- post {
- always {
- publishHTML(target: [
- reportDir: 'playwright-report',
- reportFiles: 'index.html',
- reportName: 'E2E Test Report'
- ])
- archiveArtifacts artifacts: 'test-results/**/*', allowEmptyArchive: true
- }
- }
- }
- }
-
- post {
- always {
- cleanWs()
- }
- failure {
- slackSend(
- channel: '#test-alerts',
- color: 'danger',
- message: "Build Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
- )
- }
- success {
- slackSend(
- channel: '#test-results',
- color: 'good',
- message: "Build Passed: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
- )
- }
- }
- }
- ```
+ The complete declarative Jenkinsfile (checkout/install, parallel static analysis,
+ unit tests with junit + coverage publishing, E2E with Playwright, Slack notify) is
+ externalized verbatim in [references/jenkins-pipeline.md](references/jenkins-pipeline.md) --
+ edit there when adapting; same stage order as the GitHub Actions pipeline above.
## GitLab CI
```yaml
stages:
- lint
- test
- e2e
- report
variables:
NODE_VERSION: "20"
CI: "true"
# Must equal the @playwright/test version in the project's package.json --
# the official image requires an exact match (derivation rule + refresh
# notes: references/pinned-versions.md). Derive per project, not from here.
PLAYWRIGHT_VERSION: "<match-playwright-package-version>"
.node-cache:
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
policy: pull
install:
stage: .pre
image: node:${NODE_VERSION}
script:
- npm ci
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
policy: push
lint:
stage: lint
image: node:${NODE_VERSION}
extends: .node-cache
script:
- npx eslint . --max-warnings=0
- npx tsc --noEmit
unit-tests:
stage: test
image: node:${NODE_VERSION}
extends: .node-cache
script:
- npx jest --coverage --ci
coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
artifacts:
when: always
reports:
junit: test-results/unit/junit.xml
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
paths:
- coverage/
e2e-tests:
stage: e2e
# Tag is derived from PLAYWRIGHT_VERSION above (must match @playwright/test
# exactly) -- never copy a literal tag from this template.
image: mcr.microsoft.com/playwright:v${PLAYWRIGHT_VERSION}-jammy
extends: .node-cache
parallel: 4
script:
- npm run start:test &
- npx wait-on http://<app-host>:<port> --timeout 30000
- npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
artifacts:
when: always
paths:
- playwright-report/
- test-results/
expire_in: 7 days
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == "main"
```
## Test Parallelization Strategies
- ### Shard-Based Parallelization (Playwright)
-
- ```yaml
- # Split tests evenly across N machines
- strategy:
- matrix:
- shard: [1/4, 2/4, 3/4, 4/4]
-
- steps:
- - run: npx playwright test --shard=${{ matrix.shard }}
- ```
-
- ### File-Based Parallelization (Jest)
-
- ```yaml
- # Jest automatically parallelizes by file
- steps:
- - run: npx jest --maxWorkers=4 --ci
- ```
-
- ### Tag-Based Parallelization
-
- ```yaml
- jobs:
- smoke-tests:
- steps:
- - run: npx playwright test --grep @smoke
-
- regression-tests:
- steps:
- - run: npx playwright test --grep @regression
-
- visual-tests:
- steps:
- - run: npx playwright test --grep @visual
- ```
+ Shard-based (Playwright `--shard=i/N` via matrix), file-based (Jest workers), and
+ tag-based (`--grep @tag`) strategies with snippets: moved verbatim to
+ [references/parallelization-strategies.md](references/parallelization-strategies.md).
## Best Practices
1. **Cache dependencies** -- Cache `node_modules`, `.m2`, pip packages to speed up installs.
2. **Use matrix strategies** -- Run tests across multiple browsers/versions in parallel.
3. **Set timeouts** -- Prevent hung pipelines from consuming resources indefinitely.
4. **Always upload artifacts** -- Use `if: always()` to save results even on failure.
5. **Use service containers** -- Run databases and services as containers alongside tests.
6. **Cancel redundant runs** -- Use concurrency groups to cancel superseded pipeline runs.
7. **Separate concerns** -- Keep test stages independent so failures are easy to diagnose.
8. **Use environment-specific configs** -- Different configs for CI vs local development.
9. **Notify on failure** -- Integrate Slack, email, or Teams notifications.
10. **Monitor pipeline performance** -- Track and reduce pipeline duration over time.
## Anti-Patterns to Avoid
1. **Running all tests serially** -- Parallelize wherever possible.
2. **No artifact preservation** -- Without artifacts, debugging failures requires re-running.
3. **Flaky tests in CI** -- Fix or quarantine flaky tests immediately.
4. **No timeout limits** -- Hung tests can consume runner minutes for hours.
5. **Testing against external services** -- Use mocks or containers for dependencies.
6. **Hardcoded secrets** -- Always use CI/CD secret management.
7. **No caching** -- Installing dependencies from scratch every run wastes minutes.
8. **Ignoring CI-specific config** -- Some tests need different settings in CI (headless, retries).
9. **Single point of failure** -- If one shard fails, still collect results from others.
10. **Not cleaning up** -- Stale containers, files, or processes can affect subsequent runs.
+
+ ## When to Use This Skill & NOT For
+
+ Use it when the user asks to create, review, speed up, or debug **test automation in CI/CD** (GitHub Actions, Jenkins, GitLab CI): sharding, parallelization, service containers, wait-on health checks, artifact/caching setup. NOT for (say so and stop):
+
+ - **Wrong repo / no repo**: no `.git` or no CI config and the user only wants local test runs -> local tooling, not this skill.
+ - **No network / restricted network**: runners that cannot reach npm/ghcr/mcr.microsoft.com need mirror registries and self-hosted runners configured first -- declare that prerequisite, do not hand over templates that will fail on `npm ci` or image pulls.
+ - **Deploy/infra pipeline authoring** (build-push-deploy, IaC): out of scope -- testing portion only. **Local flaky triage**: use the testing skills (`playwright-best-practices` etc.); this skill only quarantines/parallelizes in CI.
+
+ ## Minimal Worked Example (prerequisite -> invocation -> output excerpt)
+
+ **Prerequisite**: a Node repo with `npm ci`, unit tests (`npx jest`) and an existing GitHub Actions workflow file (`.github/workflows/*.yml`).
+
+ **Invocation** (what the user says):
+
+ ```text
+ "Our PR pipeline takes 40 minutes. Split the E2E suite across 4 runners
+ and add Postgres as a service container."
+ ```
+
+ **What you produce**: a diff applying the templates above -- `strategy.matrix` shards `1/4..4/4` with `npx playwright test --shard=${{ matrix.shard }}` (E2E section), plus a `services: postgres:` block with `--health-cmd pg_isready` and `DATABASE_URL: postgresql://test:test@<db-host>:5432/testdb` (api-tests job). Net effect in this shape: 40m pipeline -> ~14m (fail-fast order: lint+unit gate before E2E).
+
+ **Output excerpt** (run after the change):
+
+ ```text
+ unit-tests ✓ 3m12s
+ e2e-tests (1/4) ✓ 6m41s e2e-tests (2/4) ✓ 6m55s
+ e2e-tests (3/4) ✓ 7m02s e2e-tests (4/4) ✓ 6m48s
+ ```
+
+ ## Troubleshooting (Failure Exits)
+
+ Observable CI failure -> what it means -> exit action (each CI run ends red/green; the red run is the observable output, not a reason to improvise):
+
+ - `npm ci` fails with `E404`/registry timeout -> dependency or registry unreachable -> pin the registry/mirror in `.npmrc` or runner env; do not switch to `npm install` to mask it.
+ - Service container never healthy: job hangs then aborts -> health-check options missing or wrong port mapping -> verify `ports: 5432:5432` + `--health-cmd pg_isready` (api-tests job); the fix lands there, not in the test step.
+ - `wait-on ... timeout` (exit code 1) -> app under test did not start -> the real error is in the `Start Application` step log above it, not in wait-on.
+ - One shard red, others green (`fail-fast: false`) -> real failure in that shard until proven flaky -> read that shard's uploaded `test-results/` artifact; quarantine only after re-qualifying as flaky. Artifacts missing on failure -> step lacked `if: always()`.
+
+ ## Wrong -> Fix
+
+ | Wrong | Fix |
+ |---|---|
+ | Copying `<app-host>`/`<db-host>` placeholders literally into a real workflow | Replace with the DNS names CI provides (`localhost` for service containers on the same runner) |
+ | Copying example pins (`@v4`, `postgres:16`) into a project using different versions | Read `references/pinned-versions.md` rule + the project's own workflow/lockfile; the project always wins |
+ | `fail-fast: true` cancelling everything on one shard failure | `fail-fast: false` + merge reports (merge-e2e-reports job) so one flaky shard doesn't hide the others' results |
+ | Retrying the whole pipeline until green | Diagnose the failed stage from its artifact; retry-to-green hides real failures (anti-pattern #3) |
## Version References
Fast-changing pins are kept out of the templates above and live in one
refresh-managed table:
- `references/pinned-versions.md` — carries `lastUpdated` + `refreshInterval`
(tiered by data stability: 30/60/180 days) + a per-row confidence column.
Covers the Playwright Docker image tag rule, the k6 apt signing key ID,
GitHub Action majors, and service image tags. When filling a `<placeholder>`
in a template, read the rule from this table first; the project's own
lockfile/package.json/CI config always wins over the example values there.