git:20260724.d4db756 to git:20260725.4f623d0

65 added, 50 removed. Audit A to A.

---

name: ci-test-job-conventions

- description: "Pure-reference for cross-CI test workflow conventions - when to shard (and how many shards), retry policy (which failures are safe to retry), flake quarantine integration, artifact lifecycle (retention / structure), per-trigger filtering (per-PR vs per-merge vs nightly), concurrency control patterns, JUnit reporting standards. Use as the team's reference doc for CI test workflow design across GitHub Actions / GitLab CI / Jenkins / CircleCI."

+ description: "Pure-reference for cross-CI test workflow conventions - when to shard (and how many shards), retry policy (which failures are safe to retry), flake-quarantine integration, artifact retention, per-trigger cadence (per-PR vs per-merge vs nightly), concurrency-cancel patterns, per-job timeouts, secret management, and cross-CI portability. Use as the team's reference for CI test-workflow design across GitHub Actions / GitLab CI / Jenkins / CircleCI; per-CI reporting and per-language reporter / cache-key lookups live in references/."

---

# ci-test-job-conventions

## Overview

Per-CI platform skills (GitHub Actions / GitLab CI / Jenkins /

CircleCI) cover **how to express** workflows. This skill covers

**what to express** - the cross-platform conventions that apply

regardless of CI tool.

+ ## When to use

+ - Designing the CI test workflow for a new repo or service.

+ - Auditing an existing CI config for anti-patterns - retries that

+ mask flake, no concurrency-cancel, artifacts retained forever.

+ - PR review of a workflow / `.gitlab-ci.yml` / `Jenkinsfile` change.

+ - Standardizing test-job conventions across teams on different CI tools.

+ ## How to use this reference

+ 1. **Size the suite, pick the shard count** from the sharding matrix

+ (§1) and set the per-job timeout for each job type (§8).

+ 2. **Set the retry policy** - which failure classes are safe to retry

+ vs. quarantine as flake (§2, §3).

+ 3. **Add concurrency-cancel** so a new push cancels the superseded run (§6).

+ 4. **Tier the per-trigger cadence** - what runs per-PR vs per-merge vs

+ nightly vs pre-release (§5).

+ 5. **Emit JUnit XML and set artifact retention** - the reporting and

+ cache lookup tables live in

+ [references/junit-and-cache-lookups.md](references/junit-and-cache-lookups.md);

+ retention bands in §4.

+ 6. **Guard secrets** with the CI secret store, never in-repo (§7).

+ Then re-check the workflow against the conventions below.

## §1 - When to shard

Sharding splits a test suite across N parallel jobs. Decision

matrix:

| Suite total runtime | Sharding recommendation |

|---------------------|-----------------------------------------------|

| < 2 min | None. Overhead exceeds benefit. |

| 2-10 min | Optional. Shard if PR feedback time matters. |

| 10-30 min | 2-4 shards. |

| > 30 min | 4-8 shards. Investigate the suite - may be too big. |

| > 60 min | 8+ shards + investigate suite refactoring (per `e2e-suite-budget`). |

Sharding cost-equivalent is N parallel × ~runtime/N - same total

CPU-time, faster wall-clock.

## §2 - Retry policy

Distinguish retry classes:

| Failure class | Retry? | Pattern |

|------------------------------|------------------|---------------------------------------------|

| Runner died / system failure | Yes (1-2x) | CI platform's retry-on-runner-failure. |

| Network timeout to dependency | Yes (1x) | Test framework retry; flag for analysis. |

| Test flake (passed on retry) | No | Mark + quarantine via `flaky-test-quarantine`. |

| Test consistently fails | No | Real bug; investigate. |

**Rule:** Maximum 1 framework-level retry. More retries hide flake.

## §3 - Flake quarantine integration

Failed-on-first-run-passed-on-retry tests are flake. Pattern:

1. Test fails → CI marks `@flaky` tag.

2. Continued failures over N runs → quarantine

(move to separate suite that doesn't gate).

3. Periodic review of quarantined tests → fix or delete.

Per `flaky-test-quarantine` (in the qa-flake-triage plugin)

for the workflow.

## §4 - Artifact lifecycle

```yaml

# Recommended retention per artifact type

test-results: 14 days # short-term debugging

coverage-reports: 30 days # trend analysis

e2e-screenshots: 30 days # failure debugging

performance-traces: 90 days # historical analysis

deployment-logs: 90 days # audit / compliance

```

Per-CI:

| CI | Retention default |

|------------------|----------------------------------------------------|

| GitHub Actions | 90 days; configurable per artifact via `retention-days` |

| GitLab CI | Per-job `expire_in:`; 30 days project default |

| Jenkins | Configured via `buildDiscarder(logRotator(...))` |

| CircleCI | 30 days; non-configurable on free tier |

Don't retain forever - storage cost.

## §5 - Per-trigger filtering

```

Per-PR (push to PR branch):

- Smoke tests.

- Lint + unit tests.

- Per-changed-files coverage gate.

Per-merge to main:

- Full unit + integration suite.

- Smoke E2E.

- Coverage trend tracking.

Per-deploy to staging:

- Smoke E2E against staging.

- Synthetic monitor smoke.

Nightly scheduled:

- Full E2E across browsers.

- Full security scans (axe, OWASP).

- Mutation testing (per `stryker-mutation`).

Pre-release tag:

- Cross-platform matrix (per `mobile-device-matrix-toolkit`).

- Cross-browser matrix (per `browser-matrix-runner`).

- Manual UAT sign-off.

Manual / on-demand:

- Specific debug runs.

- Performance / load tests.

```

Tier the cadence to balance feedback latency vs cost.

## §6 - Concurrency control

When PRs receive rapid pushes:

| CI | Pattern |

|------------------|---------------------------------------------------|

| GitHub Actions | `concurrency: group: ${{ github.workflow }}-${{ github.head_ref }}` + `cancel-in-progress: true` |

| GitLab CI | `interruptible: true` per job |

| Jenkins | `disableConcurrentBuilds()` in pipeline options |

| CircleCI | `auto-cancel-redundant-workflows` in project settings |

The pattern: cancel superseded runs. Saves CI cost on stale

commits.

- ## §7 - JUnit XML reporting (cross-CI standard)

- Every modern CI accepts JUnit XML via either native plugin or

- third-party action:

- | CI | JUnit XML support |

- |------------------|----------------------------------------------------|

- | GitHub Actions | `dorny/test-reporter` action |

- | GitLab CI | `artifacts.reports.junit:` (native) |

- | Jenkins | `junit '...'` (JUnit Plugin; native) |

- | CircleCI | `store_test_results:` (native; feeds Insights) |

- Always emit JUnit XML; downstream parser via `junit-xml-analysis` (in the qa-test-reporting plugin).

- ## §8 - Secret management

+ ## §7 - Secret management

```

Never:

- Commit credentials to .yml / Jenkinsfile

- Use secrets in pull_request from forks

- Use `set -x` in scripts that handle secrets

Always:

- CI platform's secret store

- Mask in logs (`echo "::add-mask::$VALUE"` for GHA)

- Rotate on schedule

- Scope per-job (job-level env > workflow-level env > global)

```

- ## §9 - Per-language standard reporters

- | Language | Default reporter | JUnit XML output |

- |-------------------|----------------------------------------|-------------------------------------------|

- | JavaScript (Jest) | `default` | `jest-junit` (separate package) |

- | TypeScript | (same as JS) | (same) |

- | Python (pytest) | `pytest` | `pytest --junitxml=reports/junit.xml` |

- | Java (Maven) | Surefire | `target/surefire-reports/*.xml` (default) |

- | Java (Gradle) | Gradle Test | `build/test-results/test/*.xml` (default) |

- | .NET | dotnet test | `--logger "junit;LogFilePath=..."` |

- | Go | `go test` | `gotestsum --junitfile=junit.xml` |

- | Ruby (RSpec) | RSpec | `--format RspecJunitFormatter --out junit.xml` |

- The same JUnit XML feeds every CI's reporting + downstream

- analysis tools.

- ## §10 - Per-job timeouts

+ ## §8 - Per-job timeouts

| Job type | Recommended timeout |

|--------------------|--------------------------------------------------|

| Lint | 5 min |

| Unit tests | 10 min |

| Integration tests | 20 min |

| E2E (per-browser) | 30 min |

| E2E (full matrix) | 60 min |

| Deploy | 30 min |

| Performance / load | 60 min |

Hard timeouts prevent runaway jobs from consuming runners.

- ## §11 - Cache strategies

- ```

- Per-language cache key recommendations:

- - Node: cache key on `package-lock.json` hash

- - Python: cache key on `requirements.txt` / `poetry.lock` hash

- - Java (Maven): cache key on `pom.xml` hash; cache `~/.m2`

- - Java (Gradle): cache `~/.gradle`

- - Go: cache `~/go/pkg/mod` on `go.sum` hash

- - Rust: cache `~/.cargo` on `Cargo.lock` hash

- Benefits: repeat installs are sub-second vs 30s-2min cold.

- Trade-off: cache eviction when key changes; extra config to

- manage.

- ```

- ## §12 - Cross-CI portability

+ ## §9 - Cross-CI portability

If the team needs CI portability (multiple CIs in use, or anticipates

migration):

- Encapsulate logic in **shell scripts** (`scripts/test.sh`,

`scripts/build.sh`); CI calls the script.

- Use **standard env vars** (`CI=true`, `CI_BRANCH`,

`CI_COMMIT_SHA`); abstract per-CI vars.

- Containerize the build (Docker image with all deps); CI just

runs the container.

The goal: `.github/workflows/test.yml`, `.gitlab-ci.yml`, and

`Jenkinsfile` are thin wrappers calling the same scripts.

+ ## Worked example: a 25-minute Playwright E2E suite on GitHub Actions

+ Walk the How-to-use steps for a suite that runs 25 minutes end-to-end.

+ 1. **Shard + timeout.** 25 min falls in the 10-30 min band (§1), so

+ split into 4 shards - roughly 6-7 min wall-clock each - and set a

+ 30 min hard timeout per shard job (the E2E per-browser row in §8).

+ 2. **Retry.** Allow at most 1 framework-level retry for a network

+ timeout to a dependency (§2); a test that only passes on that retry

+ is flake, not a pass - tag it `@flaky` and quarantine it (§3)

+ rather than retrying further.

+ 3. **Concurrency.** Set

+ `concurrency: group: ${{ github.workflow }}-${{ github.head_ref }}`

+ with `cancel-in-progress: true` (§6) so a fresh push to the PR

+ cancels the superseded run.

+ 4. **Cadence.** Run a smoke subset per-PR, the full 4-shard E2E

+ per-merge to main, and the full cross-browser E2E nightly (§5).

+ 5. **Reporting + artifacts.** Emit JUnit XML via `dorny/test-reporter`

+ so results feed `junit-xml-analysis`

+ ([references/junit-and-cache-lookups.md](references/junit-and-cache-lookups.md)),

+ and upload failure screenshots with `retention-days: 30` - the

+ `e2e-screenshots` band in §4.

+ ## Deep references

+ The per-CI reporting mechanism and the per-language reporter and

+ dependency-cache lookups live in one companion reference so this file

+ stays a decision surface:

+ - **Reporting + cache lookups** - JUnit XML support per CI, default

+ reporters per language, and per-language cache-key recommendations:

+ [references/junit-and-cache-lookups.md](references/junit-and-cache-lookups.md).

## References

- `github-actions-test-jobs`,

`gitlab-ci-test-jobs`,

`jenkinsfile-test-stages`,

`circleci-test-configs` -

per-CI implementation skills.

- `flaky-test-quarantine` - flake handling.

- `junit-xml-analysis` - JUnit XML parser.

- `e2e-suite-budget` - when to refactor instead of shard more.

+ - Reporting + cache lookups (per-CI JUnit, per-language reporters +

+ cache keys):

+ [references/junit-and-cache-lookups.md](references/junit-and-cache-lookups.md).