claude-code-ci-harness · diff

git:20261019.2ae6fdc to git:20261019.25dcc92

9 added, 4 removed. Audit A to A.

---

name: claude-code-ci-harness

allowed-tools: Bash, Read, Write, Edit, Grep, Glob, WebFetch

description: |-

Run Claude Code headlessly as an isolated CI harness against any Anthropic

Messages-compatible endpoint (local engine, gateway or hosted API) or a

Claude subscription token. Covers credential

isolation, stream-json transcripts, tool-call assertions, multi-turn

scenarios, and endpoint/model compatibility diagnosis.

when_to_use: |-

Use for Claude Code CI harnesses, headless Claude Code tests, testing an

endpoint through /v1/messages, Claude Code tool-call regressions,

chat-template or tool-call parser testing, stream-json transcripts, model

compatibility testing, or safely running Claude Code against a

non-Anthropic model.

---

# Claude Code CI Harness

Never invoke `claude` directly for a test. Always use

`scripts/isolated-run.sh`. It isolates the run from the caller's profile.

## Run

```bash

${CLAUDE_SKILL_DIR}/scripts/isolated-run.sh MODE_FLAGS \

--workdir DIR --transcript OUT.ndjson --prompt 'PROMPT'

```

| Mode | `MODE_FLAGS` |

|---|---|

| Endpoint, no auth | `--base-url URL --model SERVED_NAME` |

| Endpoint, bearer token | `--base-url URL --model SERVED_NAME --token-var VAR` |

| Endpoint, `X-Api-Key` header | `--base-url URL --model SERVED_NAME --token-var VAR --x-api-key` |

| Claude subscription | `--model CLAUDE_ALIAS_OR_ID` (no `--base-url`; token var defaults to `CLAUDE_CODE_OAUTH_TOKEN`) |

| Flag | Rule |

|---|---|

| `--base-url` | Endpoint root without `/v1`, local or remote. |

| `--model` | The exact name the endpoint serves; a Claude alias or ID in subscription mode. |

| `--token-var` | Name of a variable in the secrets file. |

| `--x-api-key` | Only for endpoints that require the token in an `X-Api-Key` header. |

| `--tools` | Built-in tool allowlist. Default: `Read,Glob`. |

| `--timeout` | Wall-clock limit in seconds. Default: 180. `0` disables it. |

## Secrets

The secrets file is `~/.config/claude-code-ci-harness/targets.env`. Each line is `KEY=value`. The file must be mode 0600, owned by the user, not a symlink, and outside Git. The runner rejects any other file.

+ - CI job: write the file from the CI secret store with `umask 077` to a path outside the checkout (for example `$RUNNER_TEMP/targets.env`). Pass it with `CLAUDE_CODE_CI_SECRETS_FILE=PATH`, which `run-matrix.sh` also honours, or `--secrets-file PATH`. A file inside the checkout is rejected as inside Git.

- Subscription token: the user runs `claude setup-token` in a separate terminal and adds `CLAUDE_CODE_OAUTH_TOKEN=<token>` to the file. Never run `setup-token` inside an agent session, because the token would land in the transcript.

- Never copy `~/.claude/.credentials.json` into a run. Its refresh token rotates, and a rotated copy logs out the user.

- Never put a token in a flag, a transcript or the skill directory.

## Do not change in the runner

| Keep | Reason |

|---|---|

| `--safe-mode` | Disables CLAUDE.md, skills, plugins and hooks. |

| `--permission-mode dontAsk` | Denies every tool not in `--allowed-tools`. The `-p` default mode changes between releases. |

| `--strict-mcp-config` with an empty config | Blocks the caller's MCP servers. |

| Empty `ANTHROPIC_API_KEY` | Stops an inherited key from overriding the intended token. |

| No `--bare` | It disables OAuth and has dropped `Glob` from `--tools`. |

## Test

1. Build the workdir: `${CLAUDE_SKILL_DIR}/scripts/make-fixtures.sh DIR`.

- 2. Run the smoke scenario. Then escalate through `references/scenario-bank.md`, and stop at the first failure.

- 3. Assert on every transcript: `${CLAUDE_SKILL_DIR}/scripts/assert-events.sh OUT.ndjson [--expect-tools Glob,Read] [--expect-model NAME] [--min-turns N] [--max-turns N]`. It checks result success without `is_error`, tool sequence, model and turn bounds. Exit status is not evidence.

+ 2. Run the smoke scenario. Then escalate through `references/scenario-bank.md`, and stop at the first failure confirmed per step 6.

+ 3. Assert on every transcript: `${CLAUDE_SKILL_DIR}/scripts/assert-events.sh OUT.ndjson [--expect-tools Glob,Read] [--expect-model NAME] [--min-turns N] [--max-turns N] [--min-parallel N]`. It checks result success without `is_error`, tool sequence, model, turn bounds and tool calls per assistant message. Exit status is not evidence.

4. For more than one endpoint: `${CLAUDE_SKILL_DIR}/scripts/run-matrix.sh --prompt-file PROMPT.txt --workdir DIR --output-dir OUT --run 'label|base-url|model[|token-var]' [--run ...]`. An empty base-url means subscription mode. A run passes only if its transcript passes `assert-events.sh`; add scenario assertions per step 3.

5. For an open-weights model, scan every transcript for leaked special tokens: `${CLAUDE_SKILL_DIR}/scripts/scan-token-leakage.sh OUT.ndjson MODEL_DIR`. `MODEL_DIR` holds the served model's `tokenizer_config.json`, `tokenizer.json` or `*.jinja`; a file with one token per line also works. Exit 1 means a leak.

- 6. When an endpoint fails, record it as an endpoint/model compatibility finding. Never change the scenario to make it pass.

- 7. Afterwards, `${CLAUDE_SKILL_DIR}/scripts/cleanup.sh --run-dirs` removes run directories that a killed run left in `/tmp`.

+ 6. A single failing run is not a finding. Re-run that scenario until it has 3 transcripts and read them as a set: failing in 2 or 3 of 3 is an endpoint/model property; failing in 1 of 3 is noise unless its transcript shows the mechanism (name the event where it diverges).

+ 7. A run that delivers nothing (no `result` event, or an empty final answer) outranks every other result. A runner or setting change after which any run delivers nothing is unsafe: revert it, even if other runs improved.

+ 8. When an endpoint fails, record it as an endpoint/model compatibility finding. Never change the scenario to make it pass.

+ 9. Afterwards, `${CLAUDE_SKILL_DIR}/scripts/cleanup.sh --run-dirs` removes run directories that a killed run left in `/tmp`.

+ Done when every endpoint has a result row: `claude --version`, base URL, model, and for every scenario either `pass` or the first failing assertion or error event, with the pass count when it was re-run (`2/3`).

## References

- `references/scenario-bank.md`: scenarios and pass criteria

- `references/known-gaps.md`: symptom → cause → check

- `references/isolation.md`: what the runner isolates, for anyone editing it

- `references/sources.md`: verification stamps (freshen mode only)