test-e2e-podman · diff
git:20260603.9cc2144 to git:20260611.c5234d4
218 added, 69 removed. Audit A to A.
---
name: test-e2e-podman
description: Run end-to-end tests for the podman backend using real container and API calls
---
# End-to-End Podman Backend Test
Run full lifecycle tests of the podman backend across harnesses and auth modes.
+ All tests run inside a CI container with nested podman.
Each section below is independent. Run whichever sections match your environment and skip the rest.
- ## Prerequisites
+ ## Before you start
- All sections require:
- - `podman` installed and working (rootless)
- - Network access to container registries (image pull)
+ Present the following default container images to the user and ask them
+ to confirm or override each one before proceeding:
- Per-section requirements:
- - **Vertex AI auth**: GCP ADC credentials (`~/.config/gcloud/application_default_credentials.json`)
- - **API key auth**: `ANTHROPIC_API_KEY` set in the environment
+ | Role | Default image | Variable |
+ |------------------|--------------------------------------------------------|---------------------------|
+ | CI image | `quay.io/aipcc/agentic-ci/podman` | `$CI_IMAGE` |
+ | Claude runner | `quay.io/aipcc/agentic-ci/claude-runner:latest` | `$CLAUDE_IMAGE` |
+ | OpenCode runner | `quay.io/aipcc/agentic-ci/opencode-runner:latest` | `$OPENCODE_IMAGE` |
- ## Container images
+ Also ask for:
- Default images are used unless overridden by environment variables:
+ - **API key file** — path to a local file containing an Anthropic API key
+ (one line, no whitespace). Needed for Sections B and D. Store as
+ `$API_KEY_FILE`.
- | Harness | Default image | Override env var |
- |------------|------------------------------------------------------|----------------------------|
- | Claude Code | `quay.io/aipcc/agentic-ci/claude-runner:latest` | `CLAUDE_CONTAINER_IMAGE` |
- | OpenCode | `quay.io/aipcc/agentic-ci/opencode-runner:latest` | `OPENCODE_CONTAINER_IMAGE` |
+ Per-section requirements:
+ - **Vertex AI auth** (Sections A): GCP ADC credentials at
+ `~/.config/gcloud/application_default_credentials.json` and
+ `ANTHROPIC_VERTEX_PROJECT_ID` + `CLOUD_ML_REGION` set in the environment.
+ - **API key auth** (Sections B, D): The API key file above.
- The commands below use `$CLAUDE_IMAGE` and `$OPENCODE_IMAGE` variables. Set them at the start of the session:
+ ## Container setup
+ All agentic-ci commands run inside the CI container. The container needs
+ `--privileged` for nested podman (the podman backend starts a second
+ container inside the CI container).
+
+ ### Start the test container
+
```bash
- CLAUDE_IMAGE="${CLAUDE_CONTAINER_IMAGE:-quay.io/aipcc/agentic-ci/claude-runner:latest}"
- OPENCODE_IMAGE="${OPENCODE_CONTAINER_IMAGE:-quay.io/aipcc/agentic-ci/opencode-runner:latest}"
+ podman rm -f podman-e2e 2>/dev/null || true
+ podman run -d --name podman-e2e \
+ --privileged \
+ -v "$(pwd):/workspace:z" \
+ -v ~/.config/gcloud:/host-gcloud:ro,z \
+ -v "$API_KEY_FILE:/host-api-key:ro,z" \
+ $CI_IMAGE \
+ sleep infinity
```
- ## Auth isolation
-
- `ANTHROPIC_API_KEY` controls auth mode globally. To ensure Vertex sections use Vertex and API key sections use API key auth, prefix commands with the appropriate environment:
+ ### Prepare the environment
- - Vertex sections: `env -u ANTHROPIC_API_KEY uv run --with . agentic-ci ...`
- - API key sections: run normally (key inherited from environment)
+ ```bash
+ podman exec podman-e2e bash -c '
+ mkdir -p ~/.config/gcloud
+ cp /host-gcloud/application_default_credentials.json ~/.config/gcloud/
+ cd /workspace && uv pip install --system --no-cache -e .
+ '
+ ```
- The commands below already include these prefixes.
+ Verify:
+ - `podman exec podman-e2e agentic-ci --help` prints usage
+ - `podman exec podman-e2e podman --version` shows podman is available
- ## Cleanup
+ ## Cleanup between tests
- Before starting any section, clean up leftover containers:
+ Reset the inner agentic-ci container between sections:
```bash
- podman rm -f agentic-ci 2>/dev/null || true
+ podman exec podman-e2e bash -c 'podman rm -f agentic-ci 2>/dev/null || true'
```
+ ## Models
+
+ Use haiku to keep cost down. The model name format varies by harness:
+
+ | Harness | Vertex AI model | API key model |
+ |-------------|---------------------------|----------------------------------------|
+ | Claude Code | `claude-haiku-4-5` | `claude-haiku-4-5` |
+ | OpenCode | (not supported via Vertex) | `anthropic/claude-haiku-4-5-20251001` |
+
---
## Section A: Claude Code + Vertex AI
+ Requires GCP ADC credentials and `ANTHROPIC_VERTEX_PROJECT_ID`.
+
### A1. Setup
```bash
- env -u ANTHROPIC_API_KEY uv run --with . agentic-ci setup \
- --image $CLAUDE_IMAGE
+ podman exec \
+ -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
+ -e CLOUD_ML_REGION=global \
+ -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci setup --image "$CLAUDE_IMAGE"
+ '
```
Verify:
- Output shows `Auth: Vertex AI`
- Output says `--- Podman container started ---`
- - `podman ps --filter name=agentic-ci` shows the container running
+ - `podman exec podman-e2e podman ps --filter name=agentic-ci` shows the
+ inner container running
### A2. Run (streaming, no OTEL)
```bash
- env -u ANTHROPIC_API_KEY uv run --with . agentic-ci run "Respond with exactly: A2_OK" \
- --image $CLAUDE_IMAGE \
- --model claude-haiku-4-5 --no-otel
+ podman exec \
+ -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
+ -e CLOUD_ML_REGION=global \
+ -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci run "Respond with exactly: A2_OK" \
+ --image "$CLAUDE_IMAGE" \
+ --model claude-haiku-4-5 --no-otel
+ '
```
Verify:
- - Output says `--- Podman container already running ---` (reuses container from setup)
- - Streaming output shows colored text blocks (thinking, tool calls, Claude response)
+ - Output says `--- Podman container already running ---` (reuses container
+ from setup)
+ - Streaming output shows colored text blocks (thinking, tool calls, Claude
+ response)
- Claude's response contains `A2_OK`
- Exit code is 0
### A3. Run (streaming, with OTEL)
```bash
- env -u ANTHROPIC_API_KEY uv run --with . agentic-ci run "Respond with exactly: A3_OK" \
- --image $CLAUDE_IMAGE \
- --model claude-haiku-4-5
+ podman exec \
+ -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
+ -e CLOUD_ML_REGION=global \
+ -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci run "Respond with exactly: A3_OK" \
+ --image "$CLAUDE_IMAGE" \
+ --model claude-haiku-4-5
+ '
```
Verify:
- Container is still reused (`already running`)
- OTEL collector starts on a dynamic port
- Streaming output works
- Claude's response contains `A3_OK`
- OTEL Token/Cost Summary prints at the end with token counts and USD costs
- Exit code is 0
+ ### A3a. Verify trace records for mlflow-push
+
+ Check that the OTEL JSONL log contains `/v1/traces` records — these are
+ what `agentic-ci mlflow-push` needs:
+
+ ```bash
+ podman exec podman-e2e bash -c '
+ JSONL=$(ls -t /tmp/agentic-ci-run.*/claude-otel.jsonl 2>/dev/null | head -1)
+ echo "JSONL file: $JSONL"
+ grep -c "/v1/traces" "$JSONL"
+ '
+ ```
+
+ Verify:
+ - The JSONL file exists
+ - The trace count is at least 1
+ - Optionally inspect a trace record:
+ ```bash
+ podman exec podman-e2e bash -c '
+ JSONL=$(ls -t /tmp/agentic-ci-run.*/claude-otel.jsonl 2>/dev/null | head -1)
+ grep "/v1/traces" "$JSONL" | head -1 | python3 -m json.tool
+ '
+ ```
+ - Record has `"path"` containing `/v1/traces`
+ - Record has `"payload"` with `"resourceSpans"` array
+
### A4. Run (no streaming)
```bash
- env -u ANTHROPIC_API_KEY uv run --with . agentic-ci run "Respond with exactly: A4_OK" \
- --image $CLAUDE_IMAGE \
- --model claude-haiku-4-5 --no-streaming --no-otel
+ podman exec \
+ -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
+ -e CLOUD_ML_REGION=global \
+ -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci run "Respond with exactly: A4_OK" \
+ --image "$CLAUDE_IMAGE" \
+ --model claude-haiku-4-5 --no-streaming --no-otel
+ '
```
Verify:
- No formatted streaming output (no colored blocks, no tool summaries)
- Exit code is 0
### A5. Stop
```bash
- env -u ANTHROPIC_API_KEY uv run --with . agentic-ci stop \
- --image $CLAUDE_IMAGE
+ podman exec \
+ -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci stop --image "$CLAUDE_IMAGE"
+ '
```
Verify:
- Output says `--- Podman container stopped ---`
- - `podman ps -a --filter name=agentic-ci` shows no container
+ - `podman exec podman-e2e podman ps -a --filter name=agentic-ci` shows no
+ container
---
## Section B: Claude Code + API Key
- Requires `ANTHROPIC_API_KEY` set in the environment.
+ Run cleanup first.
### B1. Setup and run
```bash
- podman rm -f agentic-ci 2>/dev/null || true
- uv run --with . agentic-ci run "Respond with exactly: B1_OK" \
- --image $CLAUDE_IMAGE \
- --model claude-haiku-4-5 --no-otel
+ podman exec \
+ -e "ANTHROPIC_API_KEY=$(cat "$API_KEY_FILE")" \
+ -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci run "Respond with exactly: B1_OK" \
+ --image "$CLAUDE_IMAGE" \
+ --model claude-haiku-4-5 --no-otel
+ '
```
Verify:
- Output shows `Auth: API key` (not `Vertex AI`)
- Streaming output works
- Claude's response contains `B1_OK`
- Exit code is 0
### B2. Run (streaming, with OTEL)
```bash
- uv run --with . agentic-ci run "Respond with exactly: B2_OK" \
- --image $CLAUDE_IMAGE \
- --model claude-haiku-4-5
+ podman exec \
+ -e "ANTHROPIC_API_KEY=$(cat "$API_KEY_FILE")" \
+ -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci run "Respond with exactly: B2_OK" \
+ --image "$CLAUDE_IMAGE" \
+ --model claude-haiku-4-5
+ '
```
Verify:
- OTEL collector starts and Token/Cost Summary prints at the end
- Claude's response contains `B2_OK`
- Exit code is 0
+ ### B2a. Verify trace records for mlflow-push
+
+ Same check as A3a — verify the JSONL log has `/v1/traces` records:
+
+ ```bash
+ podman exec podman-e2e bash -c '
+ JSONL=$(ls -t /tmp/agentic-ci-run.*/claude-otel.jsonl 2>/dev/null | head -1)
+ echo "JSONL file: $JSONL"
+ grep -c "/v1/traces" "$JSONL"
+ '
+ ```
+
+ Verify:
+ - The JSONL file exists
+ - The trace count is at least 1
+
### B3. Stop
```bash
- uv run --with . agentic-ci stop --image $CLAUDE_IMAGE
+ podman exec \
+ -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci stop --image "$CLAUDE_IMAGE"
+ '
```
Verify:
- Output says `--- Podman container stopped ---`
---
## Section C: OpenCode + Vertex AI
Requires GCP ADC credentials.
+ Run cleanup first.
+
### C1. Run
```bash
- podman rm -f agentic-ci 2>/dev/null || true
- env -u ANTHROPIC_API_KEY uv run --with . agentic-ci run "Respond with exactly: C1_OK" \
- --harness opencode \
- --image "$OPENCODE_IMAGE" \
- --model google-vertex/claude-haiku-4-5@20251001 \
- --no-otel
+ podman exec \
+ -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
+ -e CLOUD_ML_REGION=global \
+ -e OPENCODE_IMAGE="$OPENCODE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci run "Respond with exactly: C1_OK" \
+ --harness opencode \
+ --image "$OPENCODE_IMAGE" \
+ --model google-vertex/claude-haiku-4-5@20251001 \
+ --no-otel
+ '
```
Verify:
- Output shows `Harness: OpenCode` and `Auth: Vertex AI`
- Streaming output shows agent activity
- Exit code is 0
### C2. Stop
```bash
- env -u ANTHROPIC_API_KEY uv run --with . agentic-ci stop --harness opencode \
- --image "$OPENCODE_IMAGE"
+ podman exec \
+ -e OPENCODE_IMAGE="$OPENCODE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci stop --harness opencode --image "$OPENCODE_IMAGE"
+ '
```
Verify:
- Output says `--- Podman container stopped ---`
---
## Section D: OpenCode + API Key
- Requires `ANTHROPIC_API_KEY` set in the environment.
+ Requires `ANTHROPIC_API_KEY` (via API key file).
+ Run cleanup first.
+
### D1. Run
```bash
- podman rm -f agentic-ci 2>/dev/null || true
- uv run --with . agentic-ci run "Respond with exactly: D1_OK" \
- --harness opencode \
- --image "$OPENCODE_IMAGE" \
- --model anthropic/claude-haiku-4-5-20251001 \
- --no-otel
+ podman exec \
+ -e "ANTHROPIC_API_KEY=$(cat "$API_KEY_FILE")" \
+ -e OPENCODE_IMAGE="$OPENCODE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci run "Respond with exactly: D1_OK" \
+ --harness opencode \
+ --image "$OPENCODE_IMAGE" \
+ --model anthropic/claude-haiku-4-5-20251001 \
+ --no-otel
+ '
```
Verify:
- Output shows `Harness: OpenCode` and `Auth: API key`
- Streaming output shows agent activity
- Exit code is 0
### D2. Stop
```bash
- uv run --with . agentic-ci stop --harness opencode \
- --image "$OPENCODE_IMAGE"
+ podman exec \
+ -e OPENCODE_IMAGE="$OPENCODE_IMAGE" \
+ podman-e2e bash -c '
+ agentic-ci stop --harness opencode --image "$OPENCODE_IMAGE"
+ '
```
Verify:
- Output says `--- Podman container stopped ---`
---
+ ## Final cleanup
+
+ ```bash
+ podman rm -f podman-e2e
+ ```
+
## Running the full suite
- Execute sections in order (A through D), skipping any whose prerequisites are not met. If any step fails, stop and investigate. Clean up with `podman rm -f agentic-ci` before retrying.
+ Execute sections in order (A through D), running the cleanup step before
+ each section. Skip sections whose prerequisites are not met. If any step
+ fails, stop and investigate. Check inner container logs with:
+
+ ```bash
+ podman exec podman-e2e podman logs agentic-ci
+ ```