agent-tool-replay-test · diff
git:20260911.25436b5 to git:20261005.2bc0998
70 added, 11 removed. Audit A to A.
---
name: agent-tool-replay-test
- description: "Replay recorded AI tool calls against deterministic fixtures to test argument validation, error handling, and side-effect boundaries."
+ description: "Replay recorded agent tool/LLM trajectories against deterministic fixtures to test argument validation, errors, idempotency, and side-effect boundaries—separate from live tool-selection quality."
category: development
risk: safe
source: self
source_type: self
date_added: "2026-09-11"
- tags: ["ai-workflows", "evaluation", "agent-tool-replay-test"]
+ tags: ["ai-workflows", "testing", "tool-calling", "record-replay", "cassette", "idempotency", "ci"]
tools: ["claude", "cursor", "gemini", "codex"]
---
- # Tool Replay Test
+ # Agent Tool Replay Test AI Skill Guide (Claude)
+ ## Overview & Engine Architecture
+
+ Live agents are expensive and flaky. **Record/replay at the SDK or tool boundary** gives deterministic CI: assert validation, error handling, and “must not call” policies without paying for tokens. Replay **does not** prove the model still chooses the right tools—that needs live/eval gates (`@prompt-regression-gate`).
+
+ Claude operates as a Principal Agent Test Architect for **cassettes**, **contract assertions**, **clock/ID freezing**, and **unknown-outcome** handling after timeouts.
+
+ ```
+ ┌─────────────────────────────────────────────────────────────┐
+ │ RECORD (once): LLM decisions ± tool results → cassette │
+ │ REPLAY (CI): inject recordings; optionally run real tools │
+ │ ASSERT: schemas, sequences, deny-lists, idempotency │
+ └─────────────────────────────────────────────────────────────┘
+ ```
+
+ ---
+
## Scope
- Read the tool contracts and identify read-only versus mutating operations. Use the existing test runner and mock interfaces. Record sanitized inputs, expected results, and correlation IDs without credentials.
+ - Read tool contracts: **read-only vs mutating**.
+ - Use project test runner + mocks; no production networks in `replay` mode.
+ - Sanitize cassettes (no credentials)—`@ai-pii-redaction-review`.
+ - Label coverage: **replay contract** ≠ **e2e model behavior**.
- ## Procedure
+ ---
- Build deterministic fixtures for success, timeout, permission denial, malformed output, and partial failure. Replace clocks and random IDs where needed. Prevent fixtures from reaching real networks or production resources.
+ ## Fixture matrix (minimum)
- ## Checks
+ | Case | Expect |
+ | :--- | :--- |
+ | Success | Valid args → OK result |
+ | Timeout | Bounded retry policy; **unknown outcome** path |
+ | Permission denied | No side effect; surfaced error |
+ | Malformed tool JSON | Reject via `@llm-json-contract-check` |
+ | Partial failure | Compensating / reconcile logic |
+ | Unknown fields / bad IDs | Validation reject before dispatch |
+ | Mutating call | AuthZ + idempotency key before effect |
+ | Policy deny | Tool absent from allow-list never invoked |
- Replay exact arguments through the validation and dispatch layers. Test that unknown fields and invalid identifiers are rejected. For mutating calls, verify authorization checks and idempotency behavior before any side effect.
+ ---
- ## Failure Handling
+ ## Determinism rules
- Check bounded retries and distinguish a known failure from an unknown outcome after a timeout. An unknown outcome requires reconciliation; blindly replaying a payment or message send can duplicate it.
+ - Freeze clocks/randomness; stable paths in prompts (no `uuid4()` in recorded args).
+ - Modes: `record` | `replay` (CI; miss = fail, no network) | `passthrough`.
+ - Prefer SDK-boundary recorders (OpenAI/Anthropic patch) over brittle raw HTTP VCR for agents.
+ - Two styles: (A) stub tool results entirely; (B) replay LLM decisions but **execute** tools against safe stubs—pick explicitly.
+ ---
+
+ ## Timeout / idempotency (critical)
+
+ After a timeout on `charge`/`send_email`, state is **unknown**. Blind replay can duplicate. Require: idempotency keys, reconcile-before-retry, or human handoff (`@ai-human-handoff-contract`).
+
+ ---
+
+ ## Assertions to ship
+
+ - Tool sequence / `called_with` schemas
+ - Forbidden tools never called
+ - Budget/step caps
+ - Cassette diff on prompt drift (fail or re-record deliberately)
+
+ ---
+
## Deliverable
- Deliver fixtures, invocation traces, and assertions about calls that must not occur. Label replay coverage separately from end-to-end model behavior because a replay does not test tool selection.
+ Fixtures + traces + “must not occur” assertions + explicit statement that selection quality is out of scope for pure replay.
+ ## Related skills
+
+ `@llm-json-contract-check`, `@prompt-regression-gate`, `@agent-injection-boundary-test`, `@llm-cost-latency-benchmark`
+
+ ## Agent Operational Directive
+
+ > **MANDATORY**: Keep replay offline in CI. Sanitize cassettes. Test validation and side-effect gates. Never blind-retry mutating calls after unknown outcomes. Separate replay coverage from model tool-selection evals.
+
+ ## Sources
+
+ Cassette/agentverify/pytest-agentcontract/langchain-replay patterns; industry emphasis on SDK-level record/replay and idempotent mutating tools.