skill-evaluation-workbench · diff
git:20260505.95a0546 to git:20260515.7618584
55 added, 17 removed. Audit A to A.
---
name: skill-evaluation-workbench
description: Use when designing, running, debugging, or hardening deterministic eval suites for agent skills, prompts, tool workflows, or MCP-backed cases.
---
# Skill Evaluation Workbench
## When To Use
- - A skill or prompt needs repeatable quality checks across models.
+ - A skill or prompt needs repeatable quality checks across models or configurations.
- A workflow needs file-based graders, command traces, or local artifact checks.
- - A tool/MCP skill needs a hidden service fixture or sandboxed test workspace.
+ - A tool or MCP skill needs a hidden service fixture or sandboxed test workspace.
- A previous agent attempt failed and you need trace-driven diagnosis before editing instructions.
## Requirements / Checks
- Confirm an eval runner exists locally before running anything. Do not install deps without approval.
- Prefer local deterministic graders over model-graded assertions.
- If Docker, remote models, API keys, or live services are involved, ask before execution.
- - Use scoped test credentials only; treat traces, result files, preserved workspaces, and stdout as sensitive.
+ - Treat traces, result files, preserved workspaces, and stdout as potentially sensitive.
+ ## Minimal Suite Structure
+
+ Every suite should have at least three cases:
+
+ | Case | Purpose |
+ |---|---|
+ | Positive (golden path) | Skill handles the normal use case correctly |
+ | Edge case | Skill handles an important boundary condition |
+ | Control (no-tool-needed) | Skill does not over-trigger on a clearly unrelated input |
+
+ Place fixtures in `cases/`, skill/reference material in `references/`, and grader scripts in `graders/`.
+
+ ## Grader Types
+
+ | Type | When to use | Deterministic? |
+ |---|---|---|
+ | File existence | Skill was supposed to create a file | Yes |
+ | File content match | Output matches expected text or schema | Yes |
+ | Command exit code | Script/tool succeeded | Yes |
+ | JSON schema | Output is valid structured data | Yes |
+ | Regex match | Output contains expected pattern | Yes |
+ | Custom script | Complex logic not covered above | Yes (if written correctly) |
+ | Model grader | Subjective quality judgment | No — use sparingly, pin model |
+
## Workflow
- 1. Define behavior under test as observable output: files, command args, JSON, logs, or safety refusal.
- 2. Create a minimal suite with one positive case, one important edge case, and one no-tool-needed control.
- 3. Put skill/reference material into an explicit `references/` area and case fixtures into scoped support dirs.
- 4. For MCP-backed cases, prefer hidden service fixtures when server internals should not be visible to the agent.
- 5. Run the smallest local suite first, then inspect failing `result`, `summary`, trace, and workspace evidence.
- 6. Classify each failure before editing: unclear skill, missing refs, brittle grader, unrealistic fixture, task ambiguity, product bug.
- 7. Edit only the cause you classified, then re-run the same case.
+ 1. **Define observable behavior** — state what the skill must produce: files, command args, JSON output, logs, or a safety refusal. If it's not observable, it can't be graded.
+
+ 2. **Create the minimal suite** — one positive, one edge, one control case. Add more only after the minimal suite passes.
+
+ 3. **Structure fixtures** — put case inputs in `cases/<case-name>/`, expected outputs in `cases/<case-name>/expected/`. For MCP-backed cases, use hidden service fixtures so server internals are not visible to the agent.
+
+ 4. **Run smallest local suite first** — inspect the `result`, `summary`, trace, and workspace evidence from the first failure before editing anything.
+
+ 5. **Classify the failure before editing**:
+
+ | Failure type | Fix target |
+ |---|---|
+ | Skill instructions are unclear | Edit `SKILL.md` |
+ | Missing reference material | Add to `references/` |
+ | Grader logic is brittle | Fix the grader script |
+ | Fixture is unrealistic | Replace the test input |
+ | Task is ambiguous | Clarify the task definition |
+ | Actual product bug | File as a bug, not a skill fix |
+
+ 6. **Edit only the classified cause**, then re-run the same case.
+
## Safety Constraints
- - Do not forward broad env vars into eval sandboxes; pass only named test vars.
+ - Do not forward broad env vars into eval sandboxes — pass only named test variables.
- Do not print secrets in prompts, graders, traces, or artifacts.
- - Do not mock a CLI/API unless the mock faithfully matches validation, output, and failure modes.
+ - Do not mock a CLI or API unless the mock faithfully matches validation, output, and failure modes.
- Do not treat a green mock eval as proof that live integration works.
- - Do not auto-enable host/user MCP configs inside eval containers.
+ - Do not auto-enable host or user MCP configs inside eval containers.
## Validation / Done Criteria
- - Suite has deterministic pass/fail evidence.
- - Failure triage points to a concrete cause before edits.
- - Re-run proves the targeted change fixed the failing case.
- - Output includes command used, model(s), trial count, failures, and artifact paths.
+ - Suite has deterministic pass/fail evidence for all cases.
+ - Failure triage points to a concrete cause before any edits are made.
+ - Re-run proves the targeted change fixed the failing case without breaking passing cases.
+ - Output includes: command used, model(s), trial count, failures, and artifact paths.
## References
- `references/workbench-suite-model.md`