git:20260716.6355db6 to git:20260813.53fd4ac
53 added, 21 removed. Audit A to A.
---
name: bosskuai-eval-driven-agent-improvement
description: Use this for agent eval design, routing tests, retrieval tests, LLM quality cases, regression harnesses, scorecards, and continuous agent improvement.
---
- # Bosskuai Eval Driven Agent Improvement
+ # BosskuAI Eval-Driven Agent Improvement
- Use this for agent eval design, routing tests, retrieval tests, LLM quality cases, regression harnesses, scorecards, and continuous agent improvement.
+ Use this skill to make agent quality **measurable**, so changes to prompts, tools, or models can be judged instead of guessed at.
- ## Fast Path
+ ## How this differs from nearby skills
- 1. Turn recurring agent failures into small deterministic or LLM-judged eval cases.
- 2. Separate routing, retrieval, workflow, quality, token, and safety evals.
- 3. Track pass rate and false confidence, not just green checks.
- 4. Avoid overfitting exact phrases; add fresh generalization cases.
+ - **`bosskuai-agent-architecture-audit`**: inspects agent structure for design faults; this skill measures behavior empirically.
+ - **`bosskuai-agent-introspection`**: debugs one failing run; this skill turns that failure into a permanent test case.
+ - **`bosskuai-prompt-optimizer`**: improves a prompt; this skill tells you whether the improvement was real.
+ - **`bosskuai-qa-automation-strategy`**: tests deterministic software; this skill handles nondeterministic output.
- ## Default Checks
+ ## Build the eval set from real failures
- - Turn recurring agent failures into small deterministic or LLM-judged eval cases.
- - Separate routing, retrieval, workflow, quality, token, and safety evals.
- - Track pass rate and false confidence, not just green checks.
- - Avoid overfitting exact phrases; add fresh generalization cases.
- - Require changelog entry for behavior changes.
+ The highest-value cases come from runs that actually went wrong. Each recurring failure becomes a small, named case with a fixed input and a checkable expectation. An eval suite written from imagination tests what you already knew.
- ## When To Open The Playbook
+ ## Separate eval types
- Open `../../references/playbooks/bosskuai-eval-driven-agent-improvement-playbook.md` only when the task needs detailed workflow, implementation examples, or release-grade depth.
+ Aggregating everything into one score hides which layer broke.
- ## Output Quality
+ - **Routing**: did the right skill, tool, or agent get selected?
+ - **Retrieval**: were the right documents or context fetched, and was anything critical missed?
+ - **Workflow**: did the multi-step sequence complete, and did it stop when it should?
+ - **Quality**: is the final output correct and useful?
+ - **Token/cost**: did it stay in budget?
+ - **Safety**: did it refuse, confirm, or escalate where required, including injection resistance?
- - Start with the verdict or action.
- - Separate confirmed facts, assumptions, and risks.
- - Include exact files, commands, tests, metrics, or rollback triggers when relevant.
- - Do not claim legal, security, or cost certainty without evidence.
+ ## Grade with the cheapest sufficient method
+ Prefer deterministic checks: exact match, schema validity, regex, tool-call assertions, file state. Use an LLM judge only for genuinely open-ended quality, and when you do, give it a rubric and validate the judge against human labels on a sample. An unvalidated judge is an opinion with a number attached.
+
+ ## Track false confidence
+
+ Pass rate alone is misleading. Watch how often the agent is confidently wrong, since that is the failure mode users trust and act on. Record both correctness and calibration, and treat a confident wrong answer as more severe than an admitted uncertainty.
+
+ ## Guard against overfitting
+
+ Tuning prompts against a fixed set produces agents that pass the set and fail reality. Keep a held-out set the prompt author does not read, add fresh generalization cases regularly, and avoid encoding exact phrases from eval cases into prompts.
+
+ ## Guardrails
+
+ - Never tune on the held-out set.
+ - Do not delete a failing case because it is inconvenient; mark it known-failing with an owner.
+ - Run evals on a fixed model version; a model upgrade is a change to measure, not a confound.
+ - Report per-category results, not just an aggregate.
+ - Small suites that run on every change beat large suites nobody runs.
+
+ ## Output format
+
+ ```text
+ Failure being addressed: [observed behavior]
+
+ Eval set:
+ [case id] - [category] - [input] - [expectation] - [grader: exact/schema/judge]
+
+ Baseline: [pass rate per category]
+ After change: [pass rate per category]
+ Held-out result: [pass rate, confirming no overfit]
+
+ False confidence: [rate of confident-wrong]
+ Cost per run: [tokens / spend]
+ Decision: [ship / iterate / revert] - [why]
+ ```
+
## References
- - `../../references/playbooks/bosskuai-eval-driven-agent-improvement-playbook.md`
- `../../references/checklists/eval-driven-agent-improvement-checklist.md`