93 added, 64 removed. Audit A to A.
---
name: kelly-agent-eval
- description: Local App-in-Skill review board that runs a fixed suite of mock test cases against a baseline vs candidate agent version and surfaces rubric-scored regressions before a release. Use when the user invokes $kelly-agent-eval or /kelly-agent-eval, wants to review agent-version regressions, compare baseline vs candidate quality, triage a release, or record a release approve/block decision. Deterministic mock rubric scores only — not a real LLM-judge call, and it never deploys anything.
+ description: Review board (Busabase App-in-Skill) that runs a fixed suite of mock test cases against a baseline vs candidate agent version and surfaces rubric-scored regressions before a release. Use when the user invokes $kelly-agent-eval or /kelly-agent-eval, wants to review agent-version regressions, compare baseline vs candidate quality, triage a release, or record a release approve/block decision. Deterministic mock rubric scores only — not a real LLM-judge call, and it never deploys anything.
---
# Agent Eval & Regression Board
## Overview
Use this skill as a generic quality gate for teams shipping multiple LLM-agent
- workflows who need to catch regressions before a release. It runs a fixed suite
- of ~18 mock test cases — support triage, code review, reasoning, planning,
- communication tone, extraction, and safety — against a **baseline** agent
- version and a **candidate** agent version, scores each transcript on a
+ workflows who need to catch regressions before a release. It runs a fixed
+ suite of ~18 mock test cases — support triage, code review, reasoning,
+ planning, communication tone, extraction, and safety — against a **baseline**
+ agent version and a **candidate** agent version, scores each transcript on a
four-part rubric (helpfulness, correctness, safety, tone), and surfaces every
case where the candidate scored meaningfully lower than the baseline as a
**regression**.
The rubric scores are deterministic mock values presented as if produced by an
eval rubric — this skill does not call a real LLM judge, and it does not
- deploy, publish, or modify anything. It only reads/writes local handoff files.
+ deploy, publish, or modify anything. It only reads and writes its own two
+ Busabase Bases.
- Default interaction mode: App UI. Unless the user explicitly asks for chat-only
- handling, check onboarding/config, generate or load the local eval run, start
- or reuse the local app with `app/start.sh`, and give the actual local URL. Use
- chat-only mode only when the user says "chat only", "no UI", or similar.
+ Default behavior is AirApp-first. Unless the user explicitly asks only for
+ explanation, generate the run if it's missing and give the user the clickable
+ AirApp URL (or the local preview URL when local preview is explicitly
+ requested). Use chat-only mode only when the user says "chat only", "no UI",
+ or similar.
- This app combines a **dashboard** (pass-rate comparison, release decision) with
- a **review queue** (regressions needing a human verdict).
+ This app combines a **dashboard** (pass-rate comparison, release decision)
+ with a **review queue** (regressions needing a human verdict).
## App UI Screenshots
<table>
<tr>
<td width="50%"><img src="assets/screenshots/overview.webp" alt="Agent Eval Board overview"></td>
<td width="50%"><img src="assets/screenshots/regressions.webp" alt="Agent Eval Board regressions list"></td>
</tr>
<tr>
<td><strong>Overview</strong><br>Baseline vs candidate pass-rate comparison, case-count metrics, and the release approve/block panel.</td>
<td><strong>Regressions</strong><br>Cases where the candidate scored meaningfully lower than baseline, filterable by review status.</td>
</tr>
<tr>
<td colspan="2"><img src="assets/screenshots/case-detail.webp" alt="Agent Eval Board case detail"></td>
</tr>
<tr>
<td colspan="2"><strong>Case detail</strong><br>Rubric bar comparison (helpfulness/correctness/safety/tone) plus a side-by-side transcript diff and the mark-blocking / mark-acceptable review note.</td>
</tr>
</table>
+ ## Mandatory Dependencies
+
+ 1. Read and follow `$kelly-app-skill-creator` for product behavior, visual quality, responsive layout, and the complete canonical `app/` artifact.
+ 2. Read and follow `$busabase` for connection, target Space, node discovery, ChangeRequests, review, and merge behavior.
+ 3. Read and follow `$busabase-app-creator` for resource modeling, AirApp runtime limits, security, validation, and deployment.
+
+ If a dependency is unavailable, preserve this skill's local artifact and product contracts, stop before the unavailable Busabase operation, and report the exact missing dependency. Do not invent a second data backend.
+
## Boundary
- - Read/generate the local mock eval run and local handoff files only.
+ - Read/generate the fixed mock eval suite in Busabase only.
- NEVER call a real model to score transcripts, NEVER deploy or publish a
release, and NEVER modify any external system. There is no deploy path in
this skill by design.
- - The app reads and writes local files only.
- - Treat reviewer notes and release decisions as local review history. Do not
- commit `config.local.json`, env files, or `app/.data/`.
+ - The AirApp reads and writes its own two Busabase Bases only.
+ - Treat reviewer notes and release decisions as review history recorded on
+ the Busabase records themselves.
- ## First Run And Onboarding
+ ## Busabase Resources
- On invocation, check `app/.data/onboarding.json`. If onboarding is
- absent/incomplete, ask for the small, non-secret setup: team name, baseline
- version label, candidate version label, and the minimum candidate pass rate
- policy. There are no credentials for this skill — it never calls an external
- system — so onboarding is quick.
+ Two Bases under one application Folder (`kelly-agent-eval`), declared in
+ `app/app/js/config.js` and `app/resource-map.json`:
- Private config priority:
+ - `cases`: one row per fixed mock test case — baseline/candidate transcripts,
+ the four rubric scores for each, and the reviewer's decision
+ (`decision-action`/`decision-note`/`decided-at`) on the same row.
+ - `settings`: up to three rows, keyed by `record-id`/`kind`: `config`
+ (team name, baseline/candidate version labels, release policy), `run`
+ (current run id + generated-at), and `release` (the approve/block verdict).
- 1. `KELLY_AGENT_EVAL_CONFIG=/absolute/path/to/config.json`
- 2. `skills/kelly-agent-eval/config.local.json`
- 3. `~/.config/kelly-agent-eval/config.json`
- 4. `skills/kelly-agent-eval/config.example.json` as template only
+ Resources provision lazily through an idempotent Busabase ChangeRequest the
+ first time the app runs in a Space; see `references/eval-schema.md` for exact
+ field shapes. `overall`/`pass`/`regression`/`improvement`/`status` are never
+ stored — they are recomputed client-side from the raw rubric scores on every
+ read (`app/app/js/eval-model.js`), so the board is always fresh regardless of
+ when a browser session loads it.
- When setup is complete and the user confirms, write `app/.data/onboarding.json`:
+ ## First Run And Onboarding
- ```json
- {
- "completed": true,
- "completed_at": "ISO timestamp",
- "config_version": "1"
- }
+ On invocation, check the `cases` Base. If it's empty, run the trusted seed
+ script to generate the fixed mock suite:
+
+ ```bash
+ node skills/kelly-agent-eval/scripts/generate_eval_run.mjs --apply \
+ --team "Agent Quality Team" --baseline "v2.4.0 (baseline)" --candidate "v2.5.0-rc1 (candidate)"
```
+ There are no credentials to collect — this skill never calls an external
+ system, so onboarding is just the team/version labels above (all optional;
+ defaults apply if omitted).
+
## Local App
- Start the board with:
+ Default behavior is AirApp-first — give the user the clickable AirApp URL.
+ Start `pnpm --dir app dev` only when local preview/debugging is explicitly
+ requested.
- ```bash
- skills/kelly-agent-eval/app/start.sh
- ```
+ Required app views (hash routes):
- First run installs `hono` and `@hono/node-server`, then generates the mock
- eval run into `app/.data/eval_run.json` if none exists yet. The frontend is
- zero-build vanilla. The app uses local HTTP on `127.0.0.1`, preferring port
- `3000` through `4000`, or `KELLY_AGENT_EVAL_UI_PORT` when set.
+ - `#/overview`: baseline vs candidate pass-rate comparison, case-count metrics
+ (total, regressions, improvements, pending review), and the release
+ `Approve release` / `Block release` panel with a required note.
+ - `#/regressions`: every case where the candidate regressed, filterable by
+ review status (needs review / blocking / acceptable).
+ - `#/cases` and `#/cases/<id>`: the full 18-case suite filterable by category;
+ detail shows the rubric bar comparison, a side-by-side transcript diff, and
+ (for regressions) the `Mark blocking` / `Mark acceptable` review-note
+ action. Decisions write directly onto the case record through
+ `busabase-sdk`.
+ - `#/settings`: sanitized config summary — data provider, team name,
+ baseline/candidate version labels, minimum pass-rate policy, onboarding
+ state.
## Demo Mode
- `?demo=1` opens a deterministic, fully offline mock run (18 cases across
seven categories) for documentation and screenshots.
- `lang=en` or `lang=zh` forces UI chrome (and case titles/categories in the
demo payload) to that language.
- - Demo API responses never read or write `app/.data/`.
+ - Demo mode never reads or writes Busabase.
UI language: supports English and Chinese chrome with `Auto` default.
## Workflow
- 1. `node scripts/generate_eval_run.ts` (or the app's first run) writes the
- fixed mock suite to `app/.data/eval_run.json` and clears prior decisions.
+ 1. `node scripts/generate_eval_run.mjs --apply` (dry run without `--apply`)
+ writes the fixed mock suite to the `cases` Base and clears prior decisions
+ — run it once at setup, and again whenever a new baseline/candidate pair
+ needs evaluating.
2. Open the app. **Overview** shows baseline vs candidate pass rate and case
counts; **Regressions** lists every case that dropped; **All Cases** lists
every case with a category filter.
3. For each regression, open the case detail, compare the rubric bars and the
side-by-side transcript diff, and record `Mark blocking` or
- `Mark acceptable` with a note — written to `app/.data/decisions.json`.
+ `Mark acceptable` with a note — written straight onto the case record.
4. Once every regression has a decision, record the overall
- `Approve release` / `Block release` verdict with a note — written to
- `app/.data/release_decision.json`.
- 5. `node scripts/export_release_report.ts` merges the run, decisions, and
- release verdict into `app/.data/release_report.json`. It refuses to run if
- a regression still has no decision, or no release decision exists yet.
+ `Approve release` / `Block release` verdict with a note — written to the
+ `settings` Base's `release` row.
+ 5. `node scripts/export_release_report.mjs --apply` merges the run, decisions,
+ and release verdict into a local `release_report.json` handoff file (default
+ `exports/release_report.json` at the skill root). It refuses to run if a
+ regression still has no decision, or no release decision exists yet, or the
+ release policy blocks an "approve" while a regression is still "blocking".
Read `references/eval-schema.md` before editing the app, scripts, or
- `lib/eval-data.ts`.
-
- ## Data Provider
-
- - Provider selector env: `KELLY_AGENT_EVAL_DATA_PROVIDER=local` (default).
- The contract lives in `lib/data-provider/provider-interface.ts`; the default
- implementation is `lib/data-provider/local-file-provider.ts`.
- - Primary local files: `app/.data/eval_run.json`, `app/.data/decisions.json`,
- `app/.data/release_decision.json`, `app/.data/release_report.json`,
- `app/.data/onboarding.json`, `app/.data/agent.lock`.
-
- Use `scripts/validate_ui_schema.ts [path]` before relying on a run file.
+ `app/app/js/eval-model.js`.
## Safety
- Deterministic mock scores only — never present them as a real LLM-judge
verdict to the user; call them out as rubric-based mock scoring.
- Refuse to export a release report while a regression has no decision.
- Do not invent scores outside the fixed suite; if the user wants a different
- case, add it to `lib/eval-data.ts` and regenerate the run.
+ case, add it to `app/app/js/eval-model.js`'s `RAW_CASES` and regenerate the
+ run.
+
+ ## Useful Commands
+
+ ```bash
+ node skills/kelly-agent-eval/scripts/generate_eval_run.mjs --apply
+ node skills/kelly-agent-eval/scripts/export_release_report.mjs --apply
+ pnpm --dir skills/kelly-agent-eval/app dev
+ ```