with-eval · diff
v0.1.0 to v0.1.0
85 added, 69 removed. Audit A to A.
---
name: with-eval
- description: PR review checklist with a bundled deterministic eval that catches "faking" agents. Use as a template when your course has measurable behavior worth verifying locally before publication. The agent reviews a code diff against the bundled checklist; the eval runner mechanically scores the review by checking whether planted issues were flagged. No LLM is required in the scoring loop — deterministic substring presence against bundled expectations.
+ description: Reviews code diffs against a structured checklist covering security (sql-injection, hardcoded-secret, unvalidated-input), reliability (missing-error-handling, resource-leak), and correctness (off-by-one, wrong-comparison). Use when reviewing PRs, audit diffs, or any code change where you want a categorised, line-anchored review instead of free-form prose. Ships with a bundled deterministic self-test that catches "faking" agents that rubber-stamp without reading the diff.
license: MIT
- compatibility: Requires Python 3.10+.
+ compatibility: Requires Python 3.10+ (for the bundled self-test).
metadata:
author: logion-examples
version: "0.1.0"
---
- # With-Eval: PR Review With Anti-Faking Eval
+ # With-Eval: Checklist-Driven PR Review
- This example shows how to bundle an evaluation harness alongside a course. The evaluation tests one specific thing: **does the agent actually apply the checklist, or does it rubber-stamp?**
+ This skill reviews code diffs against a bundled checklist. The agent reads `references/checklist.md`, walks the diff category by category, and writes a structured review naming any violated categories by their stable code (e.g. `security:sql-injection`, `reliability:missing-error-handling`).
- The checklist itself is a methodology — structured prompts the agent follows when reviewing a code diff. The eval verifies the agent isn't faking by feeding it a known-buggy diff with planted issues, then mechanically checking the agent's review output for those issues.
+ The output format is deliberately structured — one line per finding, with the category code, file path, line number, and a short description. The categories don't change between versions, so reviews are comparable across diffs and across time.
- ## Why this pattern matters
+ ## How the agent uses this skill
- Most LLM-driven workflows can be faked. An agent that says *"the code looks fine"* on every PR appears to be doing review work but isn't. Bundled evals are the cheapest defense: deterministic ground truth that catches the rubber-stamp pattern before publication.
+ When asked to review a diff:
- This is also the seam future eval-backed bounty work will plug into. A course that ships a working eval today is a course that supports bounty-funded improvement tomorrow — same fixtures, same scoring, same ground truth, just paid evaluators instead of local self-check.
+ 1. Read `references/checklist.md` (loaded on-demand via progressive disclosure — not loaded at activation).
+ 2. Walk the diff against each category in the checklist.
+ 3. For each issue found, write one line in the format:
+ ```
+ <category-code> <path>:<line> — <short description>
+ ```
+ 4. If the diff is clean, write a single line: `no issues`.
- ## Structure
+ That's the entire workflow. The user gets a list of categorised findings or a clean bill of health.
+ ## Example output
+
+ For a diff that introduces a string-concatenated SQL query and a hardcoded API key:
+
```
- with-eval/
- ├── SKILL.md # this file
- ├── course/
- │ └── capabilities.yaml # file + terminal; no network, no secrets
- ├── references/
- │ └── checklist.md # the actual review checklist
- └── evals/
- ├── scenarios.json # which fixtures to run, expected outcomes
- ├── fixtures/
- │ ├── clean_pr.diff # a diff with no issues
- │ └── buggy_pr.diff # a diff with four planted issues
- ├── expected/
- │ ├── clean_pr.json # must NOT mention planted categories
- │ └── buggy_pr.json # MUST mention all four planted categories
- ├── runner.py # deterministic scorer (no LLM)
- ├── reviews/ # agent writes its reviews here
- └── reports/ # runner writes verdicts here
+ security:sql-injection users/dao.py:7 — query built via string concatenation with user-supplied `email`
+ security:hardcoded-secret users/dao.py:4 — API key stored as a string literal
```
- ## How the agent uses this course
+ ## When to use this skill
- When asked to review a diff, the agent:
+ - Pre-merge PR review (the obvious case).
+ - Auditing a vendor patch before applying it.
+ - Self-review before pushing a feature branch.
+ - Anywhere you'd otherwise get a free-form "looks good 👍" from an agent — this skill forces a categorised answer.
- 1. Loads `references/checklist.md` (progressive disclosure — not loaded at activation).
- 2. Reads the diff (either supplied by the user, or one of `evals/fixtures/*.diff` for self-check).
- 3. Walks the diff against each checklist category.
- 4. Writes a structured review to `evals/reviews/<scenario>.txt` (when running the bundled eval) or to wherever the user asks.
+ ## Capability declarations
- The review **must** mention any checklist category code (e.g. `security:sql-injection`) that the diff violates. The categories are deliberately specific so the eval can mechanically check whether they were addressed.
+ - `tools: [file, terminal]` — reads the diff, the checklist, writes the review.
+ - `filesystem.write: [./evals/reviews, ./evals/reports]` — review output + self-test verdicts (see below).
+ - No network, no secrets, no human approval. The skill runs entirely locally.
- ## How the eval works
+ ---
- ```
+ ## Verifying the skill works: bundled self-test
+
+ The skill ships with a small evaluation harness under `evals/` that lets you (or a CI job, or a future bounty-eval) verify the agent **actually applies the checklist** instead of rubber-stamping.
+
+ ### Why this matters
+
+ LLM-driven review can be faked. An agent that says *"the code looks fine"* on every diff appears to be doing review work but isn't. The bundled self-test catches the rubber-stamp pattern: it feeds the agent two known fixtures and mechanically scores its reviews.
+
+ ### How to run it
+
+ ```bash
+ # 1. Ask the agent to review each fixture, writing each review to:
+ # evals/reviews/buggy_pr_must_flag_planted_issues.txt
+ # evals/reviews/clean_pr_no_false_positives.txt
+ #
+ # 2. Run the scorer:
python evals/runner.py
```
For each scenario in `evals/scenarios.json`:
-
- 1. The runner reads `evals/reviews/<scenario>.txt` (the review the agent wrote).
- 2. Loads the expected outcome from `evals/expected/<scenario>.json`.
- 3. Scores by substring presence:
- - **Buggy fixture** (`verdict: must_flag`): every category in `required_categories` must appear in the review.
- - **Clean fixture** (`verdict: must_pass`): no category in `forbidden_categories` may appear.
- 4. Writes a verdict to `evals/reports/<scenario>.json`.
- 5. Exits non-zero if any scenario failed.
-
- The runner does **not** call an LLM. It does not need one — the agent has already produced the review; the runner is the *check*, not the *reviewer*. This keeps the eval cheap, deterministic, and trust-clean.
+ - The runner reads the review the agent wrote.
+ - Loads the expected outcome from `evals/expected/<scenario>.json`.
+ - Scores by substring presence:
+ - **Buggy fixture** (`verdict: must_flag`): every category in `required_categories` must appear in the review.
+ - **Clean fixture** (`verdict: must_pass`): no category in `forbidden_categories` may appear.
+ - Writes a verdict to `evals/reports/<scenario>.json`. Exits non-zero if any scenario failed.
- ## What gets planted in `buggy_pr.diff`
+ The runner does **not** call an LLM. The agent has already written the review; the runner is the *check*, not the *reviewer*. Deterministic, cheap, trust-clean.
- The buggy fixture intentionally introduces four issues, one per checklist category:
+ ### What gets planted in `buggy_pr.diff`
- | Planted issue | Checklist category code |
+ | Planted issue | Checklist category |
|---|---|
| SQL query built via string concatenation | `security:sql-injection` |
| API key hardcoded as a string literal | `security:hardcoded-secret` |
| `urlopen` call without try/except | `reliability:missing-error-handling` |
| `range(n + 1)` where `range(n)` is intended | `correctness:off-by-one` |
- The agent's review of `buggy_pr.diff` **must** mention each of these category codes (verbatim) to pass the eval. A faking agent that writes "LGTM" on both diffs fails this scenario.
-
- ## How to run the eval yourself
-
- ```bash
- # 1. Ask the agent (or your model) to review each fixture, writing to:
- # evals/reviews/buggy_pr_must_flag_planted_issues.txt
- # evals/reviews/clean_pr_no_false_positives.txt
+ A faking agent that writes "LGTM" or "no issues" on the buggy fixture fails on all four. An over-eager agent that flags every change fails the clean fixture by mentioning forbidden categories.
- # 2. Run the scorer:
- python evals/runner.py
- ```
+ ### Forward-compat note
- Verdicts land in `evals/reports/`. Exit code 0 = all scenarios passed.
+ This bundled self-test is the same shape future eval-backed marketplace bounties will use: bundled fixtures + bundled expectations + a deterministic scorer. A course that ships a working self-test today is one that supports paid-evaluator bounties tomorrow with no schema change — same fixtures, same scorer, just network evaluators instead of local self-check.
- ## Capability declarations
+ ---
- - `tools: [file, terminal]` — reads bundle files, runs Python.
- - `filesystem.write: [./evals/reviews, ./evals/reports]` — agent writes reviews here; runner writes reports here.
- - No network, no secrets, no human approval. The eval runs entirely locally against bundled fixtures.
+ ## Structure
- If you later want the runner to call an external LLM (rather than scoring whatever review the agent already wrote), add the appropriate `network.allow_domains` and `secrets.env`. But consider whether deterministic scoring against bundled expectations is sufficient — it usually is, and it's the harder-to-game design.
+ ```
+ with-eval/
+ ├── SKILL.md # this file
+ ├── course/
+ │ └── capabilities.yaml # file + terminal; no network, no secrets
+ ├── references/
+ │ └── checklist.md # the review checklist
+ └── evals/ # bundled self-test
+ ├── scenarios.json # which fixtures to run, expected outcomes
+ ├── fixtures/
+ │ ├── clean_pr.diff # a diff with no issues
+ │ └── buggy_pr.diff # a diff with four planted issues
+ ├── expected/
+ │ ├── clean_pr.json
+ │ └── buggy_pr.json
+ ├── runner.py # deterministic scorer (no LLM)
+ ├── reviews/ # agent writes its reviews here
+ └── reports/ # runner writes verdicts here
+ ```
## What an author should change when copying
1. `name:` in frontmatter (must match the directory name).
- 2. Replace `references/checklist.md` with your domain's checklist. Keep the category codes specific and stable.
- 3. Replace `evals/fixtures/` with diffs / inputs / artifacts from your domain.
- 4. Replace `evals/expected/` with the categories a correct response should mention (or avoid).
- 5. The runner is generic — substring presence against expected JSON — so you usually don't need to edit it. Adjust only if your scoring rule isn't substring-presence.
+ 2. `description:` (specific about what your checklist covers).
+ 3. Replace `references/checklist.md` with your domain's checklist. Keep the category codes specific and stable — the self-test depends on substring presence.
+ 4. Replace `evals/fixtures/` with diffs / inputs / artifacts representative of your domain.
+ 5. Update `evals/expected/` with the categories a correct response should (or should not) mention.
+ 6. The runner is generic — substring presence against expected JSON — so you usually don't need to edit it. Adjust only if your scoring rule isn't substring-presence.