nasde-benchmark-calibration git:20260608.bfa9af3A

Immutable. This exact content is served forever at /api/v1/blob/d4714d5c291054ff.

---
name: nasde-benchmark-calibration
description: |
  Calibrate assessment rubrics by reviewing agent work in GitHub/GitLab PRs and feeding human comments back into the rubric. Use this skill when the user wants to:
  - Calibrate, tune, or sanity-check assessment criteria / dimensions of a benchmark
  - Review trial diffs alongside the LLM-as-a-Judge scores in a PR/MR
  - Investigate why judge scores feel off, too harsh, too lenient, or misaligned with how a human would grade the code
  - Pull review comments back from PRs/MRs and turn them into concrete rubric edits
  Even if the user doesn't say "calibrate" — if they're worried the LLM judge's scores diverge from human judgment, or want to align scores with a real developer's opinion before freezing a benchmark, this skill applies.
---

# NASDE Benchmark Calibration

Close the loop between the LLM-as-a-Judge and a human reviewer. The judge scores trials against
`assessment_criteria.md` (per task) and `assessment_dimensions.json` (benchmark-wide) — but an LLM
judge is an imperfect grader, and how it reads the rubric may diverge from how a human grades the code.
This skill publishes trial diffs + scores as Pull/Merge Requests for human review, pulls the comments
back, and proposes concrete rubric edits.

This is the third skill in the benchmark lifecycle: `nasde-benchmark-creator` writes the rubric,
`nasde-benchmark-runner` runs trials and scores them, and **this skill calibrates the rubric** against
human judgment before the benchmark is frozen.

## Prerequisites

- A sink repository that already exists (creation is out of scope). Configure it in `nasde.toml`:
  ```toml
  [calibration]
  repo = "https://github.com/Org/nasde-calibration"   # full URL or owner/repo slug
  # platform = "gitlab"        # only needed for a bare slug or a self-hosted host
  # base_branch = "main"
  # throttle_sec = 2.0
  ```
- The platform CLI for that repo's host: `gh` (GitHub) or `glab` (GitLab), installed and logged in
  (`gh auth login` / `glab auth login`). The platform is auto-detected from the repo URL host. nasde
  never handles tokens — the CLI's keyring does. See ADR-010.
- `git` on PATH.

## Workflow

### 1. Decide which trials to calibrate

Calibration is about the **criteria**, not individual runs. Discuss with the user which trials to
publish: a representative trial per (task, variant) is usually enough; publishing every repetition
floods the sink with PRs. Selecting trials is the agent's judgment here — there is no `--select` flag
yet (that logic may move into the CLI once the workflow settles).

You can pass job directories (all their trials) or individual trial directories — mixed is fine.

### 2. Publish

```bash
nasde calibrate publish jobs/<job>/<trial>__<id> -C <project_dir>
# or a whole job:
nasde calibrate publish jobs/<job> -C <project_dir>
```

Each trial becomes one PR/MR:
- An **orphan base branch** `base/<repo>-<sha>` holds the agent's start-state codebase (seeded once
  per (repo, commit); git deduplicates shared blobs across bases).
- The **feature branch** `calib/<repo>-<sha>/<trial>` carries the agent's diff applied as a real
  commit, so the PR diff is *exactly* the agent's work — clean to review.
- The **description** renders the dominant evaluator cluster's per-dimension `mean ± std`, the
  normalized score, and a "How to calibrate" note.
- A **`.calibration/` directory** ships the review context: the task's `instruction.md` (what the
  agent was asked to do), the `assessment_criteria.md` + `assessment_dimensions.json` the judge scored
  against, every `assessment_eval_<N>.json` (full per-dimension reasoning), and `metrics.json`. This
  lets the reviewer judge whether a score is fair without leaving the PR.

Re-running is idempotent — a trial whose **open** PR/MR already exists is skipped. Closing a
calibration round lets the same trials be re-published into a fresh round. Give the user the PR/MR
URLs and ask them to review the diffs and comment inline where a score disagrees with their judgment.
If a second person should review, the user adds them as a collaborator on the sink repo.

### 3. Pull the comments back

After the user has reviewed:

```bash
nasde calibrate pull-comments jobs/<job>/<trial>__<id> -C <project_dir> --json
```

`--json` emits a machine-readable structure (issue-level + inline comments, each with `path`/`line`
when inline) — consume it directly. Without `--json` it prints a summary table.

### 4. Diagnose the divergence per dimension

For each comment, line it up against the judge's score for that trial (read
`assessment_summary.json` for the per-dimension `mean ± std`, and `assessment_eval_*.json` for the
per-dimension `reasoning`). Inline comments carry a `path`/`line` — map that back to which
dimension's rubric is implicated.

Classify each divergence:
- **Judge too harsh** — human says the code is fine, judge scored low → the rubric's threshold for
  that dimension is too strict, or its description rewards the wrong thing.
- **Judge too lenient** — human flags a problem the judge missed → the rubric doesn't describe the
  failure mode, or its scale lacks resolution there.
- **Aligned** — no action; record it as a positive signal.

A human "I think this score should be X" comment is a free-text gold label — interpret it; do not
expect a rigid format.

### 5. Propose the rubric edit (for approval)

The rubric to edit lives at `evals/<source>/tasks/<task>/assessment_criteria.md` (per task) or
`evals/<source>/assessment_dimensions.json` (benchmark-wide, when the divergence is about a
dimension's scale/description, not one task's thresholds). The `<source>` and `<task>` come from the
trial's `result.json` (`source`, `task_name`).

Show the user a concrete **diff of the rubric** — the specific threshold/description change that would
have moved the judge toward the human's score — and **wait for approval before writing**. Never edit
the rubric silently.

After the edit, the loop repeats: re-run the trial through `nasde-benchmark-runner` (or re-eval an
existing job with `nasde eval`) and re-publish to confirm the judge now aligns. This iterative
measure → diagnose → fix → re-measure loop is the point of calibration.

## Notes

- Trajectories are intentionally NOT published to the PR (they may contain secrets and clutter the
  diff). The PR carries the diff, the assessment files, and metrics only.
- One trial = one PR. All of that trial's repeated evaluations are summarized in the description and
  shipped as `.calibration/assessment_eval_<N>.json`.
- The platform is inferred from the repo URL — GitHub and GitLab both work with no code change, only
  the right CLI installed.