CLAUDE.md · diff

git:20260610.cf306c9 to git:20260624.4d96bf0

21 added, 2 removed. Audit A to A.

# nasde-toolkit

AI coding agent evaluation toolkit. CLI entry point: `nasde`.

## Package structure

```

src/nasde_toolkit/

__init__.py # Re-exports __version__ from _version.py

_version.py # Auto-generated by hatch-vcs from git tags (gitignored)

cli.py # Typer CLI (init, run, eval, install-skills + harbor/opik pass-through)

config.py # nasde.toml + task.toml parsing into dataclasses

runner.py # Harbor Python API — variant resolution, config merging, Job execution

evaluator.py # Post-hoc assessment via pluggable CLI subprocess backends

evaluator_backends/

__init__.py

protocol.py # EvaluatorBackend Protocol

claude_subprocess.py # `claude -p` subprocess backend (default)

codex_subprocess.py # `codex exec` subprocess backend

results_exporter.py # `nasde results-export` — copy trial artifact essence to a plain dir

- pricing.py / pricing.toml # Versioned model price catalog (ADR-011) — load + compute_cost_usd

+ pricing.py / pricing.toml # Versioned model price catalog (ADR-011) — load + compute_cost_usd; layered override via load_pricing_layered + provenance via resolve_pricing_layers/effective_pricing_with_source (ADR-013)

+ pricing_report.py # Shared Rich table renderer for the effective price catalog (nasde pricing show + run summary, ADR-013)

token_metrics.py # Token usage + cost economics from trajectory final_metrics (ADR-011)

calibration_publisher.py # `nasde calibrate` — publish trial diffs+assessments as PRs/MRs (ADR-010)

calibration_resolve.py # Resolve [calibration] repo → (api slug, push url, platform)

git_platform_backends/

__init__.py # create_git_backend(url) — URL auto-detect + factory

protocol.py # GitPlatformBackend Protocol + RepoRef/PrRef/ReviewComment

detect.py # detect_platform(url) — github/gitlab from host (+ override)

git_ops.py # GIT layer — orphan base + feature branch push (not behind Protocol)

github_cli.py # GitHubCliBackend (`gh`)

gitlab_cli.py # GitLabCliBackend (`glab`)

docker.py # Docker environment helpers ([nasde.source] + [nasde.plugin] staging)

plugin_registration.py # Shared skill + MCP registration ([nasde.plugin] + variant [[skill]])

scaffold/

__init__.py # Project scaffolding templates and file creation

skills_installer.py # `nasde install-skills` — copies bundled authoring skills to ~/.claude/skills/

agents/

__init__.py

configurable_claude.py # Harbor-compatible Claude Code agent with sandbox file injection

configurable_codex.py # Harbor-compatible Codex agent with sandbox file injection

configurable_gemini.py # Harbor-compatible Gemini CLI agent with sandbox file injection

tests/

pyproject.toml

```

## How to run

```bash

uv tool install .

nasde --version

```

## Testing

```bash

uv run pytest

```

+ ## Development workflow

+ **A user-visible change is not done until `CHANGELOG.md` is updated.** When you

+ finish a feature, fix, dependency bump, CVE pin, or any change a user would

+ notice, add an entry under `## [Unreleased]` in `CHANGELOG.md` (in the right

+ Keep-a-Changelog section — Added / Changed / Fixed / Removed / Security) with a

+ `[#NN]` PR link-reference at the bottom — **as part of that change, not at

+ release time.** Deferring it lets `[Unreleased]` silently drift behind merged

+ PRs, so release notes ship incomplete. The full post-change checklist

+ (quality gates, docs/skills/scaffold consistency, smoke + e2e) lives in the

+ `nasde-dev` skill; releasing itself is in [docs/RELEASING.md](docs/RELEASING.md)

+ (move `[Unreleased]` under the new version, then tag → publish).

## Code style

1. PEP 8 with type hints on all public functions.

2. `@dataclass` for internal data models (see `config.py`, `evaluator.py`).

3. Rich console for all CLI output — no bare `print()`.

4. Do NOT use comments in method bodies. Use descriptive function and variable names instead.

5. Split large functions into a hierarchy of private helpers with descriptive names.

6. Structure functions: public first (alphabetical), then private helpers ordered by dependency (caller before callee).

7. Snake_case for file and directory names.

## Architecture

See [ARCHITECTURE.md](ARCHITECTURE.md) for the full system architecture with diagrams (end-to-end flow, trial lifecycle, cloud sandbox providers, assessment evaluation). Keep it in sync with code changes.

## Architecture decisions

- **CLI framework**: Typer with Rich markup mode. The `app` object in `cli.py` is the entry point registered in `pyproject.toml` as `nasde`.

- **Configuration**: Two-layer config — `nasde.toml` for project-level settings, `task.toml` per task (single file, shared with Harbor; nasde-specific fields live under `[nasde.*]`). Both parsed into `@dataclass` models in `config.py`. Task discovery walks `tasks/` (or `.nasde/tasks/`) automatically.

- **Benchmark runner**: Uses Harbor Python API (`Job`, `JobConfig`) directly instead of subprocess. The runner merges variant config with task registry into a dict, passes it to `JobConfig.model_validate()`, then runs `await job.run()`. Opik tracking via `track_harbor()` (monkey-patches Harbor at runtime).

- **Evaluator**: Subprocess-based assessment evaluation with pluggable backends. The evaluator spawns a CLI agent (`claude -p` or `codex exec`) as a subprocess to read trial artifacts and score them against assessment criteria. Backend selection via `[evaluation] backend` in `nasde.toml` (`"claude"` default, `"codex"` available). Claude backend uses `--output-format json` (without `--bare` — preserves OAuth/keychain auth for subscription billing and skills auto-discovery). Codex backend uses `--json --quiet --full-auto` for JSONL event streaming. Both backends respect the user's existing CLI authentication (OAuth subscription or API key) — no Agent SDK required, which avoids Anthropic's SDK-billing restrictions on subscription accounts. Configurable via `[evaluation]` in `nasde.toml` — backend, model, tools, MCP servers, skills, system prompt, and trajectory inclusion can all be customized. When `include_trajectory = true`, the evaluator also has access to the agent's ATIF trajectory (`agent/trajectory.json`). Default Claude model is `claude-opus-4-7`. Skills dir is mounted via temp-dir + `cwd` with `--add-dir <workspace>` so Claude's auto-discovery picks them up. **Repeated evaluations (output contract):** the judge is non-deterministic, so each trial is evaluated `eval_repetitions` times (default 3, `[evaluation] eval_repetitions` in `nasde.toml`, override `--eval-repetitions` on `run`/`eval`). The N evaluations of one trial run concurrently via `asyncio.gather` under the shared `--max-concurrent-eval` semaphore; the gather barrier ensures aggregation runs only after all N finish. Each is written **append-only** as `assessment_eval_<N>.json` (never overwritten — the bug being fixed was a single `assessment_eval.json` clobbered on every re-eval). A derived `assessment_summary.json` holds per-cluster mean/std(sample,n-1)/min/max aggregates: averages are computed **only within one `(evaluator_model, dimensions_fingerprint)` cluster**. A different judge model OR a changed `assessment_dimensions.json` rubric (added/removed dimension, changed `max_score`, OR changed `description`) is a different benchmark and is never averaged together — the fingerprint is the first 12 hex chars of the sha256 of the normalized (`sort_keys`) rubric JSON, stamped onto each `assessment_eval_<N>.json` at eval time (legacy files with no fingerprint → `""` cluster). Clustering by fingerprint also makes the per-group `members[0]` dimension/max_score read correct by construction. The largest cluster (tie → newest) is `dominant`. Opik upload uses the **dominant cluster's mean** (plus per-dimension std and `eval_n`), not the last evaluation. Legacy jobs are normalized by `eval_migration.py`, exposed as the **hidden** one-shot command `nasde migrate-evals` (`@app.command(hidden=True)` — invokable, but kept out of `nasde --help` and user docs because it is a migration tool, not a routine command; its tests stay to guard the logic). It takes a `jobs/` root (recurses) or a single trial dir, not a single job dir.

- **Variant system**: Each variant is a directory under `variants/` with a required `variant.toml` declaring the agent type (`agent = "claude"`, `agent = "codex"`, or `agent = "gemini"`). For Claude Code variants, `CLAUDE.md` is injected into `/app/CLAUDE.md`; for Codex variants, `AGENTS.md` is injected into `/app/AGENTS.md`; for Gemini CLI variants, `GEMINI.md` is injected into `/app/GEMINI.md`. An optional `skills/` subdirectory contains Claude skill snapshots — each `skills/<name>/` is injected **whole** (incl. `references/` and sibling files, via the shared `plugin_registration.stage_skill_dir`) into `/app/.claude/skills/<name>/`. A `variant.toml` may also declare a `[[skill]]` array (skill-by-reference, ADR-009): each entry points at a source skill dir (optional `ref`) without a copy under `variants/` — for Claude staged whole into `/app/.claude/skills/<name>/`, for Codex/Gemini routed through Harbor's native `config.agent.skills` (ADR-012, see the native-injection note below). An optional `agents_skills/` subdirectory contains Codex skill snapshots and an optional `gemini_skills/` subdirectory contains Gemini skill snapshots — each `<name>/` (a dir with a `SKILL.md`) is registered through **Harbor's native skill-injection** (`config.agent.skills`), NOT via `sandbox_files`. See the "Codex/Gemini native skill injection" decision below for why: Codex and Gemini auto-discover skills only from a HOME-scoped dir (`$HOME/.agents/skills`, `~/.gemini/skills`), never from a cwd `/app/.agents/skills` or `/app/.gemini/skills` dir. If no `harbor_config.json` exists, one is auto-generated from `variant.toml`.

- **Codex/Gemini native skill injection (ADR-012)**: Codex and Gemini auto-discover skills **only from a HOME-scoped directory** — `$HOME/.agents/skills` for Codex, `~/.gemini/skills` for Gemini — never from a `.agents/skills` / `.gemini/skills` dir sitting in the agent's `/app` cwd. The old path uploaded skill files into `/app/.agents/skills/` and `/app/.gemini/skills/` via `sandbox_files`, where the CLI never scans, so the skill was **never registered as a native slash-command** — the agent only stumbled on the file by reading it by hand. The fix routes `agents_skills/<name>/` and `gemini_skills/<name>/` through **Harbor's native `config.agent.skills`** list (`runner._collect_native_skill_dirs`): at trial time Harbor uploads each skill dir into the container's `skills_dir`, passes `skills_dir` to the agent ctor, and the agent's `_build_register_skills_command` copies them into the correct HOME-scoped location *inside* `run()` — the only point with the right timing (an earlier `setup()`/`sandbox_files` upload is in the wrong session relative to the CLI's skill scan, empirically verified). The `skills` list on the harbor_config agent is regenerated each run (tracked via `_nasde_derived_skills`), so a removed skill dir drops out and a hand-authored `skills` entry is preserved — mirroring the `sandbox_files` / `_nasde_derived_keys` contract. This covers **all three** skill paths for Codex/Gemini: the variant snapshot (`agents_skills/`, `gemini_skills/`) plus `[[skill]]` by-reference and `[nasde.plugin]` skills — the latter two are resolved to skill dirs by `plugin_registration.collect_referenced_skill_dirs` / `collect_plugin_skill_dirs` and **unioned** into the same `skills` list (one `_nasde_derived_skills` marker tracks both; nasde warns when two derived dirs share a basename since Harbor's `resolve_skills` is last-wins). **Claude is intentionally NOT routed here**: its `variants/<v>/skills/` → `sandbox_files` → `/app/.claude/skills/` (+ `~/.claude/skills/`) path already lands where Claude Code auto-discovers from cwd, is tested, and is the carrier for its own `[nasde.plugin]` / `[[skill]]` skills (via `stage_skill_dir`); a Claude agent drops `extra_skill_dirs` at merge time to avoid double-injecting. See ADR-012.

- **Codex ChatGPT OAuth**: Harbor's `Codex` agent uploads `~/.codex/auth.json` into the sandbox as `$CODEX_HOME/auth.json` — but **harbor 0.13 made this opt-in**: the agent now *defaults to `OPENAI_API_KEY`* and only uploads the OAuth `auth.json` when `CODEX_AUTH_JSON_PATH` (a specific file) or `CODEX_FORCE_AUTH_JSON` (truthy → `~/.codex/auth.json`) is set. Without an API key it would otherwise write an **empty** key into the sandbox (`Incorrect API key provided: ''` on `wss://api.openai.com/v1/responses`). So `_ensure_auth` opts into OAuth automatically: when no `OPENAI_API_KEY`/`CODEX_API_KEY` is set but `~/.codex/auth.json` exists, it sets `CODEX_FORCE_AUTH_JSON=true` (respecting a user-set `CODEX_AUTH_JSON_PATH`/`CODEX_FORCE_AUTH_JSON`). API key (`OPENAI_API_KEY`) always takes priority over OAuth. Tokens are not extracted back from sandbox after runs. Validate with `source scripts/export_codex_oauth_token.sh`. The runner bridges `CODEX_API_KEY` → `OPENAI_API_KEY` in `_ensure_auth` for users who prefer the Codex-specific env var name.

- **Gemini Google OAuth**: When `~/.gemini/oauth_creds.json` exists (created by `gemini login`) and no API key env var is set (`GEMINI_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_APPLICATION_CREDENTIALS`), `ConfigurableGemini` injects the OAuth credentials into the sandbox via env var. API key always takes priority over OAuth. Validate with `source scripts/export_gemini_oauth_token.sh`.

- **All dependencies are core**: `harbor`, `opik` are in `[project.dependencies]`. The evaluator depends on having `claude` CLI (default) or `codex` CLI installed and authenticated on the host — not bundled. No optional extras — `uv tool install .` gives full functionality. Assessment evaluation is on by default (`--without-eval` to skip).

- **Versioning**: Derived from git tags via `hatch-vcs` plugin. Tag `v0.1.0` → version `0.1.0`. Commits after a tag produce dev versions like `0.1.1.dev3+gabcdef`. `_version.py` is auto-generated at build time and gitignored. See ADR-007.

- **Bundled authoring skills**: `.claude/skills/nasde-benchmark-*` are shipped inside the wheel via `[tool.hatch.build.targets.wheel.force-include]` → `nasde_toolkit/_bundled_skills/`. `nasde install-skills` discovers them via `importlib.resources` (wheel install) with a fallback to the live `.claude/skills/` when running from an editable / source checkout. `nasde-dev` is deliberately excluded from the bundle (internal dev skill, not for end users).

- **Auto-generated Dockerfile**: When a task has no `environment/Dockerfile`, nasde generates one from `source.git` + `[docker]`. For local paths, also generates `docker-compose.yaml` to override the build context. See `docker.py:ensure_task_environment()`.

- **`[nasde.plugin]` (ADR-009)**: Ships a local Claude Code plugin (dir with `.claude-plugin/plugin.json`) into the sandbox with one `task.toml` declaration. `docker.py:ensure_task_plugin()` stages the plugin tree (at `ref` via a temp worktree) into a gitignored `_nasde-plugin/` inside the **active** build context (Harbor pins context to `environment/`; with `[nasde.source]` the context is the source repo/worktree — nasde reads it back from the generated compose and stages there), then appends a sentinel-fenced `COPY`+build stage to the Dockerfile (generated base if none, hand-written preserved). `plugin_registration.py` then registers the plugin's own `skills/` (whole dirs — for Claude into `/app/.claude/skills/`, for Codex/Gemini through Harbor's native `config.agent.skills` via `collect_plugin_skill_dirs`, ADR-012) and injects its MCP server (from `<plugin>/.mcp.json`, env-wrapped) into the task's `task.toml` — idempotent, fenced, never clobbers an author-declared same-name server. Skill-by-reference (`[[skill]]` in `variant.toml`) feeds the same agent-aware skill-registration machinery. Derived sandbox files are merged into the variant's `harbor_config.json` each run. MCP injection writes `task.toml` because Harbor reads MCP servers only from there (`trial.py` → `task.config.environment.mcp_servers`).

- **Pass-through CLI**: `nasde harbor ...` delegates to Harbor's Typer app via `add_typer()`. `nasde opik ...` forwards args to Opik's Click CLI via `ctx.args`.

- **Rubric calibration (ADR-010)**: `nasde calibrate publish PATHS...` / `pull-comments` close the loop between the LLM-as-a-Judge and a human reviewer by publishing trial diffs + assessments as PRs/MRs and pulling review comments back for rubric tuning. Two layers, deliberately separated: **GIT** (`git_platform_backends/git_ops.py` — `git push`/`ls-remote`, platform-agnostic, subprocess pattern from `docker.py`, not behind a Protocol) and **PLATFORM** (`git_platform_backends/` behind a `@runtime_checkable GitPlatformBackend` Protocol — `repo_exists`/`find_open_pr_for_branch`/`create_pr`/`fetch_pr_comments`/`validate_cli_installed`/`validate_auth`, mirroring `evaluator_backends/`). The base is keyed on `(repo, commit)` as an **orphan branch** `base/<repo>-<sha>` seeded once via `git archive HEAD` from the trial workspace (git deduplicates blobs by content across orphan bases — no shared ancestor needed); each trial is a feature branch `calib/<repo>-<sha>/<trial>` = base + the agent's `changes.patch` applied as a real commit + `.calibration/` files (no trajectory — secrets/clutter). `.calibration/` carries the reviewer's context: the task's `instruction.md` + `assessment_criteria.md` + `assessment_dimensions.json` (resolved from `result.json` `task_name`/`source`, trying both `tasks/<task>` and `evals/<source>/tasks/<task>` layouts), all `assessment_eval_<N>.json`, `assessment_summary.json`, and `metrics.json`. Idempotency is **open-only**: `find_open_pr_for_branch` matches only OPEN PRs/MRs (`gh pr list --state open`, `glab mr list` default), so a re-run skips a live round but lets a fresh round publish once the prior one is closed. The PR body is a pure transform of the dominant `AssessmentSummary` cluster (`calibration_publisher._render_pr_body`). Backend is **auto-detected from the sink repo URL host** (`github.com`→`gh`, `*gitlab*`→`glab`; `[calibration] platform` overrides for self-hosted) — no `backend` config field, eliminating the backend≠host mismatch. Preflight before any work: detect → `validate_cli_installed` (`shutil.which`, precise per-platform message) → `validate_auth` (`gh|glab auth status` exit code) → `repo_exists` (parses OUTPUT — `gh repo view` exits 0 even for a missing repo). Repo creation is out of scope (push creates branches ad-hoc in an existing repo). Reuses `_expand_to_trials`/`_capture_patch`/`_build_metrics` from `results_exporter.py` and `_aggregate_evaluations`/`_load_raw_evaluations`/`AssessmentSummary` from `evaluator.py`. Prerequisites mirror the evaluator's CLI requirement (ADR-002): `git` + `gh`/`glab` + login, no SDK, CLI keyring holds auth. The `nasde-benchmark-calibration` skill orchestrates the human-in-the-loop flow.

- **Results export (EXPERIMENTAL)**: `results_exporter.py` + `nasde results-export PATHS... --to DIR` copy the analytic *essence* of trial artifacts out of the gitignored `jobs/` tree into a flat per-trial layout (`DIR/<job>__<trial>/` with `metrics.json`, `assessment_eval_*.json` (all repetitions), `assessment_summary.json`, `trajectory.json`, `changes.patch`, `verifier_stdout.txt`, `reward.txt`). Re-export **merges**: missing eval files are copied and the summary/metrics refreshed, while immutable files (trajectory, patch) are left as-is — so evaluations added after a first export are picked up. A legacy bare `assessment_eval.json` (pre-migration trial) is exported as `assessment_eval_1.json` with a `nasde migrate-evals` hint, so the export is never silently empty. Filesystem-as-interface: `DIR` is any plain path (iCloud/Dropbox/git repo) — no cloud SDK. It scans Harbor artifacts (`result.json`/`config.json`/`assessment_eval*.json`/`agent/trajectory.json`/workspace), **not** the best-effort `EXPERIMENT_LOG.md`. `metrics.json` is a self-contained summary composed from `result.json`+`config.json`+`agent/trajectory.json` — including **token & cost economics** (`token_usage`, `cost_usd`, `pricing_as_of`, `reasoning_effort`); see the token-cost note below and [ADR-011](docs/adr/011-token-cost-metrics.md). The code diff is captured as a patch (`git diff HEAD` + untracked via `git ls-files -z` + `git diff --no-index`, never `git add` — the workspace `.git` index is left untouched; `-z`/NUL parsing means non-ASCII untracked filenames are not dropped under `core.quotepath`). Selection is a mixed positional list of job and/or trial dirs (auto-classified: a dir whose children have `result.json` is a job; else a dir whose own `result.json` carries a `trial_name` key is a trial; a dir with a job-level `result.json` but no trial-shaped children/`trial_name` is skipped with a warning rather than mis-exported as garbage); re-export is idempotent and merge-based (a trial is reported `exported` only when something new was copied, else `skipped`). Reuses `_collect_trial_dirs`/`_load_json`/`_compute_duration_sec`/`_resolve_agent_name` from `evaluator.py`. Deliberately does **not** model "experiments" (one job can belong to many — a future UI layer's concern).

- - **Token & cost metrics (ADR-011)**: every trial gets **token usage + USD cost** computed from the agent's `agent/trajectory.json` `final_metrics` and a versioned price catalog. `token_metrics.py` is the **single extractor** feeding both write paths: `evaluator.py` writes them onto `assessment_summary.json` (run) and `results_exporter.py` onto `metrics.json` (export). Definitions: `input = total_prompt_tokens` (full, cache included), `output = total_completion_tokens + extra.reasoning_output_tokens` (Codex reasoning folded into output), `total = input + output`. **Cost is "as if every run were the first"** — full prompt volume at full input rate, **no cache discount** — so it is deterministic and independent of run order / cache TTL (the prompt-token count is fixed for a task; the cache hit rate is not). **The scalar `token_efficiency`/`cost_efficiency` ratios were REMOVED** — `normalized_score / denominator` has an arbitrary zero (score 0 = empty rubric, unreachable), so the ranking is not invariant to a baseline shift; model comparison is now a **Pareto front** (quality vs cost, quality vs tokens), shift-invariant, living in the `nasde-benchmark-runner` skill, not the toolkit. The raw signals (`token_usage`, `cost_usd`, `pricing_as_of`, score) stay and are the source of truth. Economics are **per-trial** (one agent run) → they live on `AssessmentSummary`, not `EvaluatorGroupSummary`. `model_name` and `reasoning_effort` are stamped on the summary because cross-model analysis groups by `(agent_name, model_name, reasoning_effort)` (agent_name = variant name, does not distinguish models; a different effort is a different configuration, never averaged together — mirrors how a changed `dimensions_fingerprint` is a different benchmark). The `reasoning_effort` stamp is read back from the per-trial Harbor `config.json` (`config.agent.kwargs.reasoning_effort`); when no override was set the stamp is `""` (only explicit overrides are recorded — Codex's implicit `high` default is NOT fabricated, and an unset effort is a valid "family default" state). Pricing lives in the bundled `pricing.toml` (`pricing.py::load_pricing`), each model stamped with `as_of` + `source`; `cached_input_per_1m` is recorded for reference but **not** used in the cost formula. An **unpriced model** → `cost_usd` = `null` + a warning (token metrics still computed); a **missing/legacy trajectory** → all economics `null`. Never crashes the run. `nasde run` prints a per-`(agent, model, effort)` cost table (trials, score, tokens, $cost) plus the job path and an export hint (`runner.py::_print_job_summary`, called after assessment so the summaries exist); raw cost/token columns carry an inter-trial `±std` when the group has ≥2 trials (n=1 → bare value). Backfilling existing exports whose source jobs are gone is a **one-shot ad-hoc script** (reads the export's own flat `trajectory.json`) — deliberately NOT a CLI command.

+ - **Token & cost metrics (ADR-011)**: every trial gets **token usage + USD cost** computed from the agent's `agent/trajectory.json` `final_metrics` and a versioned price catalog. `token_metrics.py` is the **single extractor** feeding both write paths: `evaluator.py` writes them onto `assessment_summary.json` (run) and `results_exporter.py` onto `metrics.json` (export). Definitions: `input = total_prompt_tokens` (full, cache included), `output = total_completion_tokens + extra.reasoning_output_tokens` (Codex reasoning folded into output), `total = input + output`. **Cost is "as if every run were the first"** — full prompt volume at full input rate, **no cache discount** — so it is deterministic and independent of run order / cache TTL (the prompt-token count is fixed for a task; the cache hit rate is not). **The scalar `token_efficiency`/`cost_efficiency` ratios were REMOVED** — `normalized_score / denominator` has an arbitrary zero (score 0 = empty rubric, unreachable), so the ranking is not invariant to a baseline shift; model comparison is now a **Pareto front** (quality vs cost, quality vs tokens), shift-invariant, living in the `nasde-benchmark-runner` skill, not the toolkit. The raw signals (`token_usage`, `cost_usd`, `pricing_as_of`, score) stay and are the source of truth. Economics are **per-trial** (one agent run) → they live on `AssessmentSummary`, not `EvaluatorGroupSummary`. `model_name` and `reasoning_effort` are stamped on the summary because cross-model analysis groups by `(agent_name, model_name, reasoning_effort)` (agent_name = variant name, does not distinguish models; a different effort is a different configuration, never averaged together — mirrors how a changed `dimensions_fingerprint` is a different benchmark). The `reasoning_effort` stamp is read back from the per-trial Harbor `config.json` (`config.agent.kwargs.reasoning_effort`); when no override was set the stamp is `""` (only explicit overrides are recorded — Codex's implicit `high` default is NOT fabricated, and an unset effort is a valid "family default" state). Pricing is loaded via `pricing.py::load_pricing_layered(project_dir)` (ADR-013), which merges three layers per-model (whole-entry, higher wins): `<project>/pricing.toml` > `~/.nasde/pricing.toml` > bundled `pricing.toml`. **Convention, not config** — a file literally named `pricing.toml` (mirrors `assessment_dimensions.json`); no `[pricing]` key. User layer is a HOME dotfolder (`~/.nasde/`, like `~/.claude`/`~/.codex`/`~/.gemini`), deliberately NOT `platformdirs` (that maps to `~/Library/Application Support` on macOS = app-state, not user-editable config; `platformdirs` stays for cache in `update_check.py`). Both write paths thread `project_dir` so run (`assessment_summary.json`) and export (`metrics.json`) agree on cost — the ADR-011 single-extractor invariant. The merged catalog is NOT cached (depends on `project_dir` + on-disk contents); bundled `_load_bundled_pricing` keeps its `lru_cache`, `load_pricing(path)` is unchanged. An applied override prints a dim transparency line. **Layer provenance is exposed** (ADR-013): `pricing.py::resolve_pricing_layers(project_dir)` returns the ordered `PricingLayer` stack and `effective_pricing_with_source(project_dir)` returns `{model: (price, layer_name)}`. Surfaced three ways — `nasde pricing show [--show-source]` (sub-app, room for future `pricing validate`/`path`), a "Pricing used" table at the end of `nasde run` (only models in the run), and a `pricing_used.json` written by `results-export` (effective rate + layer per priced model, self-contained audit). The Rich table renderer is shared in `pricing_report.py::render_pricing_table` (used by `pricing show` + run summary). Each model stamped with `as_of` + `source`; `cached_input_per_1m` is recorded for reference but **not** used in the cost formula. An **unpriced model** → `cost_usd` = `null` + a warning (token metrics still computed); a **missing/legacy trajectory** → all economics `null`. Never crashes the run. `nasde run` prints a per-`(agent, model, effort)` cost table (trials, score, tokens, $cost) plus the job path and an export hint (`runner.py::_print_job_summary`, called after assessment so the summaries exist); raw cost/token columns carry an inter-trial `±std` when the group has ≥2 trials (n=1 → bare value). Backfilling existing exports whose source jobs are gone is a **one-shot ad-hoc script** (reads the export's own flat `trajectory.json`) — deliberately NOT a CLI command.

- See `docs/adr/` for detailed decision records.

## CLI reference

```

nasde run [OPTIONS] # Run benchmark (Harbor trial + assessment eval)

--variant TEXT # Variant name (default: from nasde.toml)

--tasks TEXT # Comma-separated task names (default: all)

--model TEXT # Model override

--effort TEXT # Reasoning-effort override (> variant.toml reasoning_effort > Harbor family default)

--timeout INT # Agent timeout override (default: task.toml [agent] timeout_sec)

--with-opik # Enable Opik tracing

--without-eval # Skip assessment evaluation

--eval-repetitions INT # Judge evaluations per trial (default: nasde.toml [evaluation], fallback 3)

--job-suffix TEXT # Custom suffix for job directory (default: random 6-char hex)

--harbor-env TEXT # Harbor execution environment (docker, daytona, modal, e2b, runloop, gke)

-C, --project-dir PATH # Path to benchmark project

nasde eval JOB_DIR [OPTIONS] # Re-run assessment on existing job

--with-opik # Upload scores to Opik

--eval-repetitions INT # Judge evaluations per trial (default: nasde.toml [evaluation], fallback 3)

-C, --project-dir PATH

nasde results-export PATHS... [OPTIONS] # [EXPERIMENTAL] Export trial artifact essence to a plain dir

-t, --to PATH # Destination dir (iCloud/Dropbox/git repo/any path) — required

-C, --project-dir PATH # PATHS may mix job and trial dirs (type auto-detected)

nasde calibrate publish PATHS... [OPTIONS] # Publish trial diffs + assessments as PRs/MRs for review

--repo TEXT # Sink repo URL or owner/repo slug (default: [calibration] repo)

--throttle FLOAT # Seconds between PR-creating calls (default: [calibration] throttle_sec)

-C, --project-dir PATH

nasde calibrate pull-comments PATHS... [OPTIONS] # Fetch human review comments from each trial's PR/MR

--repo TEXT # Sink repo URL or owner/repo slug (default: [calibration] repo)

--json # Machine-readable output for the calibration orchestrator agent

-C, --project-dir PATH

+ nasde pricing show [OPTIONS] # Print the effective merged price catalog (ADR-013)

+ --show-source # Add a Layer column (project/user/bundled) — debug

+ -C, --project-dir PATH

nasde init [PROJECT_DIR] # Scaffold new benchmark project

-n, --name TEXT

nasde install-skills [OPTIONS] # Copy bundled Claude Code authoring skills to a skills dir

--scope user|project # Default user: ~/.claude/skills; project: ./.claude/skills

--target-dir PATH # Custom skills directory (overrides --scope)

-f, --force # Overwrite existing skill directories

nasde harbor ... # Harbor CLI pass-through (view, jobs, trials, etc.)

nasde opik ... # Opik CLI pass-through (configure, usage-report, etc.)

```

## Benchmark project structure

A benchmark project managed by `nasde` has this layout:

```

my-benchmark/

nasde.toml # Project config (name, defaults, docker, evaluation, reporting)

assessment_dimensions.json # Scoring dimensions (benchmark-wide, each with independent max_score)

+ pricing.toml # Optional: per-project model price overrides (project > ~/.nasde > bundled, ADR-013)

tasks/

<task-name>/

task.toml # Task config: Harbor sections + [nasde.source] for auto-Dockerfile

instruction.md # Agent-facing task description

assessment_criteria.md # Per-task rubric for LLM-as-a-Judge

environment/Dockerfile # Docker container setup

tests/test.sh # Harbor verifier (writes 0/1 to /logs/verifier/reward.txt)

solution/solve.sh # Optional reference solution

variants/

<variant-name>/

variant.toml # Required: agent type + model (e.g. agent = "claude", model = "claude-sonnet-4-6")

CLAUDE.md # Claude Code instructions (injected into /app/CLAUDE.md)

AGENTS.md # Codex instructions (injected into /app/AGENTS.md)

GEMINI.md # Gemini CLI instructions (injected into /app/GEMINI.md)

skills/ # Optional: Claude skill snapshots (injected into /app/.claude/skills/)

<skill-name>/

SKILL.md # Skill content (snapshot for deterministic testing)

agents_skills/ # Optional: Codex skill snapshots (native skill-injection → $HOME/.agents/skills/)

<skill-name>/

SKILL.md # Skill content with YAML frontmatter (name + description)

gemini_skills/ # Optional: Gemini skill snapshots (native skill-injection → ~/.gemini/skills/)

<skill-name>/

SKILL.md # Skill content with YAML frontmatter (name + description)

harbor_config.json # Optional: agent import path + sandbox_files mapping

claude_config.json # Optional: MCP server configuration

evaluator_skills/ # Optional: skills for the evaluator agent

<skill-name>/

SKILL.md

evaluator_mcp.json # Optional: MCP server config for the evaluator agent

jobs/ # Trial output (gitignored)

```

## Key file formats

### nasde.toml

```toml

[project]

name = "my-benchmark"

version = "1.0.0"

[defaults]

variant = "vanilla"

# harbor_env = "daytona" # Optional: cloud sandbox provider (default: docker)

[docker]

base_image = "ubuntu:22.04"

build_commands = []

[evaluation]

backend = "claude" # "claude" (default) | "codex"

model = "claude-opus-4-7"

dimensions_file = "assessment_dimensions.json"

# max_turns = 60 # Max evaluator conversation turns (default 60)

# allowed_tools = ["Read", "Glob", "Grep"] # Override default tool whitelist

# mcp_config = "./evaluator_mcp.json" # MCP server config for evaluator

# skills_dir = "./evaluator_skills" # Skills directory for evaluator

# append_system_prompt = "" # Extra system prompt for evaluator

# include_trajectory = false # Include ATIF trajectory in evaluation (default: false)

[reporting]

platform = "opik"

project_name = "my-benchmark"

```

### assessment_dimensions.json

```json

{

"dimensions": [

{

"name": "snake_case_name",

"title": "Human-Readable Title",

"max_score": 10,

"description": "What this dimension measures (max_score can be any positive integer — pick what fits the granularity)"

}

]

}

```

Dimensions are benchmark-specific. Each dimension declares its own `max_score` (any positive integer) — there is no requirement that scores sum to a particular total or that you use a particular dimension count. Pick the scale per dimension that matches the granularity you can actually distinguish; `normalized_score` is computed from the actual sum of `max_score` values. See [ADR-008](docs/adr/008-independent-dimension-scales.md).

### task.toml

Single task config file, shared with Harbor. Harbor reads its standard sections (`[task]`, `[agent]`, `[environment]`, `[verifier]`, `[metadata]`) directly. nasde-specific fields live under `[nasde.*]` and are ignored by Harbor.

```toml

version = "1.0"

[task]

name = "my-benchmark/task-name" # Harbor requires org/name format; optional but recommended.

description = "Brief description"

[agent]

timeout_sec = 1800 # Per-task agent timeout (primary source — overridden only by --timeout flag).

[environment]

memory_mb = 4096 # Container memory limit (default: 2048). Claude Code needs 4096+.

[verifier]

timeout_sec = 300 # Per-task verifier timeout (runs tests/test.sh).

[nasde.source] # Only needed when task has no environment/Dockerfile (auto-generation).

git = "https://github.com/org/repo.git"

ref = "main"

[nasde.plugin] # Optional (ADR-009). Ship a local Claude Code plugin into the sandbox.

path = "../../../src/plugins/my-plugin" # dir with .claude-plugin/plugin.json (required)

ref = "abc1234" # optional git ref, same semantics as [nasde.source]

install_root = "/opt/my-plugin" # optional, default /opt/<plugin-name>

build = "bun install --frozen-lockfile" # optional, run at image-build time

[nasde.plugin.env] # optional, exported in the generated MCP wrapper

CLAUDE_PLUGIN_DATA = "/opt/my-plugin-data"

```

`[nasde.plugin]` is parsed into `config.PluginConfig` and consumed by

`docker.ensure_task_plugin()` + `plugin_registration.py`. The MCP server and

plugin skills are auto-registered — do NOT also hand-write

`[environment.mcp_servers]` for the plugin or copy its skills into a variant.

nasde writes the generated `[[environment.mcp_servers]]` into `task.toml`

between sentinel comments (fenced/idempotent); an author-declared server of

the same name is respected and left untouched.

**Required files per task directory** (Harbor conventions):

- `instruction.md` — agent-facing task description (required by Harbor)

- `tests/test.sh` — verifier script (required; writes 0/1 to `/logs/verifier/reward.txt`)

- `environment/Dockerfile` — OR `[nasde.source]` to auto-generate one

- `solution/solve.sh` — optional reference solution

- `assessment_criteria.md` — per-task LLM-as-a-Judge rubric

**Auto-generated environment:** If `environment/Dockerfile` is absent, nasde auto-generates one from `[nasde.source]` + `[docker]` config in `nasde.toml`. For local repos (paths not starting with `http`/`https`/`git`/`file`), the generated Dockerfile uses `COPY . /app` instead of `git clone`, and a `docker-compose.yaml` is also generated to set the Docker build context to the repo root.

**Timeout priority**: `--timeout` CLI flag > `task.toml [agent] timeout_sec` > Harbor default (1800s). Timeouts are per-task — there is no project-wide default in `nasde.toml`.

### variant.toml (required per variant)

Declares the agent type and the model the variant runs against. Every variant MUST specify both — there is no project-level model default (different agent families need different models).

```toml

agent = "claude" # "claude" | "codex" | "gemini"

model = "claude-sonnet-4-6" # REQUIRED. Model appropriate for the agent family.

reasoning_effort = "high" # Optional. Passed straight to the agent (no local validation, see below).

tasks = ["my-benchmark/task-a"] # Optional: task-scope. Restrict this variant to specific tasks.

[[skill]] # Optional (ADR-009): skill-by-reference. Works for any agent.

path = "../../../src/plugins/my-plugin/skills/my-skill" # source skill dir (required)

ref = "abc1234" # optional git ref, same semantics as [nasde.source]

```

**Model priority**: `--model` CLI flag > `variant.toml [model]`. Missing model in both places → SystemExit with a clear error.

**`reasoning_effort` (optional)**: controls how hard the model thinks. Priority: `--effort` CLI flag > `variant.toml reasoning_effort` > unset (unset → not passed → Harbor's per-family default; defaults are UNEQUAL across families). Threaded to Harbor via the agent's `reasoning_effort` kwarg, which Harbor turns into the right CLI flag (Claude `--effort`, Codex `-c model_reasoning_effort=`, Gemini ctor arg). **No local validation**: `_resolve_effort` only resolves priority + emptiness, then passes the value straight through. Each layer validates and rejects an unknown level itself — verified empirically: Claude/Gemini via Harbor's CliFlag `choices` (a `ValueError` at agent construction; Gemini also has per-model rules — `minimal` is Flash-only, 2.5 models reject effort entirely), Codex via its own CLI (`Error loading config.toml: unknown variant ...` at startup). A hardcoded per-family allow-list was deliberately removed — effort scales change too often and a stale list wrongly blocks a newly-valid level (it had `codex: low/medium/high`, but Codex actually accepts `none`/`minimal`/`low`/`medium`/`high`/`xhigh`). Typical levels for reference (not enforced): Claude `low`/`medium`/`high`/`xhigh`/`max`, Codex `none`/`minimal`/`low`/`medium`/`high`/`xhigh`, Gemini `minimal`/`low`/`medium`/`high`.

The effort is stamped onto each trial's `assessment_summary.json` / `metrics.json` (read back from `config.json` `config.agent.kwargs.reasoning_effort`; `""` when no override was set — only explicit overrides are recorded) and the run-summary economics group by `(agent_name, model_name, reasoning_effort)`.

**`tasks` (variant task-scope)**: optional list of task names this variant is

meant to run against. Use it for a *repo-specific* variant — e.g. a skill whose

examples reference one repo's conventions — so it never runs against the wrong

codebase. The scope is enforced in **both** run modes: with `--all-variants` a

scoped variant runs only against its declared tasks (others are skipped with a

SKIPPED status); with a single `--variant`, if none of the requested tasks fall in

the variant's scope the run aborts with a clear error. Either way the scope wins

over an explicit `--tasks` filter. Absent or empty → unscoped (runs against every

task, the default).

**`[[skill]]` (skill-by-reference)**: each entry references the **whole** skill

dir (incl. `references/`) from a source path — no copy under

`variants/<v>/skills/`. Optional `ref` reads from a temp git worktree at that

commit. **Routing is agent-aware** (ADR-012): for **Claude** the dir is staged

into `/app/.claude/skills/<name>/` via `plugin_registration.stage_skill_dir`

(same machinery as `[nasde.plugin]`); for **Codex/Gemini** the dir is resolved

by `collect_referenced_skill_dirs` and routed through Harbor's native

`config.agent.skills` (→ `$HOME/.agents/skills`, `~/.gemini/skills`), because

those CLIs never scan the `/app` cwd. The legacy `variants/<v>/skills/<name>/`

copy path still works (and carries `references/`).

### harbor_config.json (per variant)

Claude Code variant:

```json

{

"agents": [

{

"import_path": "nasde_toolkit.agents.configurable_claude:ConfigurableClaude",

"name": "variant-name",

"kwargs": {

"sandbox_files": {

"/app/CLAUDE.md": "/absolute/path/to/variants/variant-name/CLAUDE.md",

"/app/.claude/skills/my-skill/SKILL.md": "/absolute/path/to/variants/variant-name/skills/my-skill/SKILL.md"

}

]

}

```

Codex variant:

```json

{

"agents": [

{

"import_path": "nasde_toolkit.agents.configurable_codex:ConfigurableCodex",

"name": "variant-name",

"model_name": "gpt-5.3-codex",

"kwargs": {

"sandbox_files": {

"/app/AGENTS.md": "/absolute/path/to/variants/variant-name/AGENTS.md"

"reasoning_effort": "high"

}

]

}

```

Gemini CLI variant:

```json

{

"agents": [

{

"import_path": "nasde_toolkit.agents.configurable_gemini:ConfigurableGemini",

"name": "variant-name",

"model_name": "google/gemini-3-flash-preview",

"kwargs": {

"sandbox_files": {

"/app/GEMINI.md": "/absolute/path/to/variants/variant-name/GEMINI.md"

}

]

}

```

Critical: `"name"` field is REQUIRED — without it, Opik tagging breaks.

If `harbor_config.json` is absent, `nasde` auto-generates one based on `variant.toml` agent type (`CLAUDE.md` → Claude, `AGENTS.md` → Codex, `GEMINI.md` → Gemini).

### tests/test.sh (Harbor verifier)

Every failure path must `echo 0 > /logs/verifier/reward.txt && exit 1`.

Final success must `echo 1 > /logs/verifier/reward.txt && exit 0`.

## Known issues and workarounds

- **opik 2.x (and 1.10.x)**: token usage=None for Harbor spans — runtime monkeypatch in `runner.py` (`_patch_opik_deferred_metrics`). Defers Step span creation to `__setattr__` because Harbor assigns metrics after `Step.__init__`. See ADR-006. Remove when opik fixes upstream.

- **Nested Claude Code sessions**: `claude` CLI detects `CLAUDECODE` env var. Runner unsets it before assessment eval.

- **Opik REST API auth**: use `authorization: <OPIK_API_KEY>` header (not `Comet-Api-Key`), plus `Comet-Workspace` header.

- **Opik verification**: always use Python `urllib.request`, not curl (curl drops the `Comet-Workspace` header).