# nasde-toolkit AI coding agent evaluation toolkit. CLI entry point: `nasde`. ## Package structure ``` src/nasde_toolkit/ __init__.py # Re-exports __version__ from _version.py _version.py # Auto-generated by hatch-vcs from git tags (gitignored) cli.py # Typer CLI (init, run, eval, install-skills + harbor/opik pass-through) config.py # nasde.toml + task.toml parsing into dataclasses runner.py # Harbor Python API — variant resolution, config merging, Job execution evaluator.py # Post-hoc assessment via pluggable CLI subprocess backends evaluator_backends/ __init__.py protocol.py # EvaluatorBackend Protocol claude_subprocess.py # `claude -p` subprocess backend (default) codex_subprocess.py # `codex exec` subprocess backend results_exporter.py # `nasde results-export` — copy trial artifact essence to a plain dir pricing.py / pricing.toml # Versioned model price catalog (ADR-011) — load + compute_cost_usd token_metrics.py # Token usage + cost economics from trajectory final_metrics (ADR-011) calibration_publisher.py # `nasde calibrate` — publish trial diffs+assessments as PRs/MRs (ADR-010) calibration_resolve.py # Resolve [calibration] repo → (api slug, push url, platform) git_platform_backends/ __init__.py # create_git_backend(url) — URL auto-detect + factory protocol.py # GitPlatformBackend Protocol + RepoRef/PrRef/ReviewComment detect.py # detect_platform(url) — github/gitlab from host (+ override) git_ops.py # GIT layer — orphan base + feature branch push (not behind Protocol) github_cli.py # GitHubCliBackend (`gh`) gitlab_cli.py # GitLabCliBackend (`glab`) docker.py # Docker environment helpers ([nasde.source] + [nasde.plugin] staging) plugin_registration.py # Shared skill + MCP registration ([nasde.plugin] + variant [[skill]]) scaffold/ __init__.py # Project scaffolding templates and file creation skills_installer.py # `nasde install-skills` — copies bundled authoring skills to ~/.claude/skills/ agents/ __init__.py configurable_claude.py # Harbor-compatible Claude Code agent with sandbox file injection configurable_codex.py # Harbor-compatible Codex agent with sandbox file injection configurable_gemini.py # Harbor-compatible Gemini CLI agent with sandbox file injection tests/ pyproject.toml ``` ## How to run ```bash uv tool install . nasde --version ``` ## Testing ```bash uv run pytest ``` ## Code style 1. PEP 8 with type hints on all public functions. 2. `@dataclass` for internal data models (see `config.py`, `evaluator.py`). 3. Rich console for all CLI output — no bare `print()`. 4. Do NOT use comments in method bodies. Use descriptive function and variable names instead. 5. Split large functions into a hierarchy of private helpers with descriptive names. 6. Structure functions: public first (alphabetical), then private helpers ordered by dependency (caller before callee). 7. Snake_case for file and directory names. ## Architecture See [ARCHITECTURE.md](ARCHITECTURE.md) for the full system architecture with diagrams (end-to-end flow, trial lifecycle, cloud sandbox providers, assessment evaluation). Keep it in sync with code changes. ## Architecture decisions - **CLI framework**: Typer with Rich markup mode. The `app` object in `cli.py` is the entry point registered in `pyproject.toml` as `nasde`. - **Configuration**: Two-layer config — `nasde.toml` for project-level settings, `task.toml` per task (single file, shared with Harbor; nasde-specific fields live under `[nasde.*]`). Both parsed into `@dataclass` models in `config.py`. Task discovery walks `tasks/` (or `.nasde/tasks/`) automatically. - **Benchmark runner**: Uses Harbor Python API (`Job`, `JobConfig`) directly instead of subprocess. The runner merges variant config with task registry into a dict, passes it to `JobConfig.model_validate()`, then runs `await job.run()`. Opik tracking via `track_harbor()` (monkey-patches Harbor at runtime). - **Evaluator**: Subprocess-based assessment evaluation with pluggable backends. The evaluator spawns a CLI agent (`claude -p` or `codex exec`) as a subprocess to read trial artifacts and score them against assessment criteria. Backend selection via `[evaluation] backend` in `nasde.toml` (`"claude"` default, `"codex"` available). Claude backend uses `--output-format json` (without `--bare` — preserves OAuth/keychain auth for subscription billing and skills auto-discovery). Codex backend uses `--json --quiet --full-auto` for JSONL event streaming. Both backends respect the user's existing CLI authentication (OAuth subscription or API key) — no Agent SDK required, which avoids Anthropic's SDK-billing restrictions on subscription accounts. Configurable via `[evaluation]` in `nasde.toml` — backend, model, tools, MCP servers, skills, system prompt, and trajectory inclusion can all be customized. When `include_trajectory = true`, the evaluator also has access to the agent's ATIF trajectory (`agent/trajectory.json`). Default Claude model is `claude-opus-4-7`. Skills dir is mounted via temp-dir + `cwd` with `--add-dir ` so Claude's auto-discovery picks them up. **Repeated evaluations (output contract):** the judge is non-deterministic, so each trial is evaluated `eval_repetitions` times (default 3, `[evaluation] eval_repetitions` in `nasde.toml`, override `--eval-repetitions` on `run`/`eval`). The N evaluations of one trial run concurrently via `asyncio.gather` under the shared `--max-concurrent-eval` semaphore; the gather barrier ensures aggregation runs only after all N finish. Each is written **append-only** as `assessment_eval_.json` (never overwritten — the bug being fixed was a single `assessment_eval.json` clobbered on every re-eval). A derived `assessment_summary.json` holds per-cluster mean/std(sample,n-1)/min/max aggregates: averages are computed **only within one `(evaluator_model, dimensions_fingerprint)` cluster**. A different judge model OR a changed `assessment_dimensions.json` rubric (added/removed dimension, changed `max_score`, OR changed `description`) is a different benchmark and is never averaged together — the fingerprint is the first 12 hex chars of the sha256 of the normalized (`sort_keys`) rubric JSON, stamped onto each `assessment_eval_.json` at eval time (legacy files with no fingerprint → `""` cluster). Clustering by fingerprint also makes the per-group `members[0]` dimension/max_score read correct by construction. The largest cluster (tie → newest) is `dominant`. Opik upload uses the **dominant cluster's mean** (plus per-dimension std and `eval_n`), not the last evaluation. Legacy jobs are normalized by `eval_migration.py`, exposed as the **hidden** one-shot command `nasde migrate-evals` (`@app.command(hidden=True)` — invokable, but kept out of `nasde --help` and user docs because it is a migration tool, not a routine command; its tests stay to guard the logic). It takes a `jobs/` root (recurses) or a single trial dir, not a single job dir. - **Variant system**: Each variant is a directory under `variants/` with a required `variant.toml` declaring the agent type (`agent = "claude"`, `agent = "codex"`, or `agent = "gemini"`). For Claude Code variants, `CLAUDE.md` is injected into `/app/CLAUDE.md`; for Codex variants, `AGENTS.md` is injected into `/app/AGENTS.md`; for Gemini CLI variants, `GEMINI.md` is injected into `/app/GEMINI.md`. An optional `skills/` subdirectory contains Claude skill snapshots — each `skills//` is injected **whole** (incl. `references/` and sibling files, via the shared `plugin_registration.stage_skill_dir`) into `/app/.claude/skills//`. A `variant.toml` may also declare a `[[skill]]` array (skill-by-reference, ADR-009): each entry points at a source skill dir (optional `ref`) staged whole into `/app/.claude/skills//` without a copy under `variants/`. An optional `agents_skills/` subdirectory contains Codex skill snapshots — all files under `agents_skills//` are injected into `/app/.agents/skills//`. An optional `gemini_skills/` subdirectory contains Gemini skill snapshots — all files under `gemini_skills//` are injected into `/app/.gemini/skills//`. If no `harbor_config.json` exists, one is auto-generated from `variant.toml`. - **Codex ChatGPT OAuth**: Harbor's `Codex` agent uploads `~/.codex/auth.json` into the sandbox as `$CODEX_HOME/auth.json` — but **harbor 0.13 made this opt-in**: the agent now *defaults to `OPENAI_API_KEY`* and only uploads the OAuth `auth.json` when `CODEX_AUTH_JSON_PATH` (a specific file) or `CODEX_FORCE_AUTH_JSON` (truthy → `~/.codex/auth.json`) is set. Without an API key it would otherwise write an **empty** key into the sandbox (`Incorrect API key provided: ''` on `wss://api.openai.com/v1/responses`). So `_ensure_auth` opts into OAuth automatically: when no `OPENAI_API_KEY`/`CODEX_API_KEY` is set but `~/.codex/auth.json` exists, it sets `CODEX_FORCE_AUTH_JSON=true` (respecting a user-set `CODEX_AUTH_JSON_PATH`/`CODEX_FORCE_AUTH_JSON`). API key (`OPENAI_API_KEY`) always takes priority over OAuth. Tokens are not extracted back from sandbox after runs. Validate with `source scripts/export_codex_oauth_token.sh`. The runner bridges `CODEX_API_KEY` → `OPENAI_API_KEY` in `_ensure_auth` for users who prefer the Codex-specific env var name. - **Gemini Google OAuth**: When `~/.gemini/oauth_creds.json` exists (created by `gemini login`) and no API key env var is set (`GEMINI_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_APPLICATION_CREDENTIALS`), `ConfigurableGemini` injects the OAuth credentials into the sandbox via env var. API key always takes priority over OAuth. Validate with `source scripts/export_gemini_oauth_token.sh`. - **All dependencies are core**: `harbor`, `opik` are in `[project.dependencies]`. The evaluator depends on having `claude` CLI (default) or `codex` CLI installed and authenticated on the host — not bundled. No optional extras — `uv tool install .` gives full functionality. Assessment evaluation is on by default (`--without-eval` to skip). - **Versioning**: Derived from git tags via `hatch-vcs` plugin. Tag `v0.1.0` → version `0.1.0`. Commits after a tag produce dev versions like `0.1.1.dev3+gabcdef`. `_version.py` is auto-generated at build time and gitignored. See ADR-007. - **Bundled authoring skills**: `.claude/skills/nasde-benchmark-*` are shipped inside the wheel via `[tool.hatch.build.targets.wheel.force-include]` → `nasde_toolkit/_bundled_skills/`. `nasde install-skills` discovers them via `importlib.resources` (wheel install) with a fallback to the live `.claude/skills/` when running from an editable / source checkout. `nasde-dev` is deliberately excluded from the bundle (internal dev skill, not for end users). - **Auto-generated Dockerfile**: When a task has no `environment/Dockerfile`, nasde generates one from `source.git` + `[docker]`. For local paths, also generates `docker-compose.yaml` to override the build context. See `docker.py:ensure_task_environment()`. - **`[nasde.plugin]` (ADR-009)**: Ships a local Claude Code plugin (dir with `.claude-plugin/plugin.json`) into the sandbox with one `task.toml` declaration. `docker.py:ensure_task_plugin()` stages the plugin tree (at `ref` via a temp worktree) into a gitignored `_nasde-plugin/` inside the **active** build context (Harbor pins context to `environment/`; with `[nasde.source]` the context is the source repo/worktree — nasde reads it back from the generated compose and stages there), then appends a sentinel-fenced `COPY`+build stage to the Dockerfile (generated base if none, hand-written preserved). `plugin_registration.py` then registers the plugin's own `skills/` (whole dirs) and injects its MCP server (from `/.mcp.json`, env-wrapped) into the task's `task.toml` — idempotent, fenced, never clobbers an author-declared same-name server. Skill-by-reference (`[[skill]]` in `variant.toml`) feeds the same skill-registration machinery. Derived sandbox files are merged into the variant's `harbor_config.json` each run. MCP injection writes `task.toml` because Harbor reads MCP servers only from there (`trial.py` → `task.config.environment.mcp_servers`). - **Pass-through CLI**: `nasde harbor ...` delegates to Harbor's Typer app via `add_typer()`. `nasde opik ...` forwards args to Opik's Click CLI via `ctx.args`. - **Rubric calibration (ADR-010)**: `nasde calibrate publish PATHS...` / `pull-comments` close the loop between the LLM-as-a-Judge and a human reviewer by publishing trial diffs + assessments as PRs/MRs and pulling review comments back for rubric tuning. Two layers, deliberately separated: **GIT** (`git_platform_backends/git_ops.py` — `git push`/`ls-remote`, platform-agnostic, subprocess pattern from `docker.py`, not behind a Protocol) and **PLATFORM** (`git_platform_backends/` behind a `@runtime_checkable GitPlatformBackend` Protocol — `repo_exists`/`find_open_pr_for_branch`/`create_pr`/`fetch_pr_comments`/`validate_cli_installed`/`validate_auth`, mirroring `evaluator_backends/`). The base is keyed on `(repo, commit)` as an **orphan branch** `base/-` seeded once via `git archive HEAD` from the trial workspace (git deduplicates blobs by content across orphan bases — no shared ancestor needed); each trial is a feature branch `calib/-/` = base + the agent's `changes.patch` applied as a real commit + `.calibration/` files (no trajectory — secrets/clutter). `.calibration/` carries the reviewer's context: the task's `instruction.md` + `assessment_criteria.md` + `assessment_dimensions.json` (resolved from `result.json` `task_name`/`source`, trying both `tasks/` and `evals//tasks/` layouts), all `assessment_eval_.json`, `assessment_summary.json`, and `metrics.json`. Idempotency is **open-only**: `find_open_pr_for_branch` matches only OPEN PRs/MRs (`gh pr list --state open`, `glab mr list` default), so a re-run skips a live round but lets a fresh round publish once the prior one is closed. The PR body is a pure transform of the dominant `AssessmentSummary` cluster (`calibration_publisher._render_pr_body`). Backend is **auto-detected from the sink repo URL host** (`github.com`→`gh`, `*gitlab*`→`glab`; `[calibration] platform` overrides for self-hosted) — no `backend` config field, eliminating the backend≠host mismatch. Preflight before any work: detect → `validate_cli_installed` (`shutil.which`, precise per-platform message) → `validate_auth` (`gh|glab auth status` exit code) → `repo_exists` (parses OUTPUT — `gh repo view` exits 0 even for a missing repo). Repo creation is out of scope (push creates branches ad-hoc in an existing repo). Reuses `_expand_to_trials`/`_capture_patch`/`_build_metrics` from `results_exporter.py` and `_aggregate_evaluations`/`_load_raw_evaluations`/`AssessmentSummary` from `evaluator.py`. Prerequisites mirror the evaluator's CLI requirement (ADR-002): `git` + `gh`/`glab` + login, no SDK, CLI keyring holds auth. The `nasde-benchmark-calibration` skill orchestrates the human-in-the-loop flow. - **Results export (EXPERIMENTAL)**: `results_exporter.py` + `nasde results-export PATHS... --to DIR` copy the analytic *essence* of trial artifacts out of the gitignored `jobs/` tree into a flat per-trial layout (`DIR/__/` with `metrics.json`, `assessment_eval_*.json` (all repetitions), `assessment_summary.json`, `trajectory.json`, `changes.patch`, `verifier_stdout.txt`, `reward.txt`). Re-export **merges**: missing eval files are copied and the summary/metrics refreshed, while immutable files (trajectory, patch) are left as-is — so evaluations added after a first export are picked up. A legacy bare `assessment_eval.json` (pre-migration trial) is exported as `assessment_eval_1.json` with a `nasde migrate-evals` hint, so the export is never silently empty. Filesystem-as-interface: `DIR` is any plain path (iCloud/Dropbox/git repo) — no cloud SDK. It scans Harbor artifacts (`result.json`/`config.json`/`assessment_eval*.json`/`agent/trajectory.json`/workspace), **not** the best-effort `EXPERIMENT_LOG.md`. `metrics.json` is a self-contained summary composed from `result.json`+`config.json`+`agent/trajectory.json` — including **token & cost economics** (`token_usage`, `cost_usd`, `pricing_as_of`, `reasoning_effort`); see the token-cost note below and [ADR-011](docs/adr/011-token-cost-metrics.md). The code diff is captured as a patch (`git diff HEAD` + untracked via `git ls-files -z` + `git diff --no-index`, never `git add` — the workspace `.git` index is left untouched; `-z`/NUL parsing means non-ASCII untracked filenames are not dropped under `core.quotepath`). Selection is a mixed positional list of job and/or trial dirs (auto-classified: a dir whose children have `result.json` is a job; else a dir whose own `result.json` carries a `trial_name` key is a trial; a dir with a job-level `result.json` but no trial-shaped children/`trial_name` is skipped with a warning rather than mis-exported as garbage); re-export is idempotent and merge-based (a trial is reported `exported` only when something new was copied, else `skipped`). Reuses `_collect_trial_dirs`/`_load_json`/`_compute_duration_sec`/`_resolve_agent_name` from `evaluator.py`. Deliberately does **not** model "experiments" (one job can belong to many — a future UI layer's concern). - **Token & cost metrics (ADR-011)**: every trial gets **token usage + USD cost** computed from the agent's `agent/trajectory.json` `final_metrics` and a versioned price catalog. `token_metrics.py` is the **single extractor** feeding both write paths: `evaluator.py` writes them onto `assessment_summary.json` (run) and `results_exporter.py` onto `metrics.json` (export). Definitions: `input = total_prompt_tokens` (full, cache included), `output = total_completion_tokens + extra.reasoning_output_tokens` (Codex reasoning folded into output), `total = input + output`. **Cost is "as if every run were the first"** — full prompt volume at full input rate, **no cache discount** — so it is deterministic and independent of run order / cache TTL (the prompt-token count is fixed for a task; the cache hit rate is not). **The scalar `token_efficiency`/`cost_efficiency` ratios were REMOVED** — `normalized_score / denominator` has an arbitrary zero (score 0 = empty rubric, unreachable), so the ranking is not invariant to a baseline shift; model comparison is now a **Pareto front** (quality vs cost, quality vs tokens), shift-invariant, living in the `nasde-benchmark-runner` skill, not the toolkit. The raw signals (`token_usage`, `cost_usd`, `pricing_as_of`, score) stay and are the source of truth. Economics are **per-trial** (one agent run) → they live on `AssessmentSummary`, not `EvaluatorGroupSummary`. `model_name` and `reasoning_effort` are stamped on the summary because cross-model analysis groups by `(agent_name, model_name, reasoning_effort)` (agent_name = variant name, does not distinguish models; a different effort is a different configuration, never averaged together — mirrors how a changed `dimensions_fingerprint` is a different benchmark). The `reasoning_effort` stamp is read back from the per-trial Harbor `config.json` (`config.agent.kwargs.reasoning_effort`); when no override was set the stamp is `""` (only explicit overrides are recorded — Codex's implicit `high` default is NOT fabricated, and an unset effort is a valid "family default" state). Pricing lives in the bundled `pricing.toml` (`pricing.py::load_pricing`), each model stamped with `as_of` + `source`; `cached_input_per_1m` is recorded for reference but **not** used in the cost formula. An **unpriced model** → `cost_usd` = `null` + a warning (token metrics still computed); a **missing/legacy trajectory** → all economics `null`. Never crashes the run. `nasde run` prints a per-`(agent, model, effort)` cost table (trials, score, tokens, $cost) plus the job path and an export hint (`runner.py::_print_job_summary`, called after assessment so the summaries exist); raw cost/token columns carry an inter-trial `±std` when the group has ≥2 trials (n=1 → bare value). Backfilling existing exports whose source jobs are gone is a **one-shot ad-hoc script** (reads the export's own flat `trajectory.json`) — deliberately NOT a CLI command. - See `docs/adr/` for detailed decision records. ## CLI reference ``` nasde run [OPTIONS] # Run benchmark (Harbor trial + assessment eval) --variant TEXT # Variant name (default: from nasde.toml) --tasks TEXT # Comma-separated task names (default: all) --model TEXT # Model override --effort TEXT # Reasoning-effort override (> variant.toml reasoning_effort > Harbor family default) --timeout INT # Agent timeout override (default: task.toml [agent] timeout_sec) --with-opik # Enable Opik tracing --without-eval # Skip assessment evaluation --eval-repetitions INT # Judge evaluations per trial (default: nasde.toml [evaluation], fallback 3) --job-suffix TEXT # Custom suffix for job directory (default: random 6-char hex) --harbor-env TEXT # Harbor execution environment (docker, daytona, modal, e2b, runloop, gke) -C, --project-dir PATH # Path to benchmark project nasde eval JOB_DIR [OPTIONS] # Re-run assessment on existing job --with-opik # Upload scores to Opik --eval-repetitions INT # Judge evaluations per trial (default: nasde.toml [evaluation], fallback 3) -C, --project-dir PATH nasde results-export PATHS... [OPTIONS] # [EXPERIMENTAL] Export trial artifact essence to a plain dir -t, --to PATH # Destination dir (iCloud/Dropbox/git repo/any path) — required -C, --project-dir PATH # PATHS may mix job and trial dirs (type auto-detected) nasde calibrate publish PATHS... [OPTIONS] # Publish trial diffs + assessments as PRs/MRs for review --repo TEXT # Sink repo URL or owner/repo slug (default: [calibration] repo) --throttle FLOAT # Seconds between PR-creating calls (default: [calibration] throttle_sec) -C, --project-dir PATH nasde calibrate pull-comments PATHS... [OPTIONS] # Fetch human review comments from each trial's PR/MR --repo TEXT # Sink repo URL or owner/repo slug (default: [calibration] repo) --json # Machine-readable output for the calibration orchestrator agent -C, --project-dir PATH nasde init [PROJECT_DIR] # Scaffold new benchmark project -n, --name TEXT nasde install-skills [OPTIONS] # Copy bundled Claude Code authoring skills to a skills dir --scope user|project # Default user: ~/.claude/skills; project: ./.claude/skills --target-dir PATH # Custom skills directory (overrides --scope) -f, --force # Overwrite existing skill directories nasde harbor ... # Harbor CLI pass-through (view, jobs, trials, etc.) nasde opik ... # Opik CLI pass-through (configure, usage-report, etc.) ``` ## Benchmark project structure A benchmark project managed by `nasde` has this layout: ``` my-benchmark/ nasde.toml # Project config (name, defaults, docker, evaluation, reporting) assessment_dimensions.json # Scoring dimensions (benchmark-wide, each with independent max_score) tasks/ / task.toml # Task config: Harbor sections + [nasde.source] for auto-Dockerfile instruction.md # Agent-facing task description assessment_criteria.md # Per-task rubric for LLM-as-a-Judge environment/Dockerfile # Docker container setup tests/test.sh # Harbor verifier (writes 0/1 to /logs/verifier/reward.txt) solution/solve.sh # Optional reference solution variants/ / variant.toml # Required: agent type + model (e.g. agent = "claude", model = "claude-sonnet-4-6") CLAUDE.md # Claude Code instructions (injected into /app/CLAUDE.md) AGENTS.md # Codex instructions (injected into /app/AGENTS.md) GEMINI.md # Gemini CLI instructions (injected into /app/GEMINI.md) skills/ # Optional: Claude skill snapshots (injected into /app/.claude/skills/) / SKILL.md # Skill content (snapshot for deterministic testing) agents_skills/ # Optional: Codex skill snapshots (injected into /app/.agents/skills/) / SKILL.md # Skill content with YAML frontmatter (name + description) gemini_skills/ # Optional: Gemini skill snapshots (injected into /app/.gemini/skills/) / SKILL.md # Skill content with YAML frontmatter (name + description) harbor_config.json # Optional: agent import path + sandbox_files mapping claude_config.json # Optional: MCP server configuration evaluator_skills/ # Optional: skills for the evaluator agent / SKILL.md evaluator_mcp.json # Optional: MCP server config for the evaluator agent jobs/ # Trial output (gitignored) ``` ## Key file formats ### nasde.toml ```toml [project] name = "my-benchmark" version = "1.0.0" [defaults] variant = "vanilla" # harbor_env = "daytona" # Optional: cloud sandbox provider (default: docker) [docker] base_image = "ubuntu:22.04" build_commands = [] [evaluation] backend = "claude" # "claude" (default) | "codex" model = "claude-opus-4-7" dimensions_file = "assessment_dimensions.json" # max_turns = 60 # Max evaluator conversation turns (default 60) # allowed_tools = ["Read", "Glob", "Grep"] # Override default tool whitelist # mcp_config = "./evaluator_mcp.json" # MCP server config for evaluator # skills_dir = "./evaluator_skills" # Skills directory for evaluator # append_system_prompt = "" # Extra system prompt for evaluator # include_trajectory = false # Include ATIF trajectory in evaluation (default: false) [reporting] platform = "opik" project_name = "my-benchmark" ``` ### assessment_dimensions.json ```json { "dimensions": [ { "name": "snake_case_name", "title": "Human-Readable Title", "max_score": 10, "description": "What this dimension measures (max_score can be any positive integer — pick what fits the granularity)" } ] } ``` Dimensions are benchmark-specific. Each dimension declares its own `max_score` (any positive integer) — there is no requirement that scores sum to a particular total or that you use a particular dimension count. Pick the scale per dimension that matches the granularity you can actually distinguish; `normalized_score` is computed from the actual sum of `max_score` values. See [ADR-008](docs/adr/008-independent-dimension-scales.md). ### task.toml Single task config file, shared with Harbor. Harbor reads its standard sections (`[task]`, `[agent]`, `[environment]`, `[verifier]`, `[metadata]`) directly. nasde-specific fields live under `[nasde.*]` and are ignored by Harbor. ```toml version = "1.0" [task] name = "my-benchmark/task-name" # Harbor requires org/name format; optional but recommended. description = "Brief description" [agent] timeout_sec = 1800 # Per-task agent timeout (primary source — overridden only by --timeout flag). [environment] memory_mb = 4096 # Container memory limit (default: 2048). Claude Code needs 4096+. [verifier] timeout_sec = 300 # Per-task verifier timeout (runs tests/test.sh). [nasde.source] # Only needed when task has no environment/Dockerfile (auto-generation). git = "https://github.com/org/repo.git" ref = "main" [nasde.plugin] # Optional (ADR-009). Ship a local Claude Code plugin into the sandbox. path = "../../../src/plugins/my-plugin" # dir with .claude-plugin/plugin.json (required) ref = "abc1234" # optional git ref, same semantics as [nasde.source] install_root = "/opt/my-plugin" # optional, default /opt/ build = "bun install --frozen-lockfile" # optional, run at image-build time [nasde.plugin.env] # optional, exported in the generated MCP wrapper CLAUDE_PLUGIN_DATA = "/opt/my-plugin-data" ``` `[nasde.plugin]` is parsed into `config.PluginConfig` and consumed by `docker.ensure_task_plugin()` + `plugin_registration.py`. The MCP server and plugin skills are auto-registered — do NOT also hand-write `[environment.mcp_servers]` for the plugin or copy its skills into a variant. nasde writes the generated `[[environment.mcp_servers]]` into `task.toml` between sentinel comments (fenced/idempotent); an author-declared server of the same name is respected and left untouched. **Required files per task directory** (Harbor conventions): - `instruction.md` — agent-facing task description (required by Harbor) - `tests/test.sh` — verifier script (required; writes 0/1 to `/logs/verifier/reward.txt`) - `environment/Dockerfile` — OR `[nasde.source]` to auto-generate one - `solution/solve.sh` — optional reference solution - `assessment_criteria.md` — per-task LLM-as-a-Judge rubric **Auto-generated environment:** If `environment/Dockerfile` is absent, nasde auto-generates one from `[nasde.source]` + `[docker]` config in `nasde.toml`. For local repos (paths not starting with `http`/`https`/`git`/`file`), the generated Dockerfile uses `COPY . /app` instead of `git clone`, and a `docker-compose.yaml` is also generated to set the Docker build context to the repo root. **Timeout priority**: `--timeout` CLI flag > `task.toml [agent] timeout_sec` > Harbor default (1800s). Timeouts are per-task — there is no project-wide default in `nasde.toml`. ### variant.toml (required per variant) Declares the agent type and the model the variant runs against. Every variant MUST specify both — there is no project-level model default (different agent families need different models). ```toml agent = "claude" # "claude" | "codex" | "gemini" model = "claude-sonnet-4-6" # REQUIRED. Model appropriate for the agent family. reasoning_effort = "high" # Optional. Passed straight to the agent (no local validation, see below). tasks = ["my-benchmark/task-a"] # Optional: task-scope. Restrict this variant to specific tasks. [[skill]] # Optional (ADR-009): skill-by-reference, Claude only. path = "../../../src/plugins/my-plugin/skills/my-skill" # source skill dir (required) ref = "abc1234" # optional git ref, same semantics as [nasde.source] ``` **Model priority**: `--model` CLI flag > `variant.toml [model]`. Missing model in both places → SystemExit with a clear error. **`reasoning_effort` (optional)**: controls how hard the model thinks. Priority: `--effort` CLI flag > `variant.toml reasoning_effort` > unset (unset → not passed → Harbor's per-family default; defaults are UNEQUAL across families). Threaded to Harbor via the agent's `reasoning_effort` kwarg, which Harbor turns into the right CLI flag (Claude `--effort`, Codex `-c model_reasoning_effort=`, Gemini ctor arg). **No local validation**: `_resolve_effort` only resolves priority + emptiness, then passes the value straight through. Each layer validates and rejects an unknown level itself — verified empirically: Claude/Gemini via Harbor's CliFlag `choices` (a `ValueError` at agent construction; Gemini also has per-model rules — `minimal` is Flash-only, 2.5 models reject effort entirely), Codex via its own CLI (`Error loading config.toml: unknown variant ...` at startup). A hardcoded per-family allow-list was deliberately removed — effort scales change too often and a stale list wrongly blocks a newly-valid level (it had `codex: low/medium/high`, but Codex actually accepts `none`/`minimal`/`low`/`medium`/`high`/`xhigh`). Typical levels for reference (not enforced): Claude `low`/`medium`/`high`/`xhigh`/`max`, Codex `none`/`minimal`/`low`/`medium`/`high`/`xhigh`, Gemini `minimal`/`low`/`medium`/`high`. The effort is stamped onto each trial's `assessment_summary.json` / `metrics.json` (read back from `config.json` `config.agent.kwargs.reasoning_effort`; `""` when no override was set — only explicit overrides are recorded) and the run-summary economics group by `(agent_name, model_name, reasoning_effort)`. **`tasks` (variant task-scope)**: optional list of task names this variant is meant to run against. Use it for a *repo-specific* variant — e.g. a skill whose examples reference one repo's conventions — so it never runs against the wrong codebase. The scope is enforced in **both** run modes: with `--all-variants` a scoped variant runs only against its declared tasks (others are skipped with a SKIPPED status); with a single `--variant`, if none of the requested tasks fall in the variant's scope the run aborts with a clear error. Either way the scope wins over an explicit `--tasks` filter. Absent or empty → unscoped (runs against every task, the default). **`[[skill]]` (skill-by-reference)**: each entry stages the **whole** skill dir (incl. `references/`) into `/app/.claude/skills//` from a source path — no copy under `variants//skills/`. Optional `ref` reads from a temp git worktree at that commit. The legacy `variants//skills//` copy path still works (and now also carries `references/`). Both feed the same `plugin_registration.stage_skill_dir` machinery as `[nasde.plugin]`. ### harbor_config.json (per variant) Claude Code variant: ```json { "agents": [ { "import_path": "nasde_toolkit.agents.configurable_claude:ConfigurableClaude", "name": "variant-name", "kwargs": { "sandbox_files": { "/app/CLAUDE.md": "/absolute/path/to/variants/variant-name/CLAUDE.md", "/app/.claude/skills/my-skill/SKILL.md": "/absolute/path/to/variants/variant-name/skills/my-skill/SKILL.md" } } } ] } ``` Codex variant: ```json { "agents": [ { "import_path": "nasde_toolkit.agents.configurable_codex:ConfigurableCodex", "name": "variant-name", "model_name": "gpt-5.3-codex", "kwargs": { "sandbox_files": { "/app/AGENTS.md": "/absolute/path/to/variants/variant-name/AGENTS.md" }, "reasoning_effort": "high" } } ] } ``` Gemini CLI variant: ```json { "agents": [ { "import_path": "nasde_toolkit.agents.configurable_gemini:ConfigurableGemini", "name": "variant-name", "model_name": "google/gemini-3-flash-preview", "kwargs": { "sandbox_files": { "/app/GEMINI.md": "/absolute/path/to/variants/variant-name/GEMINI.md" } } } ] } ``` Critical: `"name"` field is REQUIRED — without it, Opik tagging breaks. If `harbor_config.json` is absent, `nasde` auto-generates one based on `variant.toml` agent type (`CLAUDE.md` → Claude, `AGENTS.md` → Codex, `GEMINI.md` → Gemini). ### tests/test.sh (Harbor verifier) Every failure path must `echo 0 > /logs/verifier/reward.txt && exit 1`. Final success must `echo 1 > /logs/verifier/reward.txt && exit 0`. ## Known issues and workarounds - **opik 2.x (and 1.10.x)**: token usage=None for Harbor spans — runtime monkeypatch in `runner.py` (`_patch_opik_deferred_metrics`). Defers Step span creation to `__setattr__` because Harbor assigns metrics after `Step.__init__`. See ADR-006. Remove when opik fixes upstream. - **Nested Claude Code sessions**: `claude` CLI detects `CLAUDECODE` env var. Runner unsets it before assessment eval. - **Opik REST API auth**: use `authorization: ` header (not `Comet-Api-Key`), plus `Comet-Workspace` header. - **Opik verification**: always use Python `urllib.request`, not curl (curl drops the `Comet-Workspace` header).