CLAUDE.md@benchmark · diff
git:20260428.35e38bf to git:20260504.6ca8791
115 added, 141 removed. Audit A to A.
# OntoSkills Benchmark
## Prerequisites
### Environment variables
- `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` — required for both agents
- `ANTHROPIC_BASE_URL` — set to `https://api.z.ai/api/anthropic` (proxy)
- Model ID: `glm-5.1` (via proxy, NOT a standard Claude model)
### Python dependencies
```
- datasets — HuggingFace dataset loading (installed)
- anthropic — API client (installed)
- rdflib — TTL parsing for content coverage (installed)
- requests — HTTP client for API calls (installed)
- pdfplumber — PDF text extraction (installed)
- python-docx — Word document creation (installed)
- python-pptx — PowerPoint generation (installed)
- pandas — Data manipulation (installed)
- youtube-transcript-api — YouTube transcript fetching (installed)
+ benchflow — BenchFlow SDK (container lifecycle, retry, scoring)
+ harbor — Harbor (Docker environment management, verifier)
+ anthropic — API client
+ rdflib — TTL parsing for content coverage
+ datasets — HuggingFace dataset loading
```
### NOT available
- - `gaia-benchmark/GAIA` — gated dataset on HuggingFace. Requires `huggingface-cli login` first. NOT authenticated currently.
+ - `gaia-benchmark/GAIA` — gated dataset on HuggingFace. Requires `huggingface-cli login` first.
### Binary prerequisites
- - `ontomcp` at `~/.ontoskills/bin/ontomcp` — Rust MCP server, rebuild from `mcp/` if outdated
- - Compiled TTLs at `~/.ontoskills/packages/` — 1058 TTL files (640 skills + sub-skills), 11 author packages
+ - `ontomcp` at `~/.ontoskills/bin/ontomcp` — Rust MCP server
+ - Compiled TTLs at `~/.ontoskills/packages/` — TTL files compiled from skills
+ - `docker` (or `podman` with docker alias) — container runtime
## Architecture
```
benchmark/
├── run.py # Main orchestrator (CLI entry point)
+ ├── state.py # BenchmarkState — resume, incremental save
├── content_coverage.py # Parser coverage + knowledge yield (no API calls)
├── config.py # Model pricing, benchmark definitions
├── agents/
│ ├── base.py # BaseAgent with Anthropic API, run-loop, retry
- │ ├── claudecode.py # ClaudeCodeAgent: CLI-based agent (--print --bare mode)
- │ ├── traditional.py # Skill registry + read_skill tool (like Claude Code)
- │ ├── ontoskills.py # 5 MCP tools (search, get_skill_context, prefetch_knowledge, etc.)
+ │ ├── claudecode.py # ClaudeCodeAgent: legacy host-based CLI agent
+ │ ├── traditional.py # Skill registry + read_skill tool (API mode)
+ │ ├── ontoskills.py # Single ontoskill MCP tool in API mode
│ └── utils.py # Shared utilities (extract_python_code)
├── wrappers/
│ ├── gaia.py # GAIA: Q&A with file attachments
│ ├── perpackage.py # Per-package: run selected packages only
- │ ├── skillsbench.py # SkillsBench: Docker-based deterministic eval (podman + pytest)
+ │ ├── skillsbench.py # SkillsBench: ACP + ACP-MCP via BenchFlow Trial
│ ├── swebench.py # SWE-bench: repo checkout + diff patch generation
│ └── tau2bench.py # tau2-bench: agent benchmarking wrapper
├── reporting/
│ ├── metrics.py # compute_comparison()
│ ├── comparison.py # generate_comparison_report()
- │ └── chart_data.py # chart-ready JSON generation (Chart.js/D3)
+ │ └── chart_data.py # chart-ready JSON generation
├── mcp_client/
│ └── client.py # JSON-RPC MCP client for ontomcp subprocess
├── tests/ # Unit tests for benchmark framework
├── data/ # Downloaded datasets
└── results/ # Benchmark output (JSON + comparison.md)
```
## Running benchmarks
- ### Content coverage (instant, no API)
- ```bash
- ANTHROPIC_API_KEY="$ANTHROPIC_AUTH_TOKEN" \
- python benchmark/content_coverage.py --verbose --ttl-dir ~/.ontoskills/packages --json benchmark/results/content_coverage.json
- ```
-
- ### SWE-bench (requires API)
- ```bash
- ANTHROPIC_API_KEY="$ANTHROPIC_AUTH_TOKEN" \
- python benchmark/run.py --benchmark swebench --mode both --max-tasks 25 --model glm-5.1 \
- --skills-dir .agents/skills --output-dir benchmark/results -v
- ```
+ ### SkillsBench (BenchFlow-aligned Docker evaluation)
- ### GAIA (requires HF auth — currently broken)
- ```bash
- huggingface-cli login # must do this first
- python benchmark/run.py --benchmark gaia --mode both --model glm-5.1 ...
- ```
+ Both modes are **100% SkillsBench aligned**: the agent runs inside the
+ container via BenchFlow ACP. The only difference is how skills are delivered.
- ### SkillsBench (Docker-based deterministic evaluation)
```bash
- # Prerequisites: clone the SkillsBench repo and have podman/docker available
+ # Prerequisites: clone the SkillsBench repo
git clone --depth 1 https://github.com/benchflow-ai/skillsbench /tmp/skillsbench_full
- ANTHROPIC_API_KEY="$ANTHROPIC_AUTH_TOKEN" \
- python benchmark/run.py --benchmark skillsbench --mode both --max-tasks 10 --model glm-5.1 \
- --skills-dir .agents/skills --output-dir benchmark/results \
- --skillsbench-repo /tmp/skillsbench_full -v
+ # ACP mode — Traditional (SKILL.md files injected into Dockerfile)
+ python benchmark/run.py --benchmark skillsbench --mode acp --max-tasks 25 \
+ --model glm-5.1 --output-dir benchmark/results \
+ --skillsbench-repo /tmp/skillsbench_full -v --attempts 5
+
+ # ACP-MCP mode — OntoSkills (ontomcp inside container)
+ python benchmark/run.py --benchmark skillsbench --mode acp-mcp --max-tasks 25 \
+ --model glm-5.1 --output-dir benchmark/results \
+ --skillsbench-repo /tmp/skillsbench_full -v --attempts 5
+
+ # Both modes (comparison)
+ python benchmark/run.py --benchmark skillsbench --mode both --max-tasks 25 \
+ --model glm-5.1 --output-dir benchmark/results \
+ --skillsbench-repo /tmp/skillsbench_full -v --attempts 5
+
+ # No-skill-hints variants (test discovery mechanism)
+ python benchmark/run.py --benchmark skillsbench --mode acp --max-tasks 25 \
+ --no-skill-hints ...
```
- SkillsBench uses deterministic Docker evaluation:
- 1. Agent generates a Python solution script
- 2. Script is executed inside the task's Docker container (via podman)
- 3. Task's pytest test suite verifies the output files
- 4. Fractional scoring from CTRF report (passed/total tests) with binary reward.txt fallback
+ ### SkillsBench methodology
- 6 tasks are skipped: 5 exotic base images (bugswarm, suricata, oss-fuzz, erlang) + 1 Podman/BuildKit incompatibility (organize-messy-files).
+ Both modes use BenchFlow Trial for the complete lifecycle inside the
+ container, aligned with official
+ [BenchFlow](https://github.com/benchflow-ai/benchflow) /
+ [SkillsBench](https://github.com/benchflow-ai/skillsbench) evaluation:
- #### Agent design for SkillsBench
+ 1. **Container lifecycle**: BenchFlow `Trial` builds the Docker image and
+ starts the container.
+ 2. **Agent execution**: Agent runs inside the container via ACP.
+ 3. **Skill delivery**:
+ - `acp` (Traditional): SKILL.md files injected into Dockerfile
+ - `acp-mcp` (MCP): ontomcp binary + TTLs + `.mcp_config.json` injected
+ 4. **Verification**: Harbor Verifier runs `tests/test.sh` (pytest) inside the
+ container and reads CTRF report for fractional scoring.
+ 5. **Retries**: BenchFlow `RetryConfig` with exponential backoff, clean retries,
+ timeout exclusion.
- **Traditional agent**: SKILL.md content included directly in the user prompt (raw markdown).
- No tools — the agent sees all skill documentation in one shot and generates code.
+ **Comparison is fair**: both modes use identical container management (BenchFlow
+ Trial) and identical verification (Harbor Verifier). The only difference is
+ **how skills are delivered**.
- **OntoSkills agent**: Skills loaded via MCP prefetch from compiled TTLs. The user prompt
- contains only the task instruction + Dockerfile metadata (89% smaller). Skill knowledge
- (structured nodes with types, severity, context) is injected into the system prompt via
- `get_skill_context`. No tools needed — single-turn code generation.
+ 12 tasks are skipped: exotic base images, multi-container setups, and
+ BuildKit heredoc incompatibility with Podman.
- This tests OntoSkills' core advantage: **structured skill knowledge via MCP vs. raw SKILL.md**.
+ #### CLI flags
- The MCP server uses a SkillsBench-only ontology root (`/tmp/skillsbench_ontology/`) containing
- only the 218 SkillsBench TTLs (vs. 840 total). This reduces MCP startup from 10s to 1.8s and
- query time from 3.8s to 0.27s per skill.
+ - `--mode {acp,acp-mcp,baseline,both,all5}` — Agent mode (default: both)
+ - `all5`: task-first iteration — for each task runs all 5 cases, then prunes Docker
+ - `--attempts N` — Clean retries per task (default: 5, matches SkillsBench)
+ - `--workers N` — Parallel Docker workers (default: 2)
+ - `--resume` — Resume from previous state file (default: True)
+ - `--force-restart` — Ignore existing state and start fresh
+ - `--state-file PATH` — Custom state file path
+ - `--no-skill-hints` — Omit skill names from prompts (tests discovery mechanism)
+ - `--only-tasks id1,id2` — Run specific task IDs only
+ - `--skip-first N` — Skip first N tasks (combine with previous results)
- ## Known issues
+ #### Production-realistic benchmark (5 runs)
- ### SWE-bench wrapper: custom run-loop required
- The SWE-bench wrapper patches `agent.run_turn` to intercept file_read/file_edit. It does NOT use `BaseAgent.run()` — it has a custom loop because `BaseAgent.run()` double-appends messages when `run_turn` also appends. See `swebench.py:run_task()` for the custom loop.
+ | Run | Mode | Skills | Hints | Tests |
+ |-----|---------|----------|-------|------------------|
+ | 1 | acp | None | No | Baseline (agent only) |
+ | 2 | acp | SKILL.md | Yes | Knowledge quality |
+ | 3 | acp-mcp | ontomcp | Yes | Knowledge quality |
+ | 4 | acp | SKILL.md | No | Discovery |
+ | 5 | acp-mcp | ontomcp | No | Discovery |
- ### Content coverage: core.ttl must be loaded
- The knowledge yield Level 2 SPARQL uses `rdfs:subClassOf*` property paths to resolve leaf types (e.g., `oc:AntiPattern`) to top-level dimensions (e.g., `oc:NormativeRule`). This requires `core.ttl` loaded in the graph. The file is at `ontoskills/core.ttl` in the project root, NOT in `~/.ontoskills/packages/`.
+ Note: Baseline (Run 1) needs `skill_nudge=""` and no skills_dir passed. This
+ measures the raw agent without any skill delivery.
- ## Test verification
+ #### Incremental execution
- Run before any changes:
```bash
- cd /home/marcello/Documenti/onto/ontoskills/core
- python -m pytest tests/ -q
+ # Start with 15 tasks
+ python benchmark/run.py --benchmark skillsbench --mode acp --max-tasks 15 \
+ --model glm-5.1 --output-dir benchmark/results \
+ --skillsbench-repo /tmp/skillsbench_full -v --attempts 5
+
+ # Later, extend to 25 (resumes from saved state):
+ python benchmark/run.py --benchmark skillsbench --mode acp --max-tasks 25 \
+ --model glm-5.1 --output-dir benchmark/results \
+ --skillsbench-repo /tmp/skillsbench_full -v --attempts 5 --resume
```
- Run benchmark tests:
+ ### Content coverage (instant, no API)
```bash
- cd /home/marcello/Documenti/onto/ontoskills
- python -m pytest benchmark/tests/ -q
+ ANTHROPIC_API_KEY="$ANTHROPIC_AUTH_TOKEN" \
+ python benchmark/content_coverage.py --verbose --ttl-dir ~/.ontoskills/packages \
+ --json benchmark/results/content_coverage.json
```
- Run smoke test after compilation:
+ ### SWE-bench (requires API)
```bash
- python benchmark/smoketest.py
+ ANTHROPIC_API_KEY="$ANTHROPIC_AUTH_TOKEN" \
+ python benchmark/run.py --benchmark swebench --mode both --max-tasks 25 --model glm-5.1 \
+ --skills-dir .agents/skills --output-dir benchmark/results -v
```
- ## Prefetch optimization
-
- OntoSkillsAgent supports `prefetch=True` mode:
- 1. Before the first API call, calls MCP `search` + `get_skill_context`
- 2. Compacts the verbose MCP JSON into lean markdown-like text
- 3. Injects into system prompt — model has knowledge from turn 1
- 4. Removes tool schemas when knowledge is pre-loaded (no tool calls needed)
-
- Result: OntoSkills at **0.86x tokens** vs Traditional (input is 0.40x), with better quality (4.6/5 vs 3.7/5).
-
- ## MCP response compaction
-
- All MCP tool responses use compact format by default (88% token reduction):
-
- - **search**: 90% reduction — keeps only skill_id, intents, trust_tier
- - **get_skill_context**: 79% reduction — formatted as markdown with knowledge nodes sorted by priority
- - **evaluate_execution_plan**: 96% reduction — plan steps + warnings only
- - **query_epistemic_rules**: 79% reduction — directive content with context and severity
-
- Compaction happens server-side in the Rust MCP server (`mcp/src/compact.rs`). The `content[0].text`
- field contains compact markdown text. Full JSON is preserved in `structuredContent` (zero knowledge
- loss). Use `format: "raw"` parameter to get verbose JSON instead.
-
- The Python agent (`ontoskills.py`) also has compactors as a secondary layer for API-based agents.
-
- Tools accept a `format` parameter: `"compact"` (default) or `"raw"`.
-
- ## Traditional agent design
-
- The TraditionalAgent works like Claude Code:
- - System prompt contains a **skill registry** with all skill names + descriptions (~27K tokens)
- - Model has a `read_skill` tool to load full SKILL.md content on demand
- - Multi-turn loop: model reads relevant skills then answers
- - Both GAIA and SWE-bench wrappers delegate `read_skill` to `agent._resolve_skill()`
-
- ## ClaudeCodeAgent
-
- Uses the Claude Code CLI in `--print --bare --output-format json` mode for realistic evaluation.
-
- Two modes:
- - `traditional` — SKILL.md files in `.claude/skills/`, no MCP
- - `ontoskills` — MCP config for ontomcp + ontomcp-driver SKILL.md in `.claude/skills/`
-
- Key files: `benchmark/agents/claudecode.py`, `benchmark/agents/utils.py`
-
- ## Current benchmark results (2026-04-28, SkillsBench 25-task)
+ ## Known issues
- _25-task benchmark in progress with compact MCP + test-first prompting + task.toml timeouts._
+ ### SWE-bench wrapper: custom run-loop required
+ The SWE-bench wrapper patches `agent.run_turn` to intercept file_read/file_edit. It does NOT use `BaseAgent.run()` — it has a custom loop.
- ### SkillsBench (Claude Code CLI, seed=7, glm-5.1, 25 tasks)
+ ### Content coverage: core.ttl must be loaded
+ The knowledge yield Level 2 SPARQL uses `rdfs:subClassOf*` property paths. Requires `core.ttl` at `ontoskills/core.ttl`.
- Previous 10-task results (compact MCP, before test-first prompting):
- - Traditional: 4/10 (40%), avg_reward=0.40
- - OntoSkills MCP: 5/10 (50%), avg_reward=0.52 (+25% pass rate, +30% avg reward)
+ ## Test verification
- Key improvements since 10-task run:
- - Compact responses in Rust MCP server (88% token reduction)
- - `prefetch_knowledge` tool (search + context in one call)
- - Test-first prompting (test spec injected into prompt)
- - task.toml timeouts respected (was hardcoded 900s)
- - Docker image pre-build (Phase 0)
+ ```bash
+ python -m pytest benchmark/tests/ -q
+ ```
- ### GAIA
- _Results pending — run with Claude Code mode._
+ ## ClaudeCodeAgent (legacy)
- ### SWE-bench
- _Results pending — run with Claude Code mode._
+ Legacy host-based agent using the Claude Code CLI in `--print --bare --output-format json` mode.
+ No longer used by SkillsBench (replaced by ACP mode), kept for backward compatibility.
- ### Compiler bug (unrelated to benchmark)
- 10 skills across 10 tasks failed to compile to TTL. Root cause: `ontocore compile`
- with `-o` flag resolves `state_dir` incorrectly, creating `/state` instead of
- relative path. The 10 missing skills are:
- civ6lib, map-optimization-strategy, sqlite-map-parser, pymatgen, lean4-memories,
- gemini-video-understanding, senior-data-scientist, gmail-skill, threejs,
- data-reconciliation.
+ Only supports `ontoskills` mode (MCP config for ontomcp). The `traditional` mode has been
+ removed — use `--mode acp` instead.
- ## Chart data output
+ ## MCP response compaction
- All benchmark runs produce `chart_data.json` alongside `results.json` and `score.json`. This file contains per-task metrics in a format suitable for Chart.js/D3 visualization on the OntoSkills website.
+ All MCP tool responses use compact format by default (88% token reduction).
+ Compaction happens server-side in Rust (`mcp/src/compact.rs`).
+ `NODE_BUDGET_CHARS` (12000) limits knowledge node content in compact output to
+ ~3K tokens, preventing overwhelming the agent with low-priority nodes.