01-auto-arena SKILL.md skillA
01-auto-arena from agentscope-ai/openjudge is an agent skill: a SKILL.md folder an AI agent loads when a task calls for it.
mdr add agentscope-ai/openjudge/01-auto-arena@git:20260911.d1e0642Pins git:20260911.d1e0642 in mdr.lock. Pin the fingerprint instead.
- Audit grade
- A, 17 of 17 checks passed
- Kind
- SKILL.md skill
- Repository
- agentscope-ai/openjudge
- Path
skills/arena-eval/01-auto-arena/SKILL.md- Versions recorded
- 1
- Latest version
- git:20260911.d1e0642, committed Sep 11, 2026
- Size
- 9,965 bytes
- Repository stars
- 1,020
- License
- Apache-2.0
Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a…
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
# Auto Arena Skill End-to-end automated model comparison using the OpenJudge `AutoArenaPipeline`: 1. **Generate queries** — LLM creates diverse test queries from task description 2. **Collect responses** — query all target endpoints concurrently 3. **Generate rubrics** — LLM produces evaluation criteria from task + sample queries 4. **Pairwise evaluation** — judge model compares every model pair (with position-bias swap) 5. **Analyze & rank** — compute win rates, win matrix, and rankings 6. **Report & charts** — Markdown report + win-rate bar chart + optional matrix heatmap ## Prerequisites ```bash # Install OpenJudge pip install py-openjudge # Extra dependency for auto_arena (chart generation) pip install matplotlib ``` ## Gather from user before running | Info | Required? | Notes | |------|-----------|-------| | Task description | Yes | What the models/agents should do (set in config YAML) | | Target endpoints | Yes | At least 2 OpenAI-compatible endpoints to compare | | Judge endpoint | Yes | Strong model for pairwise evaluation (e.g. `gpt-4`, `qwen-max`) | | API keys | Yes | Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc. | …
Read the whole file at its exact version.
How to install
mdr add agentscope-ai/openjudge/01-auto-arena@sha256:7b21c707eb70d878The line at the top of this page pins the label git:20260911.d1e0642, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
[](https://markdownregistry.com/a/art_qt7ngf7ke4hqodfd)
4 badge image fetches in 30 days. What the badge shows.
Versions
The registry has recorded 1 version of this file, committed on Sep 11, 2026.
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (9965 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
agentscope-ai/openjudge · 1,020 stars · license Apache-2.0 · pushed 2026-10-30 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_qt7ngf7ke4hqodfd GET https://markdownregistry.com/api/v1/resolve?ref=agentscope-ai/openjudge/01-auto-arena GET https://markdownregistry.com/api/v1/blob/7b21c707eb70d8784c943ee4b03108042a89a13e8b3a5219b2421ef73f443b0e
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
Guides: the SKILL.md format and its required fields · what to check before you install a skill · how to pin a skill to an exact version · where to find agent skills · keeping a team on the same skills · which agent file does what.