align-human SKILL.md skillA
align-human from agentscope-ai/openjudge is an agent skill: a SKILL.md folder an AI agent loads when a task calls for it.
mdr add agentscope-ai/openjudge/align-human@git:20260708.2151defPins git:20260708.2151def in mdr.lock. Pin the fingerprint instead.
- Audit grade
- A, 17 of 17 checks passed
- Kind
- SKILL.md skill
- Repository
- agentscope-ai/openjudge
- Path
skills/eval_pipeline/03-align-human/SKILL.md- Versions recorded
- 1
- Latest version
- git:20260708.2151def, committed Jul 8, 2026
- Size
- 12,319 bytes
- Repository stars
- 1,020
- License
- Apache-2.0
Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
<HARD-GATE> NO calibrated:true WITHOUT TPR >= 0.8 AND TNR >= 0.8 AND boundary stratum TPR >= 0.6 AND TNR >= 0.6 AND n_dev >= 10 per class AND test-set drop < 10%. NO human_reduction_phase >= 2 WITHOUT kappa >= 0.6 AND boundary kappa >= 0.6. NO alignment conclusion WITHOUT all 5 bias checks completed. </HARD-GATE> # Align Human Measure whether your automatic judge agrees with human judgment, detect where and why they disagree, and build a roadmap to reduce human review over time. ## When to Activate - You have a working judge/grader and 50+ human-labeled examples - You want to know if the judge is trustworthy enough to replace human review - You've noticed the judge's decisions being overturned by humans - You're preparing to deploy an evaluation as a production gate ## Checklist You MUST create a task for each item and complete them in order: 1. **Load paired data** — match judge verdicts with human labels 2. **Measure TPR/TNR** — confusion matrix + per-stratum breakdown 3. **Calculate agreement** — Cohen's kappa, Gwet's AC1, systematic bias 4. **Run bias detection** — 5 systematic bias checks 5. **Analyze disagreements** — cluster patterns + diagnose root causes …
Read the whole file at its exact version.
How to install
mdr add agentscope-ai/openjudge/align-human@sha256:62d9a6ebc171f0f1The line at the top of this page pins the label git:20260708.2151def, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
[](https://markdownregistry.com/a/art_oac4khfmneau3zcn)
3 badge image fetches in 30 days. What the badge shows.
Versions
The registry has recorded 1 version of this file, committed on Jul 8, 2026.
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (12319 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
agentscope-ai/openjudge · 1,020 stars · license Apache-2.0 · pushed 2026-10-30 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_oac4khfmneau3zcn GET https://markdownregistry.com/api/v1/resolve?ref=agentscope-ai/openjudge/align-human GET https://markdownregistry.com/api/v1/blob/62d9a6ebc171f0f135ab26664f7f36c568c1eaa5545d1031f3e73674fefba918
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
Guides: the SKILL.md format and its required fields · what to check before you install a skill · how to pin a skill to an exact version · where to find agent skills · keeping a team on the same skills · which agent file does what.