ai-evaluation-engineering SKILL.md skillA
ai-evaluation-engineering from devcodex-labs/devcodex is an agent skill: a SKILL.md folder an AI agent loads when a task calls for it.
mdr add devcodex-labs/devcodex/ai-evaluation-engineering@git:20260730.6ee304aPins git:20260730.6ee304a in mdr.lock. Pin the fingerprint instead.
- Audit grade
- A, 17 of 17 checks passed
- Kind
- SKILL.md skill
- Repository
- devcodex-labs/devcodex
- Path
content/skills/ai-evaluation-engineering/SKILL.md- Versions recorded
- 1
- Latest version
- git:20260730.6ee304a, committed Jul 30, 2026
- Size
- 3,106 bytes
- Repository stars
- 432
- License
- AGPL-3.0
AI 评测工程专家 Owner — 当任务涉及模型/Prompt 评测、模型选择、黄金集、评分量表、LLM-as-judge、Judge 校准、重复采样、方差、质量-成本-延迟权衡、提示词回归或模型升级回归时使用；要求把概率性结果转化为可复现、可比较且防污染的评测证据。.
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
# AI Evaluation Engineering ## 职责 设计概率性 AI 系统的评测数据、指标、Judge、重复采样、方差和回归决策。AI Agent Skill 负责系统行为，quality-strategy 负责整体测试组合；本 Skill 负责模型/Prompt 质量证据。 ## AiEvaluationEngineeringGate | 字段 | 要求 | |---|---| | evaluationDatasetManifest | 来源、版本、许可/隐私、任务分层、难例、污染风险和 split | | goldenCaseSet | 输入、期望属性/答案、允许变体、失败标签和维护 owner | | metricRubric | deterministic/semantic/human 指标、权重、阈值和不可聚合项 | | judgeCalibration | Judge 模型/Prompt/版本、盲测、与人工一致性、偏差和漂移 | | samplingProtocol | temperature/seed、重复次数、置信区间、停止规则和失败重试 | | varianceReport | 均值、分布、尾部失败、跨 run/provider 差异和不确定性 | | costLatencyQualityFrontier | token/费用/延迟/成功率/质量的 Pareto 权衡 | | regressionDecision | baseline/candidate、显著性、阻断阈值、例外和 rollback | ## 执行流程 1. 冻结 use case、风险等级、失败类型和评测决策用途。 2. 建立 train/dev/test 或等价隔离，检查 benchmark、Prompt 和检索语料污染。 3. 将确定性断言、语义评分、人工评审和业务 outcome 分层，禁止只用单一总分。 4. 校准 Judge：随机顺序、隐藏候选身份、人工样本对照、位置/长度/风格偏差。 5. 对概率性路径重复采样，报告方差和尾部失败，不用单次成功代表稳定。 6. 同时测质量、成本、延迟和工具/结构化输出正确性。 7. 用固定版本 manifest 对比 baseline/candidate，达到阈值才进入发布或模型切换。 ## 输出字段 …
Read the whole file at its exact version.
How to install
mdr add devcodex-labs/devcodex/ai-evaluation-engineering@sha256:30d8c4287a14a2b2The line at the top of this page pins the label git:20260730.6ee304a, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
[](https://markdownregistry.com/a/art_uhw5hbav2hqj4kjx)
3 badge image fetches in 30 days. What the badge shows.
Versions
The registry has recorded 1 version of this file, committed on Jul 30, 2026.
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (3106 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
devcodex-labs/devcodex · 432 stars · license AGPL-3.0 · pushed 2026-10-27 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_uhw5hbav2hqj4kjx GET https://markdownregistry.com/api/v1/resolve?ref=devcodex-labs/devcodex/ai-evaluation-engineering GET https://markdownregistry.com/api/v1/blob/30d8c4287a14a2b21d353bda05a2bad6394e10c1ece2c1fa8777cd5bbef30f15
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
Guides: the SKILL.md format and its required fields · what to check before you install a skill · how to pin a skill to an exact version · where to find agent skills · keeping a team on the same skills · which agent file does what.