ai-evals SKILL.md skillA
ai-evals from vasilyu1983/ai-agents-public is an agent skill: a SKILL.md folder an AI agent loads when a task calls for it.
mdr add vasilyu1983/ai-agents-public/ai-evals@v1.8Pins v1.8 in mdr.lock. Pin the fingerprint instead.
- Audit grade
- A, 17 of 17 checks passed
- Kind
- SKILL.md skill
- Repository
- vasilyu1983/ai-agents-public
- Path
skills/universal/ai-evals/SKILL.md- Versions recorded
- 2
- Latest version
- v1.8, committed Oct 12, 2026
- First version recorded
- Oct 5, 2026
- Size
- 19,149 bytes
- Repository stars
- 101
- License
- MIT
Designs LLM, agent, responsible-AI, and multimodal evals. Use when measuring quality, fairness, grounding, safety, or judge reliability. Not for test harnesses (qa-agent-testing).
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
# AI Evaluation and Fine-Tuning Methodology Skill Use this skill to calibrate graders, derive release gates, and compare evaluation or adaptation methods. Domain skills own what to measure; this skill owns the measurement controls. Training loss is telemetry; held-out behavior determines whether an adaptation improved the system. ## Quick Reference | Task | Read or Run | Outcome | |------|-------------|---------| | Build a (question, ideal-answer) set and tune it | `references/dataset-construction.md` | Sourcing, ideal-answer authoring, run→compare→tune loop, robustness slice by perturbing memorized cases | | Compose an eval dataset | `references/eval-dataset-design.md` | Task distribution, difficulty, sampling, annotation, versioning, contamination, bias audits, golden vs dynamic sets | | Stop a judge from rating its own output high | `references/llm-judge-bias.md` | Self-preference, position, length, verbosity controls | | Pick / wire an eval framework | `references/framework-integration.md` | inspect-ai, lighteval, Ragas, DeepEval, promptfoo, Braintrust integration snippets + when to use each | …
Read the whole file at its exact version.
How to install
mdr add vasilyu1983/ai-agents-public/ai-evals@sha256:917c04a4a3a52918The line at the top of this page pins the label v1.8, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
[](https://markdownregistry.com/a/art_xw3q4sx4qdgc2aif)
5 badge image fetches in 30 days. What the badge shows.
Versions
The registry has recorded 2 versions of this file, between Oct 5, 2026 and Oct 12, 2026, about one every 7 days.
| version | committed | commit | size | audit | |
|---|---|---|---|---|---|
| v1.8 latest | 2026-10-12 | 34b57f4 | 19,149 B | A | view · diff |
| v1.8 | 2026-10-05 | 5db538c | 19,133 B | A | view |
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (19149 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
vasilyu1983/ai-agents-public · 101 stars · license MIT · pushed 2026-10-16 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_xw3q4sx4qdgc2aif GET https://markdownregistry.com/api/v1/resolve?ref=vasilyu1983/ai-agents-public/ai-evals GET https://markdownregistry.com/api/v1/blob/917c04a4a3a5291823da4e5fa0cdef1fb42fc00a859588f80e7d8af86f2332f3
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
Guides: the SKILL.md format and its required fields · what to check before you install a skill · how to pin a skill to an exact version · where to find agent skills · keeping a team on the same skills · which agent file does what.
More from vasilyu1983/ai-agents-public
Every file in vasilyu1983/ai-agents-public