holdout-evaluator SKILL.md skillA
holdout-evaluator from synaptiai/synapti-marketplace is an agent skill: a SKILL.md folder an AI agent loads when a task calls for it.
mdr add synaptiai/synapti-marketplace/holdout-evaluator@git:20260401.9ecedacPins git:20260401.9ecedac in mdr.lock. Pin the fingerprint instead.
- Audit grade
- A, 17 of 17 checks passed
- Kind
- SKILL.md skill
- Repository
- synaptiai/synapti-marketplace
- Path
plugins/ai-first-org-design-kit/skills/holdout-evaluator/SKILL.md- Versions recorded
- 1
- Latest version
- git:20260401.9ecedac, committed Apr 1, 2026
- Size
- 15,003 bytes
- Repository stars
- 7
- License
- Apache-2.0
Validate agent work output against hidden holdout scenarios using LLM-as-Judge evaluation, producing mapped feedback (referencing visible criteria only) and telemetry records saved to $HOME/.ai-first-kit/. Cross-references the agent's self-review evidence table against actual files to detect claims without evidence. Use when the user says 'validate holdouts', 'test gates against holdouts', 'run holdout evaluation', 'check gate effectiveness', or when invoked as a sub-agent by org-gate-review…
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
# Holdout Evaluator You are a **Quality Gate Judge** — you evaluate agent work output against hidden holdout scenarios that the executing agent never sees. Your core insight: visible gate criteria tell agents WHAT to check, but holdout scenarios test WHETHER they genuinely understand the criteria or are just checking boxes. You operate as an independent evaluator, never revealing holdout scenario content to the executing agent. Your output has two layers: a detailed layer for telemetry (which scenarios passed/failed) and a mapped layer for the agent (which visible criteria are weak, without naming scenarios). Read `../../shared/concepts.md` for the Artifact Handoff Convention and Governance Health Metrics. Work through these steps in order, announcing each step as you begin it: <required> 0. Pre-flight (artifact discovery, input validation) 1. Load gate criteria and holdout scenarios 2. Read work output and self-review evidence 3. LLM-as-Judge evaluation per scenario 4. Generate mapped feedback 5. Write telemetry record 6. Return results </required> ## Persona - **Skeptical.** Claims without evidence are failures. "I verified X" without proof is the same as not verifying. …
Read the whole file at its exact version.
How to install
mdr add synaptiai/synapti-marketplace/holdout-evaluator@sha256:d4a0528b138cdac3The line at the top of this page pins the label git:20260401.9ecedac, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
[](https://markdownregistry.com/a/art_m5lap3amrd7ki3fw)
3 badge image fetches in 30 days. What the badge shows.
Versions
The registry has recorded 1 version of this file, committed on Apr 1, 2026.
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (15003 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
synaptiai/synapti-marketplace · 7 stars · license Apache-2.0 · pushed 2026-11-06 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_m5lap3amrd7ki3fw GET https://markdownregistry.com/api/v1/resolve?ref=synaptiai/synapti-marketplace/holdout-evaluator GET https://markdownregistry.com/api/v1/blob/d4a0528b138cdac3efbd24403e1915cc7a9208401c3fa8b8a290b6daf211d1c1
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
Guides: the SKILL.md format and its required fields · what to check before you install a skill · how to pin a skill to an exact version · where to find agent skills · keeping a team on the same skills · which agent file does what.