karpathy-graph-evaluate skillA
karpathy-graph-evaluate is agent-read markdown (skill) from andersonlimahw/lemon-ai-hub: Stage 5 of the karpathy-graph pipeline. Score graph quality across extraction, resolution, structure, query, and operations, and run a ratchet loop that keeps only changes which improve the metric. Use when the user asks how good the graph is, wants to tune extraction prompts or the ontology, asks to measure or improve graph quality, or wants to detect resolution regressions..
Indexed from public GitHub and served as immutable, content-addressed versions. Install it pinned to an exact SHA-256 with the mdr CLI, and every file is verified against the hash recorded here before it reaches your agent. The deterministic audit below grades the latest version, and the same file always earns the same grade.
What the file says
# Evaluate The evaluation harness has the same shape as Karpathy's autoresearch loop. The artifact being optimized is not `train.py` — it is the extraction prompt, the ontology, the resolution policy, or the query serializer. ``` Read current extraction prompt and score history -> Propose ONE motivated change -> Run extraction on the gold set -> Compute precision, recall, F1, cost, latency -> If improved: keep. If worse: revert. -> Record and continue. ``` One change at a time. A batch of five changes that improves the metric teaches you nothing about which of the five worked. ## Build a gold set first Without one you are guessing. Take 10–30 representative units of context and hand-label the entities and relations that should be extracted. Include adversarial cases deliberately: - misleading aliases (two different things with near-identical names) - contradictory dates across sources - entities that *should not* merge despite high string similarity - claims with no supporting source - disconnected paths that look connected A gold set of only easy cases certifies a system that fails on the hard ones. ## Metrics by layer …
Read the whole file at its exact version.
How to install
mdr add andersonlimahw/lemon-ai-hub/karpathy-graph-evaluate@v1.0.0mdr add andersonlimahw/lemon-ai-hub/karpathy-graph-evaluate@sha256:52976a80ad448a27Pin to a label to follow the author's releases, or to a sha256 to freeze the exact bytes forever. Either way the resolved hash is written to mdr.lock, and mdr install reproduces it on any machine.
[](https://markdownregistry.com/a/art_kxbaiemhksulspzr)
1 badge views in 30 days
Versions
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (5437 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
andersonlimahw/lemon-ai-hub · 14 stars · license none · pushed 2026-09-20 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_kxbaiemhksulspzr GET https://markdownregistry.com/api/v1/resolve?ref=andersonlimahw/lemon-ai-hub/karpathy-graph-evaluate GET https://markdownregistry.com/api/v1/blob/52976a80ad448a2767a83fccae9fffadc408d7b2ca8cd7fb2c34df9119b5970c
Your agent does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.