compare-llm-models SKILL.md skillA
compare-llm-models from contextjet-ai/awesome-llm-observability is an agent skill: a SKILL.md folder an AI agent loads when a task calls for it.
mdr add contextjet-ai/awesome-llm-observability/compare-llm-models@git:20260705.4a58130Pins git:20260705.4a58130 in mdr.lock. Pin the fingerprint instead.
- Audit grade
- A, 17 of 17 checks passed
- Kind
- SKILL.md skill
- Repository
- contextjet-ai/awesome-llm-observability
- Path
skills/compare-llm-models/SKILL.md- Versions recorded
- 1
- Latest version
- git:20260705.4a58130, committed Jul 5, 2026
- Size
- 3,351 bytes
- Repository stars
- 45
- License
- a license file GitHub does not identify
Use this to pick or switch the LLM behind a feature, based on evidence instead of hype or the newest release. Trigger on "which model should I use", "is GPT/Claude/Gemini/Llama better for this", "should I switch models", "can a cheaper model do this", "compare models for my use case". Evaluate on YOUR task, not on leaderboards alone.
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
# Compare LLM models for your task The best model on a public leaderboard is often not the best model for *your* task at *your* cost and latency. Public benchmarks narrow the field; your own eval set makes the call. ## Use benchmarks to shortlist, not to decide - **General leaderboards** (Chatbot Arena / LMArena for human preference, HELM for multi-metric, MMLU/GPQA for reasoning) tell you the rough tier a model is in. Use them to pick 2-4 candidates, not to declare a winner for your app. - **Watch for contamination and overfitting** to popular benchmarks. A high MMLU score does not mean the model is good at your specific extraction/RAG/agent task. - **Task-relevant benchmarks** beat general ones: if you do code, look at code evals; if RAG, look at long-context/faithfulness; if tools, look at agent/tool-use benchmarks. ## Then evaluate the shortlist on YOUR eval set This is the part that actually decides it. Run each candidate model through your own eval suite (see `build-eval-dataset` and `add-llm-evals`) and compare on the axes that matter: | Axis | How to measure | |---|---| | **Quality** | Your eval scores on your dataset (not a leaderboard) | …
Read the whole file at its exact version.
How to install
mdr add contextjet-ai/awesome-llm-observability/compare-llm-models@sha256:9210054bf1f1e0e6The line at the top of this page pins the label git:20260705.4a58130, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
[](https://markdownregistry.com/a/art_o7fxw5ygjjtylhmg)
1 badge image fetches in 30 days. What the badge shows.
Versions
The registry has recorded 1 version of this file, committed on Jul 5, 2026.
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (3351 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
contextjet-ai/awesome-llm-observability · 45 stars · license a license file GitHub does not identify · pushed 2026-11-02 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_o7fxw5ygjjtylhmg GET https://markdownregistry.com/api/v1/resolve?ref=contextjet-ai/awesome-llm-observability/compare-llm-models GET https://markdownregistry.com/api/v1/blob/9210054bf1f1e0e696ab1e391074a500b9d86c8ea7dfb502c9d242aea58b7318
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
Guides: the SKILL.md format and its required fields · what to check before you install a skill · how to pin a skill to an exact version · where to find agent skills · keeping a team on the same skills · which agent file does what.