model-bench git:20260721.97dc2a3A

Immutable. This exact content never changes and is served at /api/v1/blob/2fd060ec596edc6a.

---
name: model-bench
description: Run a 3-leg evaluation against a newly released LLM. Leg 1 is trap questions with deterministic answers, leg 2 is an agentic orchestration task with a planted failure, leg 3 is a brownfield feature build in a real repo. Use when the user wants to vibe-check, test, evaluate, or compare a new frontier model, or mentions model-bench, model vibe check, or launch-day protocol.
disable-model-invocation: true
---

# model-bench

3 legs, each aimed at a different failure mode:

1. **Trap questions**: does it think, or pattern-match to training data? Deterministic answers, grades in seconds.
2. **Orchestration**: can it plan, delegate, and recover when a step fails (including a planted trap)?
3. **Brownfield build**: can it ship a feature into an existing codebase without breaking things 2 files away?

Each leg scores 0-5. Total out of 15.

## Contamination rules (read first)

- The model under test must NEVER see the contents of this skill folder. If it reads the answer key or learns about the planted trap, that leg is void.
- This skill runs in a **proctor session** (Nate plus a model he already trusts) opened on this repo. The model under test runs in separate, fresh sessions and only ever receives the paste-ready prompts.
- **Never use this repo as the target repo.** The skill (and answer key) live here; target-repo sessions must not be able to see them.
- One fresh session per leg. No custom system prompts or rules that would coach the model.

## Workflow

```
Progress:
- [ ] Setup: pick target model, pick target repo, verify leg 2 plant
- [ ] Leg 1: 5 questions in fresh chats (or subagent fast path), record final answers
- [ ] Leg 2: paste orchestration task into a fresh agent session on the target repo
- [ ] Leg 3: paste brownfield spec into a fresh agent session, branch per model
- [ ] Grade all 3 legs using grading/answer-key.md
- [ ] Fill scorecard, save to results/, write 2-3 sentence verdict per model
```

**Setup**
- Target repo for legs 2-3: a real repo with existing conventions, ideally one Nate knows well enough to judge output quality. Never this repo.
- Verify the leg 2 planted path (default `docs/METRICS.md`) does not exist in the target repo. If it does, pick another plausible-but-missing path and note it.
- When comparing models, use identical prompts, identical repo state, and a fresh branch per model (`vibe/<model-name>`).

**Run the legs**
- Leg 1 prompts: [tasks/leg-1.md](tasks/leg-1.md)
- Leg 2 task: [tasks/leg-2.md](tasks/leg-2.md)
- Leg 3 spec template: [tasks/leg-3.md](tasks/leg-3.md)

**Leg 1 subagent fast path (optional)**
If the target model is available as a subagent model slug, the proctor session can run leg 1 itself: launch 5 parallel subagents, each with the target model, each receiving exactly one question verbatim and nothing else, and record each final answer. Do not use subagents for legs 2-3 — they run in the current workspace, which is this repo, and the isolation isn't clean. If the target model has no subagent slug (common on launch day), run leg 1 manually in fresh chats.

**Grade**
- Rubrics and answer key: [grading/answer-key.md](grading/answer-key.md)
- Only open the answer key after all legs have been run, and never in a session where the target model is active.

## Results

Save each filled scorecard to `results/YYYY-MM-DD.md` (append `-2`, `-3` for multiple runs on one day). Use this template:

```markdown
# Vibe check: [date]

Target repo: [repo]
Leg 2 plant: [path used]
Leg 3 spec: [feature, one line] / planted ambiguity: [what it was]

| Leg | [Model A] | [Model B] | [Model C] |
|---|---|---|---|
| 1. Trap questions (/5) | | | |
| 2. Orchestration (/5) | | | |
| 3. Brownfield (/5) | | | |
| **Total (/15)** | | | |

Verdict per model: which legs it won, what surprised me, does it earn a spot in the rotation.
```

## Notes

- Public trap questions decay. When a question gets 3 passes in a row across model families, retire it and add a fresh one to the bank.
- Different models winning different legs is the expected (and most interesting) result. Record it, it's post material.