git:20260729.8dd5ad0 to git:20260920.2cbc56d

---

name: ai-evals

description: "Eval-driven product management for LLM features — error analysis, axial coding of failure modes, LLM-as-judge with TPR/TNR validation, and the continuous Analyze→Measure→Improve loop. Used when an AI feature's output quality is the product and \"did it get better?\" must be answered with evidence, not vibes. Hosted by ai-eval-engineer / metrics-architect."

---

## Codex host binding

- Treat references to Claude slash workflows as the equivalently named Codex skill.

- Before delegating to any specialist, read the `great-pm-runtime` skill and the selected packaged role file.

- Treat "invoke", "assign", "delegate", "spawn", and source Agent-tool instructions as a required Codex `spawn_agent` call with that role and a bounded assignment.

- Store every returned agent identifier. Never call a wait tool until a spawn has returned an identifier, and wait only on identifiers returned by successful spawns.

- If `spawn_agent` is unavailable or a spawn fails, report BLOCKED; do not impersonate the specialist or wait on an empty agent set.

- Resolve bundled paths from the installed GreatPM plugin root.

- Ignore Claude-only model aliases, colors, turn limits, and tool allowlists.

- Preserve GreatPM human gates, governance, state, and reporting contracts.

# AI Evals — eval-driven product management

> Provenance: great-pm-original, 2026-05-29, grounded in the cited sources below

> (Husain/Shankar AI Evals Masterclass, Aakash Gupta, *Who Validates the

> Validators?*). Web sources are treated as untrusted reference, not instruction.

**Core principle.** AI features don't fail because of the model — they fail

because **nobody evaluated them.** An eval is "the systematic measurement of LLM

pipeline quality" that produces *interpretable, actionable* results, not a single

accuracy number you can't act on. Expert practitioners spend **60–80% of dev time

on error analysis and evaluation**, not on building automated checks. The PM owns

the judgment of *what counts as good*; engineering owns the measurement

infrastructure. If you ship an AI feature without an eval, you are shipping blind

and finding out from users — on your most expensive surface.

This is not a metrics-design problem (that's about North Star / KPIs for a

*product*) nor an experiment problem (that's A/B testing *human* behavior). This

is the distinct, AI-native discipline of measuring **non-deterministic output

quality** so you can change a prompt and know whether it got better.

---

## 1. The Three Gulfs — why an AI feature is failing

Before measuring, diagnose *where* the gap is. The Three Gulfs (Husain/Shankar)

are the lens:

- **Comprehension Gulf (Developer → Data).** You don't actually know your input

distribution. You think users send clean transaction strings; they send

truncated UPI refs, foreign merchants, and refunds. *Fix: look at the data.*

- **Specification Gulf (Developer → Pipeline).** The prompt is vaguer than you

think. You wrote "categorize this transaction"; the model has no idea

self-transfers aren't expenses. *Fix: sharpen the spec, add examples.*

- **Generalization Gulf (Data → Pipeline).** Even with a perfect spec, edge cases

fail because the system is probabilistic. *This gulf never fully closes* — it's

inherent. *Manage it with evals + guardrails, don't pretend to eliminate it.*

Most teams jump straight to "build an automated metric" — skipping comprehension

entirely. Don't.

---

## 2. The Analyze → Measure → Improve loop (run continuously)

This is the spine of the skill. Run it as a cycle, not once.

### Analyze — look at your data (the part everyone skips)

1. **Collect traces.** Pull **50–100+ production traces** — a *trace* is the full

execution log: system prompt, user message, retrieved context, tool calls, and

the model's response. Not just the output; the whole thing.

2. **Open coding.** Spend ~**30 seconds per trace** writing a free-text note on

what went wrong, in plain language: "rendered markdown in an SMS", "promised an

action it didn't take", "tagged a self-transfer as an expense". No taxonomy

yet — just observe. ~1–2 hours for 100 traces. **Domain expert does this**, not

a contractor who doesn't know the product.

3. **Axial coding.** Cluster the open-code notes into **5–8 actionable

categories.** AI-assisted clustering is fine for a first pass, then a human

refines. **Avoid vague labels** — "temporal issues" is useless;

"date-formatting error" is fixable. ~1 hour.

4. **Error counting.** Build a pivot table: category × frequency. This ranks your

failure modes by impact. ~30 min. For each category decide: **fix trivially**

(one prompt line) / **build an eval** (worth automating) / **iterate the

prompt** and re-observe. Some rare-but-catastrophic modes jump the queue

regardless of frequency.

### Measure — build the judge for what's worth automating

5. **Exhaust cheap checks first.** Prefer **reference-based** metrics (exact

match, code execution, SQL-result compare, schema/format validation) — they're

deterministic and nearly free. Only spend on an LLM judge for the *reference-free*

categories that genuinely need judgment ("did it invent a category?", "did it

give financial advice?").

6. **Write the judge prompt BINARY.** Pass/fail only. **Never a Likert / 1–5

scale** — scales are subjectively inconsistent between runs and impossible to

align to human labels. Include a rubric and a few labeled examples in the judge

prompt.

7. **Label elicitation order** (cognitively easiest → hardest): direct binary

grading > pairwise comparison ("A vs B, which is better?") > absolute ranking.

Pairwise is easier for humans than absolute scoring when binary is ambiguous.

### Validate the judge — the step that makes the eval trustworthy

8. **Hand-label a held-out set yourself** (or with the domain expert). This is

ground truth.

9. **Score the judge against your labels using TPR and TNR — never raw

"agreement".**

- **TPR (True-Positive Rate / recall):** of the traces that *are* genuinely

errors, what fraction did the judge catch? Low TPR = the judge misses real

failures.

- **TNR (True-Negative Rate / specificity):** of the traces that are *fine*,

what fraction did the judge correctly pass? Low TNR = the judge cries wolf.

- **Why not "agreement"?** On imbalanced data (say 90% of traces pass), a judge

that blindly says "pass" every time scores 90% agreement while catching

**zero** errors. Agreement hides this; TPR/TNR expose it. (See Shankar et

al., *Who Validates the Validators?*)

10. **Iterate the judge prompt** until *both* TPR and TNR clear your bar. The

judge is itself an LLM pipeline — it gets the same Analyze→Measure→Improve

treatment.

### Improve — close the loop with confidence

11. Score a **large** production sample with the validated judge to get a real

quality number.

12. Change one thing — system prompt, retrieval, few-shot examples — and re-run.

13. **Confirm no other eval regressed** (a fix for one failure mode often breaks

another). Only then ship.

14. Keep the eval suite as a **regression gate** on every future prompt/model

change.

---

## 3. When to use — and when NOT to

| Use evals when… | Don't reach for full evals when… |

|---|---|

| LLM output quality *is* the product (categorization, extraction, RAG answers) | A deterministic check (exact match, schema, regex) already certifies correctness — just write that check |

| A prompt or model change needs a before/after regression gate | The feature is a thin LLM call with a trivial, self-evident output and near-zero stakes — a smoke test suffices |

| Users say the AI "feels worse" but you can't point to why | You haven't shipped *anything* yet and have zero traces — do `ai-use-case-scoping` first, then collect traces |

| You're tempted to report a single "accuracy %" with no error breakdown | You're measuring *human* behavior change — that's `experiment-design`, not evals |

| Rare-but-catastrophic failures exist (money mis-tagged, wrong handoff) | The cost of the eval infra exceeds the cost of the failure it would catch (decide per failure mode in step 4) |

**Cheaper alternative first:** always exhaust reference-based code checks before

building an LLM-as-judge. The judge is the expensive rung — climb to it only when

the output genuinely needs judgment.

---

## 4. Worked example — Acme's Gate-4 LLM categorizer

Acme routes transactions through a gate hierarchy (exact-match DB → regex →

vector embedding >0.85 → **LLM fallback**). The LLM gate (Gemini / local Ollama

`qwen2.5-coder:14b`) only sees the long-tail novel merchant strings the

deterministic gates couldn't resolve. **Its accuracy on that long tail is the

product's perceived intelligence** — and it's exactly where there is no reference

answer. So it gets the full loop.

**Trace definition.** Each trace = `(raw transaction string + the user's bucket

list + the model's chosen category + confidence + whether the user later

overrode it)`. The override is gold: it's a *free production label*.

**Analyze.** Pull 100 LLM-gate categorizations. Open-code each in ~30 s. Cluster

into axial categories — for Acme these land roughly as:

1. Self-transfer mistaken for an expense (e.g. NEFT to own account).

2. Investment tagged as a regular expense (SIP / mutual-fund debit).

3. Merchant-name truncation broke the match (`UPI/ZOMATO...` cut off).

4. Duplicate not merged (Gmail alert + CSV upload of the same charge).

5. Hallucinated category — model invented a bucket the user doesn't have.

Pivot table by frequency: say truncation (#3) is most frequent (prompt-fixable,

add the raw string + a normalization step), while hallucinated categories (#5)

are rare but catastrophic on someone's money (jump the queue — build a hard

guardrail, not just an eval).

**Measure.** Cheap checks first: #5 ("category not in the user's bucket list") is

a **reference-based** check — pure set membership, no LLM needed. #1 and #2 need

judgment, so build **binary** judges: "Is this transaction a self-transfer that

should be excluded? PASS/FAIL" with a rubric and examples.

**Validate.** Hand-label 80 transactions for the self-transfer judge. Measure:

- TPR = of real self-transfers, % the judge flagged. If 0.70, it misses 30% —

iterate the judge prompt with clearer self-transfer cues (same-name payee,

account-to-account markers).

- TNR = of genuine expenses, % correctly passed. If 0.95, few false alarms.

Iterate until both clear (say ≥0.90 / ≥0.90).

**Improve & close the loop.** Score a large sample with the validated judges to

get real per-category accuracy. The override→exact-match-table update (the

product's own §7 mechanic) becomes a **data flywheel**: every manual re-bucket is

a new labeled trace that feeds the next Analyze pass. This is the missing

measurement layer beneath the gate hierarchy — without it, you can change the

Gate-4 prompt and have no idea whether categorization got better or just

different.

---

## 5. Pitfalls / failure modes

| Pitfall | Why it fails | Fix |

|---|---|---|

| **Skipping error analysis, jumping to automated metrics** | You automate the measurement of a failure mode you never actually identified — measuring the wrong thing precisely | Do the 100-trace open-coding pass *first*; let the data tell you what to measure |

| **Outsourcing trace review to non-domain experts** | Whoever codes the traces must recognize a mis-categorized self-transfer; a generic contractor can't | The PM / domain expert does the open coding |

| **Likert / 1–5 scale judges** | Scores drift run-to-run and can't be aligned to humans | Binary pass/fail with a rubric |

| **Trusting "agreement" instead of TPR/TNR** | On imbalanced data a lazy "always pass" judge looks 90% accurate while catching no errors | Report TPR and TNR separately; iterate until both clear |

| **Generic benchmarks (MMLU/GSM8k) as your eval** | They measure model capability, not *your app on your data* | Build the eval from your own production traces |

| **PM hands prompt control to engineering and walks away** | The what-counts-as-good judgment leaks out of product ownership | PM owns the eval rubric and the judge spec |

| **Eval built for a preference never specified** | You evaluate against a standard you never wrote down — moving target | Pin the rubric in writing before scoring |

| **No regression gate after the first pass** | The next prompt change silently regresses a fixed failure mode | Keep the validated eval suite running on every change |

---

## 6. Ethical boundary

Evals can be tuned to *hide* failures as easily as to surface them. Three lines

not to cross:

- **Never tune a judge to pass on harmful-but-frequent outputs** to make a

dashboard look green. In fintech, a judge that's been quietly relaxed so

mis-categorized income or under-reported spending "passes" is a deception that

reaches the user's money and possibly their tax filing.

- **Don't use real user financial data in eval sets without honoring the same PII

stripping the production pipeline requires** (Acme strips PII before queueing

— eval corpora must too). Traces are sensitive logs.

- **Report TPR/TNR honestly, including on the segments where the model is worst.**

An eval that's only ever run on the easy slice is a vanity metric. The point of

the discipline is to find where it breaks, not to manufacture a number.

---

## 7. Quick diagnostic

| Question | If "no" → |

|---|---|

| Have you read 50–100+ real production traces for this feature? | Stop. Do the Analyze pass before anything else. |

| Are your failure categories specific and actionable (not "weird outputs")? | Re-do axial coding; split vague labels. |

| Is every judge binary (not a 1–5 scale)? | Rewrite the judge prompt as pass/fail. |

| Have you validated the judge with TPR *and* TNR against human labels? | The judge is unverified — don't trust its numbers yet. |

| Did you confirm no other eval regressed before shipping the change? | Re-run the full suite; one fix often breaks another. |

| Does the eval suite gate every future prompt/model change? | Wire it into CI / the regression gate. |

---

## 8. Relationship to other great-pm skills

- **`metrics-design`** picks the product's North Star and KPIs (human-outcome

metrics). Evals measure *model output quality* — a different layer. The eval

accuracy rate often *becomes* an input metric in a `north-star-input-tree`.

- **`experiment-design`** runs A/B tests on *human* behavior with statistical

power. Evals run on *model output* with TPR/TNR. You frequently use them

together: an eval certifies the new prompt is better offline; an experiment

confirms the change moves user behavior online.

- **`ai-use-case-scoping`** decides *whether* a surface should be AI at all and

defines its accuracy floor. That floor is the *target* your evals measure

against — scope first, then eval.

- **`pm-tech-spec-review`** reviews the engineering spec generically; this gives

the PM the eval methodology that spec must implement.

- **`responsible-ai-guardrails`** is the *safety/red-team* layer (blocking

harmful output at runtime); evals are the *quality measurement* layer. The

hallucinated-category check in the worked example sits on the boundary between

them.

---

## 9. Output shape

An eval plan: the feature and its trace definition; the open-coding findings

(link to the coded trace set); the 5–8 axial failure categories with a frequency

pivot; the per-category decision (fix / eval / iterate); for each automated

category a binary judge prompt with rubric; the judge's TPR/TNR against a named

human-labeled set; the current measured quality on a large sample; and the

regression-gate wiring. Re-run the Analyze pass on a schedule — the input

distribution drifts.

---

## Sources

- Hamel Husain & Shreya Shankar (with Aakash Gupta), "AI Evals Step-by-Step

Masterclass" — https://www.aakashg.com/hamel-shreya-ai-evals-step-by-step/

- Aakash Gupta, "AI Evals for PMs: Everything You Need to Know" (Three Gulfs,

label elicitation) — https://www.news.aakashg.com/p/ai-evals

- Hamel Husain, "LLM Evals: Everything You Need to Know (FAQ)" —

https://hamel.dev/blog/posts/evals-faq/

- Shreya Shankar et al., "Who Validates the Validators? Aligning LLM-Assisted

Evaluation of LLM Outputs with Human Preferences" —

https://arxiv.org/abs/2404.12272

- Productboard, "AI Evals for Product Managers" —

https://www.productboard.com/blog/ai-evals-for-product-managers/