git:20260526.b859e18 to git:20260529.4056a23

---

name: eval-diagnose

description: Use when Galileo evidence is available and the user asks why a trace, session, log stream, experiment, metric, or AI app behavior failed, regressed, or became unsafe.

---

# Eval Diagnose

Use this skill for evidence-backed RCA once a packet, URL-derived evidence, or

trace/session/log-stream context is available.

## Required Reference

Use `skills/eval-engineer/references/rca-recipe.md`,

`skills/eval-engineer/assets/diagnosis-template.md`.

## Do

- Start from fetched evidence, not source-code guesses.

- Name the failing metric contract and what it proves.

- Inspect traces, spans, sessions, tool calls, retrieval context, and scorer

status to classify the fix surface.

- Classify the fix surface: prompt, tool schema, adapter, retriever, ranker,

guardrail, metric, dataset, or SDK wiring.

- Write diagnosis and bounded fix plan only when evidence supports it.

- Honor read-only requests. If the user says read-only, dry run, no edits, or

"do not edit files", do not write `.galileo/` artifacts. Return the RCA

inline and include a short "Would write" list for any suggested artifact

paths.

## Gotchas

- Fetched debug packets are the RCA source of truth when scorer jobs are still

settling or runner output disagrees with fetched metrics.

- A prompt diff, local score, or code diff is not proof of improvement without

before/after Galileo evidence.

- Bare correctness or factuality can be a smoke test only. Prefer the metric

contract tied to the case risk.

- Safe final wording is not enough for source-authority cases if unsafe or

forbidden retrieved context entered the task surface.

## Validation Loop

Before finalizing:

1. Run `python3 skills/eval-engineer/scripts/summarize_debug_packet.py <packet>`

unless a compact summary is already available.

2. Check that every RCA claim names supporting metric, trace, span, session,

experiment, dataset, or log-stream evidence.

3. Confirm the fix surface is bounded and allowed by `.galileo/config.yml`.

4. If any claim lacks evidence, downgrade it to a hypothesis or route to

`/eval-fetch` or `/eval-measure`.

## Output

Unless the request is read-only, produce `.galileo/current/diagnosis.md` and, when justified,

`.galileo/current/fix-plan.md`. If evidence is insufficient, route to

`/eval-fetch` or `/eval-measure`.