git:20260519.1170f59 to git:20260526.b859e18

13 added, 1 removed. Audit A to A.

---

name: eval-dataset

- description: Use when turning Galileo failures, traces, metric gaps, or production examples into Eval Engineer dataset cases or reviewing candidate cases.

+ description: Use when the user asks to turn a failure into an eval, create/review/accept/reject dataset cases, or convert Galileo traces, metric gaps, or production examples into cases.

---

# Eval Dataset

Use this skill to build durable eval cases from evidence. Its job is dataset

quality control, not diagnosis or app fixing.

## Required Reference

Use `skills/eval-engineer/references/eval-datasets.md` for the canonical case

schema precedence, optional review metadata, promotion rules, bootstrap

guidance for different use cases, and Galileo SDK dataset usage.

## Required Inputs

Start from at least one evidence source:

- `.galileo/current/debug-packet.json`

- `.galileo/current/diagnosis.md`

- Galileo trace, session, experiment, or log-stream IDs

- production symptom with enough context to define expected behavior

If there is no concrete failure, regression, policy requirement, or metric gap,

ask for evidence before writing a case.

## Do

- Write new unreviewed cases to `.galileo/eval-dataset/candidates.jsonl`

unless the user asks to accept or reject a case.

- When accepting or rejecting cases, update `.galileo/eval-dataset/changelog.md`.

- Bootstrap datasets by use case: RAG, tool-calling agent, multi-turn,

workflow, safety/compliance, and tokenomics.

- Choose failure triggers that should be caught by named Galileo metrics or

explicit local gates.

- Follow the user-provided schema when the user gives one.

- Follow the existing Galileo dataset schema when appending to a fixed dataset.

- Do not force Eval Engineer fields into an upload schema. Put optional review

metadata in `.galileo/eval-dataset/candidates.jsonl`, `metadata`, or a

sidecar file when needed.

- Use retrieved-source gates when source authority matters and the chosen

schema can carry them. Final citations alone are not enough for

prompt-injection, privacy, stale-source, or source-authority cases.

- When the user wants to upload, reuse, or run cases in Galileo, use the SDK

workflow in the reference instead of improvising dataset shapes.

- Do not promote candidates without human review.

+ ## Validation Loop

+ Before finalizing a dataset change:

+ 1. Check the case against `skills/eval-engineer/references/eval-datasets.md`.

+ 2. Confirm the case has a concrete failure trigger and metric or local gate that

+ should catch it.

+ 3. Confirm required/forbidden citations, tools, permissions, source gates, and

+ abstention rules are explicit when relevant.

+ 4. Validate JSONL syntax after writing or editing.

+ 5. If accepting or rejecting a case, update `.galileo/eval-dataset/changelog.md`.

## Output

State whether the result is candidate, accepted, rejected, or blocked. Include

the target JSONL path, case IDs changed, evidence source, and remaining metric

gaps.