synthetic-trace-generation ยท diff
git:20260911.658528a to git:20260912.b3934f0
247 added, 166 removed. Audit A to A.
---
name: synthetic-trace-generation
description: >
- Generate grounded, diverse user interactions and run them against an
- instrumented application to create a verified trace dataset.
+ Generate grounded, diverse support conversations and run them against the
+ instrumented Cartwheel agent to create a verified trace dataset for error
+ analysis.
---
# Synthetic trace generation
- Use the workflow when production traces are unavailable or do not cover enough
- behavior for error analysis. The workflow assumes that the application, its
- behavior specification, and authoritative data already exist.
+ This skill generates a dataset of agent traces when production traces are unavailable or do not cover enough behavior for error analysis. The approach generates *scenarios* rather than raw user messages, because a scenario carries both a user request and an expected outcome grounded in the database and the policy documents. The expected outcome is what makes the trace useful for finding failures: without a recorded answer key, a reviewer has nothing to compare the agent's behavior against.
- ## Inputs
+ ## Why not generate user messages directly?
- - Behavior specification or rubric.
- - Authoritative data, policies, deterministic functions, or simulator.
- - Executable application interface.
- - Trace destination and export path.
- - Executable conversation plan validator.
- - Project-specific conversation plan schema.
- - Dataset size, model configuration, time budget, and cost budget.
+ Asking a model to "generate 250 support questions for an e-commerce platform" produces narrow data. The model tends to repeat the most common request type, such as order status, and ignores categories that appear less often in its training data, such as refund eligibility boundaries, store policy overrides, and permission denials. Adding more rows does not fix the problem, because more rows of the same kind do not introduce missing kinds. The synthetic data approach fixes the problem by defining the kinds of variation first, then generating requests that cover each kind.
+ ## What a scenario contains
+
+ A scenario is a planned support conversation with four parts:
+
+ 1. **A tuple of dimension values** that describe what kind of request the conversation represents, such as the user's role, intent, and the state of the relevant order.
+ 2. **User messages** (an opening message and optional followups) written in natural language from the tuple, as if a real user were typing into the chat window.
+ 3. **An expected outcome** computed from the database and the policy documents, stating what the agent should do and citing the authoritative source.
+ 4. **Metadata** linking the scenario to its group (coverage or challenge), its scenario identifier, and any relevant data quality case.
+
+ The scenario is an input to the runner, not a trace. Running a scenario through the agent produces two separate outputs: a result record for the attempted conversation, and Langfuse traces from the instrumented application. Cartwheel creates one trace per user turn and tags every trace with the scenario identifier, so the plan, the result, and the traces are always linked.
+
## Invariants
- 1. Ground expected outcomes in authoritative sources. Neither the request
- generator nor the application under evaluation is its own oracle.
- 2. Keep hidden expected answers out of request generation prompts.
- 3. Write and review conversation plans before running the application.
- 4. Preserve ordinary coverage while adding difficult cases.
- 5. Preserve human decisions at the dimension and request review gates.
+ These rules hold throughout the workflow. Each one prevents a specific mistake.
- ## Steps
+ 1. **Ground expected outcomes in authoritative sources.** The expected outcome for a refund scenario comes from the order's delivery date and the return window in `facts.yaml`, not from the model's answer. If the model's own answer were the answer key, a wrong answer would look correct. Sources include the database (SQL), the policy documents, and the data quality table.
- ### 1. Verify the application and tracing
+ 2. **Keep hidden expected answers out of generation prompts.** The model that writes the user's message receives only facts the simulated user would know (the product name, an approximate date, a vague description) and never the hidden expected outcome. A user asking about a return does not know the exact return window or whether the refund will be approved.
- Read the specification, tool contracts, permission rules, and ground truth
- sources. Run one inexpensive request and verify that its trace contains:
+ 3. **Write and review conversation plans before running the application.** Each application run costs model calls for every scenario in the file. Catching an invalid scenario before the run saves the cost of running it. Catching a bad user message before the run saves the cost of a trace that tests the wrong thing.
- - user and application messages;
- - model identity and settings;
- - tool calls and results;
- - a stable scenario identifier;
- - token usage and errors where available.
+ 4. **Preserve ordinary coverage while adding difficult cases.** Reporting the coverage and challenge sets separately prevents routine successes from hiding failures at the edges. A dataset of 175 coverage scenarios and 75 challenge scenarios lets a reviewer see both the baseline behavior and the boundary behavior.
- Do not start bulk generation until authentication, tracing, and state reset work.
+ 5. **Preserve human decisions at the dimension and request review gates.** The human approves the dimensions before generation starts, and reviews a sample of complete conversations before the runner starts. The coding agent proposes, generates, and validates, but the human decides what the dataset should contain.
- ### 2. Define dimensions
+ ## Step 1: Verify the application and tracing
- Derive dimensions from the specification, data, and intended analysis. A useful
- dimension changes the expected behavior, execution path, or quality requirement.
- Do not add a dimension merely because it is easy to vary.
+ Before generating any scenarios, confirm that the Cartwheel agent runs and that its traces are complete. Start the server and Langfuse, then send one inexpensive request (for example, "What's the status of order 4127?" as a shopper) and verify that its Langfuse trace contains:
- Examples:
+ - The user message and the agent's response.
+ - The model identity (for example, `gpt-5.5`).
+ - Tool calls and their results (for example, `get_order` with the order record).
+ - A `cartwheel.scenario_id` attribute on the trace.
+ - Token usage, if available.
- | Application | Useful dimensions | Reason |
+ Do not start bulk generation until authentication, tracing, and the database state reset (`uv run python -m seed.generate`) all work.
+
+ ## Step 2: Define dimensions of variation
+
+ A *dimension* is one named source of variation in the support requests. Each dimension should change the expected behavior, the execution path, or a quality requirement. Do not add a dimension merely because it is easy to vary.
+
+ The Cartwheel specification, the seeded data, and the policy documents motivate the following dimensions:
+
+ | Dimension | Values | Why it matters |
| --- | --- | --- |
- | Support agent | user role, intent, record state, applicable policy, user style | Permissions, tools, rules, and response requirements vary. |
- | Coding agent | task type, language, repository size, test state, ambiguity | Context needs, tool paths, and completion checks vary. |
- | Research agent | question type, source availability, recency, conflicting evidence | Retrieval, synthesis, and citation requirements vary. |
- | Browser agent | workflow, authentication state, page state, interruption point | Available actions and recovery behavior vary. |
+ | **Role** | shopper, merchant, support | Different roles have different permissions. A shopper can view their own orders, a merchant can view their store's orders, and support can view any order. The same request from different roles triggers different tool results and permission denials. |
+ | **Intent** | order status, refund, cancellation, policy question, product search, dispute, out of scope | Each intent exercises different tools and different parts of the specification. A refund request calls `get_order` and `issue_refund`, while a policy question calls `search_help_center` and `get_policy`. |
+ | **Record involved** | an order (in window, past window, above threshold, placed, shipped), a product, a store policy page, none | The state of the record determines the expected outcome. An order delivered 15 days ago is eligible for a return under the 30-day window, while an order delivered 45 days ago is not. |
+ | **Applicable policy** | platform rule, store override, none | Some stores override the platform return window. Juniper Home Goods has a 14-day window instead of the platform's 30 days. A request about a Juniper order triggers a different expected outcome than the same request about a Blue Heron Ceramics order. |
+ | **Tools needed** | none, one lookup, several calls | A policy question needs one lookup (`search_help_center`), while a refund needs several calls (`get_order` to check eligibility, then `issue_refund`). Varying the number of tool calls exercises different parts of the agent loop. |
+ | **Difficulty** | well specified, ambiguous, missing information, boundary | A well-specified request names the order and the action. An ambiguous request describes a product without an order number, so the agent must call `find_order` first. A boundary request sits exactly at the return window or the refund threshold. |
+ | **User language style** | neutral\_conversational, terse\_fragmentary, typo\_heavy, confused\_rambling, frustrated\_impatient, repetitive\_pressuring, operational\_shorthand, requests\_short\_plain\_answer | The specification requires direct and respectful responses (RESP-5) regardless of user style. Varying the style tests whether the agent handles terse fragments and frustrated messages as well as polite requests. The validator enforces one of these exact values. |
- Show the proposed dimensions, values, and reason for each dimension to a person.
- Stop for approval.
+ Each scenario also records its **turn count** (1 through 25), which equals one plus the number of followups. The turn count is set per scenario through the followups list rather than as a separate dimension. The validator checks that the recorded turn count matches the actual number of messages.
- ### 3. Build grounded conversation plans
+ ### Proposing dimensions to the human
- Sample valid combinations of approved dimension values. Do not enumerate the
- full Cartesian product. Count selected values and deliberately include rare or
- risky combinations.
+ After reading `SPEC.md`, the seeded data, and the data quality cases, propose the dimensions and their values to the human for approval. Use AskUserQuestion to present the proposed dimensions and wait for the human's decision before generating any scenarios. The human may add, remove, or revise dimensions. Do not generate scenarios until the dimensions are approved.
- Maintain two pools:
+ Any additional dimension beyond the seven above needs a reason from the specification or the seeded data. For example, "conversation language" would not be a useful dimension because the specification does not mention multilingual support.
- - `coverage`: ordinary cases that repeatedly exercise important values;
- - `challenge`: valid boundary conditions, missing information, corrections,
- permission boundaries, inconsistent records, and other supported difficulty.
+ ## Step 3: Build grounded conversation plans
- Select concrete records from ground truth. Verify access permissions. Ensure
- state changing scenarios target distinct records unless interaction is
- intentional.
+ Sample valid combinations of the approved dimension values. Do not enumerate the full Cartesian product, because many combinations are invalid (a support staff member asking about their own orders is not a valid scenario), and the product of seven dimensions with multiple values each grows quickly. Instead, count how often each value appears and deliberately include rare and risky combinations, such as store overrides, boundary cases, and damaged records.
- ### 4. Record expectations
+ ### Two pools
- Compute the expected result before generating user language.
+ Maintain two separate pools:
- Use an objective expectation when authoritative data or deterministic code fixes
- the answer or action. Record the outcome, reason, source type, and stable source
- reference.
+ - **Coverage** (175 scenarios): ordinary requests that exercise every important dimension value. These include straightforward order status checks, simple refund requests within the return window, and routine policy questions.
+ - **Challenge** (75 scenarios): intentionally difficult requests that test policy boundaries, damaged records, permission edges, corrections across turns, and missing information. Five challenge scenarios should target each of the six data quality cases in the database.
- Use a human judgment expectation when several responses could satisfy the
- requirement. Record a precise criterion and the supporting requirement or
- rubric.
+ Report results for the two pools separately, because mixing them lets the routine majority hide failures at the edges.
- Store expectations in conversation plan records, but do not expose hidden facts
- or expected answers to request generators.
+ ### Selecting records from the database
- ### 5. Generate conversations concurrently
+ Each scenario that involves an order, product, or policy must reference a real record from the seeded database. For example, a refund scenario targeting a past-window order should reference an order whose delivery date is more than 30 days ago. Query the database to find a matching record, and verify that the selected user has permission to access the record under the specified role.
- Generate user conversations separately from application execution. Use one
- independent model call per conversation and launch the calls concurrently with
- a bounded pool of at least two workers. Parallel coding subagents may provide
- the pool when the environment supports them. Otherwise, use concurrent API
- calls. Record the generation method, model, and concurrency. Do not ask one
- model call to draft the whole dataset, and do not claim to use subagents when
- none are available.
+ Scenarios that change state (refunds, cancellations) should target distinct records, so that one scenario's side effects do not invalidate another scenario's expected outcome.
- Give the generator only the role, user goal, selected style, user visible facts,
- and required length. Do not fill the dataset with a shared opening or followup
- template.
+ ### Data quality cases
- For multi-turn interactions:
+ The seeded database contains six deliberately damaged records. Query the `data_quality_cases` table to see them:
- - write one opening and an ordered list of exact user followups;
- - keep every followup plausible for an unknown preceding response;
- - develop one coherent issue through clarification, correction, pressure, or a
- decision;
- - preserve the assigned style rather than polishing every user;
- - omit stage directions, persona notes, evaluator language, and hidden facts.
+ ```bash
+ sqlite3 -header -column data/cartwheel.db \
+ 'SELECT case_id, entity_type, entity_id, description, expected_handling FROM data_quality_cases ORDER BY case_id;'
+ ```
- Use the project's turn limit. Repeated requests to check, compare, cite,
- confirm, or summarize do not create useful length.
+ For example, order 8002 has a delivered status but no delivery date. A scenario about this order should record that the agent must not compute a return deadline from a missing date, and should cite `dq-order-missing-delivery-date` as the data quality case identifier.
- ### 6. Check and review conversations
+ ## Step 4: Compute expected outcomes
- Use an independent critic call for each conversation and run critic calls in a
- bounded concurrent pool. The critic may rewrite language, but must preserve the
- grounded plan, expectation, and assigned style.
+ Compute the expected outcome for each scenario before generating the user's message. The expected outcome states what the agent should do, and cites the authoritative source that determines the answer.
- Run mechanical checks for:
+ ### Objective expectations
- - schema and turn count errors;
- - duplicate conversations or repeated utterances;
- - invented identifiers, amounts, dates, entities, or personal facts;
- - impossible role and record combinations;
- - accidental or repeated write requests;
- - followups that assume a specific unseen response;
- - references to tools, traces, prompts, tests, or expected outcomes;
- - missing coverage values.
+ Use an objective expectation when the database or a deterministic rule fixes the answer. For example:
- Regenerate weak interactions from their plans. Run the executable validator.
+ - **Order 3980**, delivered 2026-05-17, Blue Heron Ceramics, no store override. The platform return window is 30 days from delivery (`facts.yaml`). Today is 45 days after delivery. **Expected outcome:** refund denied. **Source:** the return policy in `facts.yaml` and the order row.
+ - **Order 6974**, delivered 17 days ago, Juniper Home Goods, 14-day store override. **Expected outcome:** refund denied (the store override is stricter). **Source:** the store policy document for Juniper Home Goods.
+ - **Order 8002**, delivered status but no delivery date (data quality case `dq-order-missing-delivery-date`). **Expected outcome:** do not compute a return deadline. **Source:** the data quality table.
- Show a person complete conversations, including the longest examples, both
- pools, all roles and styles, a state changing case, and difficult cases. Stop
- until the person accepts or revises the language.
+ Record the outcome, a short reason, the source type (`sql`, `eligibility_function`, `policy_document`, or `data_quality_table`), and a stable reference to the source (for example, `cw-returns` for the platform return policy, or `dq-order-missing-delivery-date` for a data quality case).
- ### 7. Run a pilot
+ ### Human judgment expectations
- Reset mutable state. Run a small representative set with one fixed application
- model configuration. Persist status, errors, duration, observed messages, and
- model identity for every scenario.
+ Use a human judgment expectation when several responses could satisfy the requirement. For example, when a shopper asks a vague question and the agent refuses or asks for clarification, the expected behavior is "a clear and respectful explanation of what information is needed" rather than one exact sentence. Record a precise criterion (for example, "the response must explain what information is missing without revealing other users' data") and cite the relevant SPEC.md requirement (for example, RESP-3 and RESP-4) with source type `specification`.
- Verify trace completeness and scenario identifiers. Compare results with
- recorded expectations. Treat provider refusals, timeouts, transport errors, and
- missing traces as execution failures, not application quality failures.
+ ## Step 5: Generate conversations
- Revise invalid scenarios and enrich challenge cases from difficult dimensions,
- not by copying requests that happened to fail.
+ Generate the user conversations separately from running the application. The generation model writes the user's messages from the tuple and the user-visible facts, but never sees the hidden expected outcome.
- ### 8. Run the final set
+ ### One call per conversation
- Repeat generation, criticism, human review, and validation for the final set.
- Reset mutable state immediately before execution and keep the application model
- configuration fixed.
+ Use one independent model call per conversation, and launch the calls concurrently with a bounded pool of at least two workers. Parallel coding subagents may provide the pool when the environment supports them. Do not ask one model call to draft the whole dataset, because a single call produces conversations that share structure and phrasing.
- Sequential execution is safest for shared mutable state. Use parallel execution
- only when state changes are independent and provider limits, timeouts, tracing,
- and the cost budget support the selected concurrency.
+ ### The generation prompt
- Rerun missing or failed scenarios without deleting completed run records.
+ Give the generation model the role, the user's goal, the selected language style, the user-visible facts (for example, the product name and an approximate time frame), and the required number of turns. Do not include the order number, the exact delivery date, the return window, or the expected outcome.
- ### 9. Verify and export
+ For example, to generate the request for the order-3980 refund scenario:
- Before error analysis:
+ ```
+ You are simulating a customer messaging the Cartwheel support agent.
+ Write a realistic one sentence message from the user described below.
+ Write naturally, as if the user is typing into a chat window. Do not
+ state every fact from the tuple. Do not mention the order number.
- 1. Count completed scenario identifiers.
- 2. Confirm that every completed scenario has the expected traces.
- 3. Inspect traces across roles, lengths, styles, and scenario groups.
- 4. Confirm that messages, tool activity, model metadata, usage, and scenario
- identifiers are present.
- 5. Export the trace set with generation and application model provenance.
- 6. Report coverage and challenge results separately.
+ Role: shopper
+ Intent: refund
+ Record: a vase from Blue Heron Ceramics, purchased about six weeks ago
+ User style: confused, rambling
+ ```
- The workflow ends with a verified trace dataset. Failure taxonomy discovery and
- quality measurement belong to the subsequent error analysis workflow.
+ The model might produce: "Can I return the vase I got a while ago? I never used it."
- ## Artifacts
+ The message leaves out the order number, the exact delivery date, and the return window, because a real user would not know or state those facts.
- The workflow creates three separate artifacts:
+ ### Multi-turn conversations
- 1. **Conversation plans**, written before application execution. Each record
- defines one complete planned conversation: its dimensions, opening message,
- followups, and expected behavior.
- 2. **Run records**, written by the runner. Each record reports whether one
- planned conversation completed and preserves its observed messages, errors,
- duration, and model configuration.
- 3. **Traces**, written by the application's instrumentation. A conversation may
- produce one trace for the whole conversation or one trace per user turn,
- depending on the application. Stable plan identifiers link traces and run
- records back to the conversation plan.
+ For scenarios with more than one turn, write one opening message and an ordered list of exact followup messages. Each followup must be plausible without knowing the agent's preceding response, because the followups are scripted before the agent runs. A followup like "Yes, go ahead with the refund" assumes the agent offered a refund, which it might not have. Instead, write followups that develop one coherent issue through clarification, correction, or added detail, such as "Actually, I think it was the desk organizer, not the vase."
- Conversation plans are inputs to execution; they are not traces.
+ Preserve the assigned language style in the followups. A terse user stays terse. A frustrated user stays frustrated. Do not polish every user into polite grammatical prose.
- ## Conceptual conversation plan record
+ ### Critic pass
- Use the project's executable schema. Preserve at least the following concepts,
- even when field names differ:
+ After generating all conversations, run an independent critic call for each conversation concurrently. The critic checks for:
+ - Invented identifiers, amounts, dates, or product names that do not match the selected database row.
+ - Followups that assume a specific agent response.
+ - Conversations that share a template opening or followup.
+ - Language that a real user would not produce, such as quoting the return policy by name.
+ - References to tools, traces, prompts, or expected outcomes that break the simulation.
+
+ The critic may rewrite language but must preserve the grounded plan, the expectation, and the assigned style.
+
+ ## Step 6: Review conversations with the human
+
+ Run the executable validator on the generated file:
+
+ ```bash
+ uv run python -m scenarios.validate scenarios/pilot_scenarios.jsonl
+ ```
+
+ The validator checks the JSON schema, required tuple fields, unique identifiers, turn counts, duplicate conversations, and the expected-outcome format. Fix every reported error before proceeding.
+
+ Then show the human a sample of complete conversations for review. Include the longest conversation, both scenario groups, all three roles, a state-changing scenario (refund or cancellation), a difficult scenario, and several user language styles. The human reads the opening and every followup and decides whether the language is realistic, the followups make sense without seeing the agent's response, and the expected outcome matches the selected record.
+
+ Do not start the runner until the human accepts the conversation sample.
+
+ ## Step 7: Run a pilot
+
+ Reset the development data first, because earlier runs may have changed order states:
+
+ ```bash
+ uv run python -m seed.generate
+ ```
+
+ Run a small representative set (about 30 scenarios) on one model:
+
+ ```bash
+ uv run python -m scenarios.runner scenarios/pilot_scenarios.jsonl \
+ --model YOUR_MODEL --output scenarios/pilot-results.jsonl
+ ```
+
+ Review at least 10 results. For each result, compare the agent's behavior with the recorded expected outcome. A confirmed failure requires a valid scenario and observed behavior that conflicts with the expected outcome or a SPEC.md requirement. Record the evidence: which database value, policy, tool result, or requirement supports the judgment.
+
+ The pilot must contain at least five confirmed failures. If it does not, add challenge scenarios from the difficult dimensions (store overrides, boundary cases, data quality defects), or use a lower-capability model from the same provider. Do not add scenarios by copying requests that happened to fail; use the underlying dimension (such as "store policy override") to generate new cases.
+
+ ## Step 8: Create and run the final set
+
+ Generate the full set of 250 scenarios (175 coverage, 75 challenge) following the same steps: generate conversations concurrently, run critic calls, validate, and show the human a sample. Give every final scenario a new identifier distinct from the pilot identifiers, because the trace export selects traces by scenario identifier.
+
+ Reset the development data immediately before the final run, then run the full set on one model:
+
+ ```bash
+ uv run python -m seed.generate
+
+ uv run python -m scenarios.runner scenarios/support_scenarios.jsonl \
+ --model YOUR_MODEL --output scenarios/final-results.jsonl
+ ```
+
+ If any scenarios fail (timeout, provider error, missing trace), rerun only the affected scenarios:
+
+ ```bash
+ uv run python -m scenarios.runner scenarios/support_scenarios.jsonl \
+ --model YOUR_MODEL --output scenarios/final-results.jsonl --resume
+ ```
+
+ The `--resume` flag keeps every completed record and reruns only the scenarios whose record is missing or incomplete.
+
+ ## Step 9: Verify and export
+
+ Export the traces and confirm completeness:
+
+ ```bash
+ uv run python -m scenarios.export_langfuse \
+ scenarios/support_scenarios.jsonl traces/support_traces.json
+ ```
+
+ Before finishing, verify:
+
+ 1. Every final scenario identifier has a matching trace in the export.
+ 2. Each trace contains the conversation messages, tool calls and results, model metadata, and the `cartwheel_scenario_id` attribute.
+ 3. The export includes traces from both scenario groups, all three roles, and a range of turn counts.
+
+ Report the coverage and challenge results separately. The workflow ends with a verified trace dataset ready for error analysis.
+
+ ## Conversation plan format
+
+ Each line of the scenario JSONL file is one complete planned conversation. Here is a Cartwheel example:
+
```json
{
- "id": "scenario-0042",
+ "id": "support-0042",
"scenario_group": "challenge",
- "dimensions": {
- "role": "user_role",
- "intent": "requested_action",
- "record_state": "relevant_state",
- "difficulty": "boundary",
- "user_style": "terse_fragmentary",
- "turn_count": 3
+ "data_quality_case_id": "dq-order-missing-delivery-date",
+ "tuple": {
+ "role": "shopper",
+ "user_id": 392,
+ "intent": "return_deadline",
+ "record_state": "order_missing_delivery_date",
+ "applicable_policy": "cw-returns",
+ "tools_needed": "one_lookup",
+ "difficulty": "missing_information",
+ "user_style": "confused_rambling",
+ "turn_count": 1,
+ "order_id": 8002
},
- "opening_message": "First user message",
- "followups": ["Second user message", "Third user message"],
+ "opening_message": "Can I still send back the pencil set I got from Atlas Stationery?",
+ "followups": [],
"expected": {
"evaluation": "objective",
- "outcome": "expected result",
- "reason": "reason derived from ground truth",
+ "outcome": "do_not_compute_return_deadline",
+ "reason": "The order has a delivered status but no delivery date.",
"source": {
- "type": "deterministic_oracle",
- "reference": "stable source reference"
+ "type": "data_quality_table",
+ "reference": "dq-order-missing-delivery-date"
}
}
}
```
- For human judgment, replace `outcome` and `reason` with a precise `criterion`
- and reference the supporting requirement or rubric.
+ The fields:
+ - **id**: a unique identifier that links the plan, the run result, and the Langfuse traces through `cartwheel.scenario_id`.
+ - **scenario_group**: `coverage` or `challenge`.
+ - **data_quality_case_id**: the `case_id` from the `data_quality_cases` table, or `null` when the scenario does not target a damaged record.
+ - **tuple**: one value per dimension, plus the `user_id` and any record identifier (such as `order_id`) that grounds the scenario in the database.
+ - **opening_message**: the first user message, written naturally without hidden facts.
+ - **followups**: an ordered list of exact user messages for subsequent turns. An empty list means a single-turn conversation.
+ - **expected**: the answer key. For an objective expectation, record `outcome`, `reason`, and `source` with one of the types `sql`, `eligibility_function`, `policy_document`, or `data_quality_table`. For a human judgment expectation, replace `outcome` and `reason` with a `criterion` describing what a correct response looks like, and use source type `specification` with the relevant SPEC.md requirement identifier.
+
+ Here is a human judgment example:
+
+ ```json
+ {
+ "id": "support-0108",
+ "scenario_group": "coverage",
+ "data_quality_case_id": null,
+ "tuple": {
+ "role": "shopper",
+ "user_id": 210,
+ "intent": "out_of_scope",
+ "record_state": "none",
+ "applicable_policy": "none",
+ "tools_needed": "none",
+ "difficulty": "well_specified",
+ "user_style": "neutral_conversational",
+ "turn_count": 1
+ },
+ "opening_message": "Can you help me file my taxes?",
+ "followups": [],
+ "expected": {
+ "evaluation": "human_judgment",
+ "criterion": "The agent declines the request in one or two sentences and points to what it can help with, without revealing inaccessible information.",
+ "source": {
+ "type": "specification",
+ "reference": "SCOPE-2, RESP-4"
+ }
+ }
+ }
+ ```
+
## Limitations
- Synthetic users are usually more relevant and consistent than production users,
- even when generation prompts request varied styles. Synthetic traces complement
- rather than replace production traffic.
+ **Synthetic users are more cooperative than real users.** Even when the generation prompt requests varied styles, generated users tend to be more polite, more grammatical, and more relevant than real users. A generated frustrated user says "this is really annoying, can you just process the refund" rather than sending three messages of unrelated complaints before arriving at the actual request. Synthetic traces complement production traffic rather than replacing it.
- Challenge enrichment changes failure frequency. A combined coverage and
- challenge failure rate is not an estimate of production prevalence.
+ **Challenge enrichment changes the observed failure rate.** A dataset with 75 intentionally difficult scenarios will show more failures than a random sample of production requests. Reporting the coverage and challenge sets separately prevents the enriched failure rate from being mistaken for the production failure rate.
- Scripted followups cannot react to an exact preceding response. Adaptive user
- simulation requires a runtime user model with separate evaluation and cost.
+ **Scripted followups cannot react to the agent's response.** A followup like "yes, go ahead" assumes the agent offered an action. Because the followups are written before the agent runs, each followup must be plausible regardless of what the agent said in the previous turn. Adaptive user simulation, where a second model reads the agent's response and generates a contextual followup, requires a runtime user model with its own evaluation and cost.