synthetic-trace-generation git:20260911.658528aA

Immutable. This exact content never changes and is served at /api/v1/blob/41d8a0ae462a64a3.

---
name: synthetic-trace-generation
description: >
  Generate grounded, diverse user interactions and run them against an
  instrumented application to create a verified trace dataset.
---

# Synthetic trace generation

Use the workflow when production traces are unavailable or do not cover enough
behavior for error analysis. The workflow assumes that the application, its
behavior specification, and authoritative data already exist.

## Inputs

- Behavior specification or rubric.
- Authoritative data, policies, deterministic functions, or simulator.
- Executable application interface.
- Trace destination and export path.
- Executable conversation plan validator.
- Project-specific conversation plan schema.
- Dataset size, model configuration, time budget, and cost budget.

## Invariants

1. Ground expected outcomes in authoritative sources. Neither the request
   generator nor the application under evaluation is its own oracle.
2. Keep hidden expected answers out of request generation prompts.
3. Write and review conversation plans before running the application.
4. Preserve ordinary coverage while adding difficult cases.
5. Preserve human decisions at the dimension and request review gates.

## Steps

### 1. Verify the application and tracing

Read the specification, tool contracts, permission rules, and ground truth
sources. Run one inexpensive request and verify that its trace contains:

- user and application messages;
- model identity and settings;
- tool calls and results;
- a stable scenario identifier;
- token usage and errors where available.

Do not start bulk generation until authentication, tracing, and state reset work.

### 2. Define dimensions

Derive dimensions from the specification, data, and intended analysis. A useful
dimension changes the expected behavior, execution path, or quality requirement.
Do not add a dimension merely because it is easy to vary.

Examples:

| Application | Useful dimensions | Reason |
| --- | --- | --- |
| Support agent | user role, intent, record state, applicable policy, user style | Permissions, tools, rules, and response requirements vary. |
| Coding agent | task type, language, repository size, test state, ambiguity | Context needs, tool paths, and completion checks vary. |
| Research agent | question type, source availability, recency, conflicting evidence | Retrieval, synthesis, and citation requirements vary. |
| Browser agent | workflow, authentication state, page state, interruption point | Available actions and recovery behavior vary. |

Show the proposed dimensions, values, and reason for each dimension to a person.
Stop for approval.

### 3. Build grounded conversation plans

Sample valid combinations of approved dimension values. Do not enumerate the
full Cartesian product. Count selected values and deliberately include rare or
risky combinations.

Maintain two pools:

- `coverage`: ordinary cases that repeatedly exercise important values;
- `challenge`: valid boundary conditions, missing information, corrections,
  permission boundaries, inconsistent records, and other supported difficulty.

Select concrete records from ground truth. Verify access permissions. Ensure
state changing scenarios target distinct records unless interaction is
intentional.

### 4. Record expectations

Compute the expected result before generating user language.

Use an objective expectation when authoritative data or deterministic code fixes
the answer or action. Record the outcome, reason, source type, and stable source
reference.

Use a human judgment expectation when several responses could satisfy the
requirement. Record a precise criterion and the supporting requirement or
rubric.

Store expectations in conversation plan records, but do not expose hidden facts
or expected answers to request generators.

### 5. Generate conversations concurrently

Generate user conversations separately from application execution. Use one
independent model call per conversation and launch the calls concurrently with
a bounded pool of at least two workers. Parallel coding subagents may provide
the pool when the environment supports them. Otherwise, use concurrent API
calls. Record the generation method, model, and concurrency. Do not ask one
model call to draft the whole dataset, and do not claim to use subagents when
none are available.

Give the generator only the role, user goal, selected style, user visible facts,
and required length. Do not fill the dataset with a shared opening or followup
template.

For multi-turn interactions:

- write one opening and an ordered list of exact user followups;
- keep every followup plausible for an unknown preceding response;
- develop one coherent issue through clarification, correction, pressure, or a
  decision;
- preserve the assigned style rather than polishing every user;
- omit stage directions, persona notes, evaluator language, and hidden facts.

Use the project's turn limit. Repeated requests to check, compare, cite,
confirm, or summarize do not create useful length.

### 6. Check and review conversations

Use an independent critic call for each conversation and run critic calls in a
bounded concurrent pool. The critic may rewrite language, but must preserve the
grounded plan, expectation, and assigned style.

Run mechanical checks for:

- schema and turn count errors;
- duplicate conversations or repeated utterances;
- invented identifiers, amounts, dates, entities, or personal facts;
- impossible role and record combinations;
- accidental or repeated write requests;
- followups that assume a specific unseen response;
- references to tools, traces, prompts, tests, or expected outcomes;
- missing coverage values.

Regenerate weak interactions from their plans. Run the executable validator.

Show a person complete conversations, including the longest examples, both
pools, all roles and styles, a state changing case, and difficult cases. Stop
until the person accepts or revises the language.

### 7. Run a pilot

Reset mutable state. Run a small representative set with one fixed application
model configuration. Persist status, errors, duration, observed messages, and
model identity for every scenario.

Verify trace completeness and scenario identifiers. Compare results with
recorded expectations. Treat provider refusals, timeouts, transport errors, and
missing traces as execution failures, not application quality failures.

Revise invalid scenarios and enrich challenge cases from difficult dimensions,
not by copying requests that happened to fail.

### 8. Run the final set

Repeat generation, criticism, human review, and validation for the final set.
Reset mutable state immediately before execution and keep the application model
configuration fixed.

Sequential execution is safest for shared mutable state. Use parallel execution
only when state changes are independent and provider limits, timeouts, tracing,
and the cost budget support the selected concurrency.

Rerun missing or failed scenarios without deleting completed run records.

### 9. Verify and export

Before error analysis:

1. Count completed scenario identifiers.
2. Confirm that every completed scenario has the expected traces.
3. Inspect traces across roles, lengths, styles, and scenario groups.
4. Confirm that messages, tool activity, model metadata, usage, and scenario
   identifiers are present.
5. Export the trace set with generation and application model provenance.
6. Report coverage and challenge results separately.

The workflow ends with a verified trace dataset. Failure taxonomy discovery and
quality measurement belong to the subsequent error analysis workflow.

## Artifacts

The workflow creates three separate artifacts:

1. **Conversation plans**, written before application execution. Each record
   defines one complete planned conversation: its dimensions, opening message,
   followups, and expected behavior.
2. **Run records**, written by the runner. Each record reports whether one
   planned conversation completed and preserves its observed messages, errors,
   duration, and model configuration.
3. **Traces**, written by the application's instrumentation. A conversation may
   produce one trace for the whole conversation or one trace per user turn,
   depending on the application. Stable plan identifiers link traces and run
   records back to the conversation plan.

Conversation plans are inputs to execution; they are not traces.

## Conceptual conversation plan record

Use the project's executable schema. Preserve at least the following concepts,
even when field names differ:

```json
{
  "id": "scenario-0042",
  "scenario_group": "challenge",
  "dimensions": {
    "role": "user_role",
    "intent": "requested_action",
    "record_state": "relevant_state",
    "difficulty": "boundary",
    "user_style": "terse_fragmentary",
    "turn_count": 3
  },
  "opening_message": "First user message",
  "followups": ["Second user message", "Third user message"],
  "expected": {
    "evaluation": "objective",
    "outcome": "expected result",
    "reason": "reason derived from ground truth",
    "source": {
      "type": "deterministic_oracle",
      "reference": "stable source reference"
    }
  }
}
```

For human judgment, replace `outcome` and `reason` with a precise `criterion`
and reference the supporting requirement or rubric.

## Limitations

Synthetic users are usually more relevant and consistent than production users,
even when generation prompts request varied styles. Synthetic traces complement
rather than replace production traffic.

Challenge enrichment changes failure frequency. A combined coverage and
challenge failure rate is not an estimate of production prevalence.

Scripted followups cannot react to an exact preceding response. Adaptive user
simulation requires a runtime user model with separate evaluation and cost.