diagnose-agent-latency git:20261019.e33b7cfA

Immutable. This exact content never changes and is served at /api/v1/blob/fda7a1e9dabef3ed.

---
name: diagnose-agent-latency
description: Diagnose and optimize Canvas Agent turn latency with the development-only Agent DevTools or Langfuse traces. Use when a Canvas chat feels slow, TTFA/TTFT/first-render regresses, a run spends unexpected time in Canvas Host, Engine, Pi, tools, or Renderer, or a performance change needs a before/after trace comparison.
---

# Diagnose Agent Latency

Run a measurement-first loop. Attribute elapsed time to an owner before changing
code, change one bottleneck at a time, then repeat the same workload to prove the
result.

## 1. Establish a Reproducible Run

Work only in development mode. Observability is intentionally absent from
packaged production builds.

1. Record the prompt, scope, model/provider, runtime owner, attachment count,
   canvas node count, and whether the run is cold or warm.
2. Keep those inputs fixed across comparisons. Do not compare different models,
   scopes, prompts, or context sizes as if they were the same workload.
3. Run one warm-up and then at least three measured turns. Use the median for a
   local conclusion; use a larger Langfuse cohort when claiming a broad win.
4. Keep the raw `runId` for every sample.

Enable the local trace subscriber:

```bash
PULSE_CANVAS_AGENT_OBSERVABILITY=1 CANVAS_AGENT_DEBUG_TRACE=1 pnpm --filter canvas-workspace dev
```

The second environment variable enables detail capture in main and the preload
DevTools flag, so this command exposes the sidebar DevTools entry directly.
Alternatively, enable `canvas-agent-debug-trace` in Settings → Experimental and
reload the renderer when that legacy flag is visible. Open `#/debug` or use the
Debug Trace card on an assistant message.

For a local record, select the run in Agent DevTools and click **Export timing
JSON**. The export includes run/model/runtime identity, timeline and structural
events, without prompt/response or tool input/output payloads. Preserve the file
when reporting a slow request. Host and UI totals have separate end boundaries;
the waterfall extends to the last observed event.

DevTools retains up to 200 runs with a soft target of 50 MiB (50 × 1024 × 1024
bytes) for serialized run files. It always keeps the newest run, even if that
run alone exceeds the target. Run files are under
`<userData>/plugins/devtools/runs/`; summaries are in
`<userData>/plugins/devtools/index.json` (`src/plugins/main/devtools-run-store.ts`
and `src/plugins/main/plugin-store.ts`). The byte target excludes the index.
Each write prunes older runs against these limits. Export a run you need before
it ages out. In memory, a run keeps
at most 500 events; the cap drops the oldest detail events and never
`run.started` or `run.completed`.

For isolated development launches, set both a temporary `HOME` and Electron's
`--user-data-dir` (the plugin store uses Electron userData, not HOME). Use the
installed electron-vite entry after dependency preparation so changing HOME does
not trigger a separate Corepack download:

```bash
task_home=$(mktemp -d /tmp/pulse-latency.XXXXXX)
HOME="$task_home" PULSE_CANVAS_AGENT_OBSERVABILITY=1 CANVAS_AGENT_DEBUG_TRACE=1 \
  ./node_modules/.bin/electron-vite dev -- --user-data-dir="$task_home/electron-user-data"
```

Run that command from `apps/canvas-workspace`. Close only the instance you launched.

Use Langfuse when the task needs cross-run filters, cohorts, percentiles, or a
shared remote trace. Add `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, and
optionally `LANGFUSE_BASE_URL`. Never put credentials in source or renderer
state. Read `../../knowledge/langfuse-observability.md` before changing the
exporter or interpreting its trace model.

The built harness launches production output, so it cannot activate this
development-only DevTools plugin. Use `canvas-harness` for surrounding UI smoke
checks, but use the development launch above for latency traces.

## 2. Read the Timeline Correctly

Treat the lanes as ownership boundaries:

- **Canvas Host**: queue, scope activation, context preparation, runtime
  dispatch, and response processing.
- **Engine** or **Pi**: runtime execution. Attribute using the runtime label on
  the run, not the feature flag the operator expected.
- **Generation**: one provider/model call. Tool-using turns can have several.
- **Tool**: a named tool interval inside the runtime.
- **Renderer**: the first committed UI content milestone.

Interpret the summary metrics as follows:

- `Host total`: UI submission (when recorded) through final conversation persistence.
- `UI total`: UI submission through the committed end of generation. Missing for
  turns without an observed live-to-settled UI transition.
- `TTFA`: first stream activity, which may be a tool call rather than text.
- `TTFT`: first user-visible text. It can legitimately be later than TTFA.
- `First render`: first assistant text committed by the renderer (not tool-only activity).
- `Bottleneck`: longest exclusive duration, not proof of root cause by itself.

TTFA and TTFT are point milestones inside runtime execution. Do not add them to
phase durations. Compare milestone deltas instead:

```text
pre-runtime      = runtime start - request start
activity wait    = TTFA - runtime start
text wait        = TTFT - TTFA
render lag       = first render - TTFT
post-runtime     = total - runtime end
```

If a timestamp is missing, report it as missing. Do not turn it into zero.

## 3. Localize the Bottleneck

Use this order:

1. Large queue, scope, or context phase: inspect Canvas Host work before the
   runtime. Check context size, canvas reads, synchronous work, and duplicated
   preparation. Scope activation is split into nested `Scope ›` steps
   (`canvas.scope.*`, published through `traceScopeActivationStep`):
   workspace availability (SQLite trash check), wait for session writes, agent
   init or waiting for an in-flight init such as the composer warm-up, and
   session reconcile. Agent init is broken down into engine init (one
   `Engine plugin · <name>` step per plugin and one `MCP server · <name>` step
   per server, with connect/list-tools split, from the Engine's
   `pluginInitTiming` / `mcpServerTiming` events) and session restore. Init
   steps are collected once and replayed to every run that awaited the init,
   so a message that joined an untraced warm-up still shows the breakdown.
   Steps overlap their parent; the parent minus its steps is unattributed time.
2. Large activity wait with one generation: when the runtime reports provider
   boundaries (Engine does), the generation is split into `Generation ›`
   request preparation, wait for first chunk, output before text (reasoning
   or tool input), and text streaming. When the provider fetch reports
   response headers (Engine-built providers do), the wait splits into
   `Wait › network` (connection, TLS, gateway, queueing until headers) and
   `Wait › provider` (headers to first chunk: prefill and provider work). A
   long network wait with a short provider wait points at the network or a
   gateway, not at the prompt. The generation detail carries
   input/cached/output/reasoning tokens and the HTTP attempt count when the
   SDK retried; the row tooltip carries the provider request id for
   provider-side log lookup. A long first-chunk wait with a large uncached input points at
   prompt size or cache misses; a long pre-text segment with reasoning tokens
   points at model reasoning. Providers that do not stream reasoning content
   put reasoning time inside the first-chunk wait instead, so read that
   segment together with the reasoning token count. `generation.started`
   carries a prompt fingerprint (system and tool-list hashes, tool count,
   serialized size, and the MCP share) when a trace subscriber is attached.
   Zero cached input with equal hashes across two runs points at the
   provider or gateway (no cache, or no `prompt_cache_key` routing affinity);
   changing hashes point at a prompt prefix that varies. Otherwise suspect provider latency, network,
   model queueing, or delayed runtime callbacks. Compare the same prompt across
   providers only as a separately named experiment.
3. Gaps between generations: a `Retry backoff › attempt N` striped bar is
   the loop sleeping after a retryable provider error (the failed generation
   before it is red). A `Context compacted` bar is the summarizer call.
   `Hook › <plugin>.<hook>` bars are plugin hook calls of 10 ms or more; the
   waterfall header sums them and names the largest. Time left unexplained
   after these is host or runtime code without a probe.
4. Small TTFA but large text wait: inspect tool-first behavior, multiple model
   iterations, clarification, or reasoning before the first text. Do not label
   this as renderer slowness.
5. Long runtime after TTFT: inspect long generations, serial tools, repeated
   model calls, or work that needlessly blocks turn completion.
6. Large render lag: inspect stream batching, main-to-renderer IPC, React
   scheduling, and Markdown/render cost.
7. Large post-runtime tail: inspect response processing, persistence, trace
   serialization, and completion notifications.

Correlate the timeline with tool names and generation boundaries. A large
runtime bar only says the runtime owns the interval; its children explain what
happened inside it.

## 4. Map Evidence to Code

Start from the owner indicated by the trace:

- Event contract and bus: `src/shared/agent-observability.ts`,
  `src/plugins/main/agent-observability-bus.ts`
- Canvas Host phases: `src/main/agent/observability/host-run.ts`,
  `src/main/agent/service.ts`, `src/main/agent/canvas-agent.ts`
- Runtime selection: `src/main/agent/run/segment-execution.ts`
- Engine children: `src/main/agent/observability/engine-plugin.ts`
- Pi generations: `src/main/agent/observability/pi-generation-events.ts`,
  `src/main/agent/backends/pi-agent-harness-backend.ts`
- Renderer milestone: `src/main/agent/observability/renderer-mark.ts` and the
  chat stream renderer that publishes it
- Local model/UI: `src/plugins/renderer/devtools/performance-model.ts`,
  `AgentDebugPage.tsx`, `ChatDebugTrace.tsx`
- Remote exporter: `src/plugins/main/langfuse-observability.ts`

Do not optimize `packages/engine/src/core/loop.ts` merely because Canvas uses
Engine. First prove the slow interval is Engine-owned and not Canvas Host, Pi,
provider, tool, or Renderer time.

## 5. Run a Controlled Optimization Experiment

1. State one falsifiable hypothesis tied to one measured interval.
2. Choose the smallest code change that should move that interval.
3. Add or update a timing/behavior regression test when changing event
   boundaries, first-activity detection, runtime attribution, or renderer marks.
4. Run focused tests plus `node scripts/harness/run-harness-check.mjs`.
5. Repeat the fixed workload with the same model and context.
6. Compare medians and report the absolute and percentage change. Call out any
   regression in TTFA, TTFT, first render, total, errors, or output quality.
7. Revert or reject an optimization that only moves time between labels,
   suppresses instrumentation, drops work, or changes the workload.

For a performance-sensitive implementation, finish with the validation level
selected by `../validate-canvas-change/SKILL.md`.

## 6. Report the Result

Return a compact diagnosis with:

```text
Workload: prompt/scope/model/runtime/context, cold or warm
Samples: run IDs and count
Baseline: median Total, TTFA, TTFT, First render
Bottleneck: owner + interval + supporting child events
Hypothesis: causal mechanism, with confidence and alternatives
Experiment: one proposed or implemented change
Result: before/after absolute and percentage deltas
Validation: tests and trace comparison actually run
Privacy: whether only timing metadata or payload capture was enabled
```

Separate observation from inference. One trace can justify a targeted next
experiment, but not a general performance claim.