v1.0 to v2.0
104 added, 107 removed. Audit A to A.
---
name: observability-instrumentation
- description: Instrument agents with structured traces, events, metrics, and execution-phase diagnostics for production debugging.
- compatibility: Reactive Agents projects using observability and EventBus layers.
+ description: Configure verbosity levels, live log streaming, JSONL file export, model I/O logging, and audit trails for monitoring agent execution.
+ compatibility: Reactive Agents TypeScript projects using @reactive-agents/*
metadata:
author: reactive-agents
- version: "1.0"
+ version: "2.0"
+ tier: "capability"
---
- # Observability Instrumentation
-
- Use this skill when you need explainable, debuggable agent behavior.
+ # Observability and Instrumentation
## Agent objective
- When implementing observability, generate code/config that:
-
- - Makes every execution phase measurable.
- - Correlates events across agent, task, and delegated sub-runs.
- - Surfaces actionable bottlenecks instead of raw logs only.
+ Produce a builder with observability configured at the right verbosity level, with optional live streaming and file export, so agent execution can be monitored and debugged.
- ## What this skill does
+ ## When to load this skill
- - Emits phase-level timing and token/cost metrics.
- - Captures tool call success/error distributions.
- - Correlates task, agent, and session identifiers in traces.
+ - Debugging unexpected agent behavior in development
+ - Capturing structured logs for post-hoc analysis
+ - Streaming live execution traces to a dashboard or log aggregator
+ - Auditing all tool calls and model decisions in production
+ - Comparing model I/O before/after a system prompt change
- ## Workflow
+ ## Implementation baseline
- 1. Enable observability in the builder.
- 2. Subscribe metrics collectors to execution events.
- 3. Log model I/O boundaries and major state transitions.
- 4. Surface bottleneck alerts in execution summaries.
+ ```ts
+ import { ReactiveAgents } from "@reactive-agents/runtime";
- ## Minimum telemetry set
+ const agent = await ReactiveAgents.create()
+ .withName("monitor")
+ .withProvider("anthropic")
+ .withReasoning({ defaultStrategy: "adaptive", maxIterations: 10 })
+ .withTools({ allowedTools: ["web-search", "http-get", "checkpoint"] })
+ .withObservability({
+ verbosity: "normal", // show metrics dashboard on completion
+ live: true, // stream events as they happen
+ file: "./logs/agent.jsonl", // write structured JSONL log
+ })
+ .withAudit() // record all tool calls and decisions
+ .build();
+ ```
- - Phase durations and iteration count.
- - Tokens and cost per task.
- - Verification outcomes.
- - Tool latency/error rate.
+ ## Verbosity levels
- ## Expected implementation output
+ | Level | Output |
+ |-------|--------|
+ | `"minimal"` | No output except final result — for programmatic/embedded use |
+ | `"normal"` | Metrics dashboard on completion (default) — recommended for production |
+ | `"verbose"` | Step-by-step phase summaries as the agent runs |
+ | `"debug"` | Full phase traces including tool call args/results and model responses |
- - Builder usage with `.withObservability()` and appropriate verbosity.
- - Structured metrics/events suitable for dashboards and alerts.
- - Diagnostics that connect slow phases to concrete tool/model causes.
+ ```ts
+ .withObservability({ verbosity: "minimal" }) // silent — result only
+ .withObservability({ verbosity: "normal" }) // dashboard on finish (default)
+ .withObservability({ verbosity: "verbose" }) // running commentary
+ .withObservability({ verbosity: "debug" }) // everything, including prompt/response dumps
+ ```
- ## Code Examples
+ ## Key patterns
- ### Enabling Observability
+ ### Live streaming
- The primary way to enable observability is with the `.withObservability()` builder method. It accepts different verbosity levels and can stream live events or log to a file.
+ ```ts
+ .withObservability({ verbosity: "verbose", live: true })
+ // Streams log events in real-time as each phase completes.
+ // Without live: true, output is buffered and printed at the end.
+ // Combine with verbosity: "verbose" or "debug" for full traces.
+ ```
- ```typescript
- import { ReactiveAgents } from "@reactive-agents/runtime";
+ ### JSONL file export
- // Example 1: Normal verbosity with dashboard on completion and JSONL file export
- const agent1 = await ReactiveAgents.create()
- .withName("observed-agent")
- .withProvider("anthropic")
- .withObservability({ verbosity: "normal", live: false, file: "/tmp/agent-run.jsonl" })
- .build();
+ ```ts
+ .withObservability({
+ verbosity: "normal",
+ file: "./logs/run-2026-04-09.jsonl", // appends structured events as JSONL
+ })
+ // Each line is a JSON object: { timestamp, event, phase, data }
+ // Suitable for ingestion into log aggregators (Datadog, Loki, etc.)
+ ```
- // Example 2: Verbose mode for detailed, structured phase logs during execution
- const agent2 = await ReactiveAgents.create()
- .withName("verbose-agent")
- .withProvider("anthropic")
- .withObservability({ verbosity: "verbose", live: true })
- .build();
+ ### Model I/O logging
- // Example 3: Minimal mode for silent execution
- const agent3 = await ReactiveAgents.create()
- .withName("minimal-agent")
- .withProvider("anthropic")
- .withObservability({ verbosity: "minimal" })
- .build();
+ ```ts
+ .withObservability({
+ verbosity: "debug",
+ logModelIO: true, // log full system prompts and model responses
+ })
+ // logModelIO defaults to true at "debug" verbosity, false at all other levels.
+ // Set logModelIO: false at "debug" to debug phases without exposing prompt content.
```
- ### Expected Dashboard Output
-
- When `verbosity` is `"normal"` or higher, a summary dashboard is printed upon completion. This provides a high-level overview of the agent's performance without requiring manual log parsing.
+ ### Audit trail
- ```text
- ┌──────────────────────────────────────────────────────────────────────────┐
- │ 📄 Agent Execution Summary │
- ├──────────────────────────────────────────────────────────────────────────┤
- │ ✅ Success Duration: 22.1s Steps: 6 │
- │ Model: cogito:14b (ollama) Tokens: 13,299 │
- └──────────────────────────────────────────────────────────────────────────┘
- 📊 Execution Timeline
- ├─ [bootstrap] 4ms ✅
- ├─ [strategy-select] 0ms ✅
- ├─ [think] 22.1s ⚠️ (6 iter, 100% of time)
- ├─ [memory-flush] 2ms ✅
- └─ [complete] 0ms ✅
- 🔧 Tool Execution (2 called)
- ├─ github/list_commits ✅ 1 calls, 281ms avg
- └─ signal/send_message_to_user ✅ 1 calls, 244ms avg
- ⚠️ Alerts & Insights
- └─ ⚠️ think phase blocked ≥10s (LLM latency)
+ ```ts
+ .withAudit()
+ // Records structured audit events for every tool call, guardrail check, contract
+ // validation, and cost tracking decision.
+ // Audit events appear in the observability stream and are written to the file if configured.
+ // Use alongside .withObservability() to capture audit events to a file.
```
- ### Structured logging and error hooks
-
- For file-based structured logging and error monitoring, combine `withLogging()` and `withErrorHandler()`:
+ ### Minimal production config (observability without noise)
- ```typescript
- const agent = await ReactiveAgents.create()
- .withProvider("anthropic")
- .withObservability({ verbosity: "normal", live: true })
- // Structured JSON logs to file with rotation
- .withLogging({
- level: "info",
- format: "json",
- output: "file",
- filePath: "/var/log/agent.jsonl",
- maxFileSizeMb: 50,
- maxFiles: 7,
- })
- // Error callback for external monitoring (Sentry, Datadog, etc.)
- .withErrorHandler((err, ctx) => {
- console.error(`[${ctx.phase}] iteration ${ctx.iteration}: ${err.message}`);
- })
- .build();
+ ```ts
+ .withObservability({ verbosity: "minimal" })
+ // Disables all terminal output — the agent runs silently.
+ // Results are returned programmatically only.
+ // Combine with .withCostTracking() to still enforce budgets without logging.
```
- ### Strategy switch observability
+ ### Development debug config
- ```typescript
- await agent.subscribe("StrategySwitchEvaluated", (event) => {
- console.log(`Eval: ${event.fromStrategy} → ${event.recommendedStrategy} (will switch: ${event.willSwitch})`);
- });
- await agent.subscribe("StrategySwitched", (event) => {
- console.log(`Switched: ${event.fromStrategy} → ${event.toStrategy} (#${event.switchNumber})`);
- });
+ ```ts
+ .withObservability({
+ verbosity: "debug",
+ live: true,
+ logModelIO: true,
+ file: "./debug.jsonl",
+ })
+ // Maximum visibility: live stream + full model I/O + JSONL file
```
- ## Pitfalls to avoid
+ ## ObservabilityOptions reference
- - Relying on ad-hoc `console.log` for root cause analysis.
- - Missing correlation IDs across sub-agent calls.
- - No alerting on long think/tool phases.
- - Using `withObservability()` without `withErrorHandler()` — errors go to the EventBus but aren't forwarded to external systems.
+ | Field | Type | Default | Notes |
+ |-------|------|---------|-------|
+ | `verbosity` | `"minimal"\|"normal"\|"verbose"\|"debug"` | `"normal"` | Output detail level |
+ | `live` | `boolean` | `false` | Stream events in real-time |
+ | `file` | `string` | — | JSONL log file path (appends) |
+ | `logModelIO` | `boolean` | `true` at debug, `false` otherwise | Log full prompts and responses |
+
+ ## Pitfalls
+
+ - `verbosity: "normal"` prints a metrics dashboard at completion — this is terminal output, not a structured event. Use `file` for structured capture
+ - `live: true` at `verbosity: "debug"` produces very high-volume output — only use for targeted debugging sessions
+ - `logModelIO: true` logs full system prompts — ensure logs are stored securely, as they may contain sensitive system prompt content
+ - JSONL file output appends to existing files — rotate or clear the file between runs in long-running test suites
+ - `.withAudit()` alone does not produce console output — combine with `.withObservability()` to see audit events
+ - At `verbosity: "minimal"`, even errors are not printed to console — check the returned `AgentResult` for failure details