149 added, 59 removed. Audit A to A.
---
name: kelly-agent-observability
- description: "Agent Fleet Observability Desk App-in-Skill — local mock dashboard visualizing a fleet of LLM agents running behind a shared AI gateway for your organization. Use when the user invokes $kelly-agent-observability or /kelly-agent-observability, wants to review agent health, call volume, latency, error rates, cost, or trace-level chain breaks across an agent fleet. This is a demo/reference dashboard over generated mock telemetry, not a live monitoring integration."
+ description: Busabase App-in-Skill dashboard visualizing a generated MOCK fleet of LLM agents running behind a shared AI gateway for a generic organization. Use when the user invokes $kelly-agent-observability or /kelly-agent-observability, wants to review agent health, call volume, latency, error rates, cost, or trace-level chain breaks across an agent fleet. This is a demo/reference dashboard over a generated mock fleet snapshot seeded into Busabase by a trusted script — never a live monitoring integration, never a real gateway or real agents.
---
# Agent Fleet Observability Desk
## Overview
- Use this skill as a local operator surface over a MOCK fleet of LLM agents
- running behind a shared AI gateway for a generic organization. It aggregates
- per-agent call volume, latency (p50/p95), error/failure rate, and cost into one
- file-backed App-in-Skill dashboard with a Fleet Overview, a sortable Agent
- Health table, and a Trace Detail view that highlights exactly where a chain of
- tool calls broke.
+ Kelly Agent Observability is a Busabase Cloud App-in-Skill. Its canonical
+ product surface is the AirApp in Busabase, not a separate local-data
+ product. The same Hono source supports an explicitly requested local preview
+ with OAuth connection bootstrap. It gives an operator a read-mostly
+ dashboard over a MOCK fleet of LLM agents running behind a shared AI gateway
+ for a generic organization: per-agent call volume, latency (p50/p95),
+ error/failure rate, cost, an hourly call-volume sparkline, and a Trace Detail
+ view that highlights exactly where a chain of tool calls broke.
- Default interaction mode: App UI. Unless the user explicitly asks for chat-only
- handling, seed mock telemetry if missing, start/reuse the local app with
- `app/start.sh`, and give the actual local URL. Use chat-only mode only when the
- user says "纯聊天", "chat only", "no UI", or similar.
+ This skill was never a live monitoring integration, even before this
+ conversion — there is no real gateway, no real agents, and no real telemetry
+ anywhere. The fleet snapshot (agents + metrics + traces) is generated by a
+ deterministic, seeded function and written into Busabase by a trusted
+ skill-root script (`scripts/generate_fleet_data.mjs`); the AirApp itself only
+ ever **reads** whatever was last generated there. The only human action is
+ narrow: append an "acknowledge" / "needs investigation" handoff note to an
+ agent or trace — always a brand-new row in the `handoffs` Base, never a field
+ update on the agent/trace's own record.
- This is a demo/reference dashboard: all data is generated locally by
- `scripts/generate_fleet_data.ts`. There is no real gateway, no real agents, and
- no external network calls anywhere in this skill.
+ Default behavior is AirApp-first. Unless the user explicitly asks only for
+ explanation, ensure the mock fleet exists (run the seed script below if the
+ `agents` Base is empty) and give the user the clickable AirApp URL. Start
+ localhost only when local preview/debugging is explicitly requested; it uses
+ the same Busabase resources and never offers another data provider. Use
+ chat-only mode only when the user says "纯聊天", "chat only", "no UI", or
+ similar.
+ This is deliberately **generic and brand-free**: agent names are generic
+ archetypes (Booking Assistant, Support Triage, etc.) for a generic
+ organization ("your organization", "the fleet") — no real company or product
+ appears anywhere in the code, config, or seed data.
+
+ ## Mandatory Dependencies
+
+ 1. Read and follow `$kelly-app-skill-creator` for product behavior, visual
+ quality, responsive layout, and the complete canonical `app/` artifact.
+ 2. Read and follow `$busabase` for connection, target Space, node discovery,
+ ChangeRequests, review, and merge behavior.
+ 3. Read and follow `$busabase-app-creator` for resource modeling, AirApp
+ runtime limits, security, validation, and deployment.
+
+ If a dependency is unavailable, preserve this skill's artifact and product
+ contracts, stop before the unavailable Busabase operation, and report the
+ exact missing dependency. Do not invent a second data backend.
+
## App UI Screenshots
<table>
<tr>
<td width="50%"><img src="assets/screenshots/overview.webp" alt="Fleet overview"></td>
<td width="50%"><img src="assets/screenshots/agent-health.webp" alt="Agent health table"></td>
</tr>
<tr>
<td><strong>Fleet Overview</strong><br>Total calls, total cost, degraded/critical/healthy agent counts, and a per-agent sparkline card grid.</td>
<td><strong>Agent Health</strong><br>Sortable table with call volume, p50/p95 latency, error rate, cost, and a status badge per agent.</td>
</tr>
<tr>
<td width="50%"><img src="assets/screenshots/trace-detail.webp" alt="Trace detail with chain break"></td>
<td width="50%"><img src="assets/screenshots/overview.zh-CN.webp" alt="Fleet overview in Chinese"></td>
</tr>
<tr>
<td><strong>Trace Detail</strong><br>Ordered step timeline for one trace; the step where the chain broke is visually flagged.</td>
<td><strong>Chinese UI</strong><br>Full zh-CN chrome via the language toggle or <code>lang=zh</code>.</td>
</tr>
</table>
## Boundary
- - Local, read-only mock dashboard. The app reads and writes local files only.
- - The only mutating endpoint is `POST /api/handoffs`, which appends a
- human-in-the-loop acknowledgement or investigation note to a local
- `app/.data/handoffs.jsonl` file. It never calls any remote system, agent
- gateway, or third-party API.
+ - Read-mostly dashboard over a generated mock fleet. NEVER contact any real
+ agent gateway, LLM provider, or third-party API. There is no live
+ monitoring integration in this skill by design.
+ - Agents and traces are never created or edited by the AirApp; they enter
+ Busabase only through the trusted `scripts/generate_fleet_data.mjs` seed
+ script, the same way `kelly-portfolio-health`'s contracts enter through an
+ external sync process the app doesn't control.
+ - The only Base the AirApp itself ever writes to is `handoffs`, and only ever
+ by creating a brand-new row (an acknowledge / needs-investigation note) —
+ never a field update on an agent or trace record.
- Never reference any real company, brand, or production system. Agent names
- are generic archetypes (Booking Assistant, Support Triage, etc.) for a
- generic organization ("your organization", "the fleet").
+ are generic archetypes for a generic organization ("your organization",
+ "the fleet").
- ## Local App
+ ## Busabase Resources
- Start the dashboard with:
+ Four Bases under one application Folder (`kelly-agent-observability`),
+ declared in `app/app/js/config.js` and `app/resource-map.json`:
- ```bash
- skills/kelly-agent-observability/app/start.sh
- ```
+ - `agents`: one row per mock agent archetype (8 rows) — identity plus the
+ latest rolled-up health metrics (status, call volume, p50/p95 latency,
+ error rate, cost) and a 48h hourly call/error series (JSON array). Written
+ only by the trusted generator script.
+ - `traces`: one row per mock trace (ordered tool-call chain), capped to fit
+ under the 100-record read limit (8 agents × 10 traces = 80 by default).
+ `steps` (JSON array) carries the ordered timeline; `broke-at-step-id` flags
+ exactly where a broken chain stopped. Written only by the trusted generator
+ script.
+ - `handoffs`: append-only human-in-the-loop log — an acknowledge /
+ needs-investigation note against one agent or trace. The only Base the
+ AirApp itself ever writes to.
+ - `settings`: sanitized fleet-generation metadata (schema version, seed,
+ generated-at), one row keyed by `kind`.
- First run installs `hono` and `@hono/node-server`, then seeds mock telemetry
- into `app/.data/fleet.json` if it doesn't already exist. The frontend is
- zero-build vanilla JS/HTML/CSS. The app uses local HTTP on `127.0.0.1`,
- preferring port `3100` through `4100`, or `KELLY_AGENT_OBS_UI_PORT` when set.
+ Resources provision lazily through an idempotent Busabase ChangeRequest the
+ first time the app runs in a Space; see `references/fleet-schema.md` for
+ exact field shapes.
- To regenerate mock telemetry at any time:
+ ## First Run And Onboarding
+ On invocation, check the `agents` Base. If it's empty, run the trusted seed
+ script to generate the mock fleet:
+
```bash
- node skills/kelly-agent-observability/scripts/generate_fleet_data.ts
+ node skills/kelly-agent-observability/scripts/generate_fleet_data.mjs --apply
```
+ There are no credentials to collect — this skill never calls an external
+ system, so onboarding is just running the seed script once.
+
+ ## Local App
+
+ Default behavior is AirApp-first — give the user the clickable AirApp URL.
+ Start `pnpm --dir app dev` only when local preview/debugging is explicitly
+ requested.
+
+ Required app views (hash routes):
+
+ - `#/overview`: total calls (24h), total cost (today), degraded/critical/
+ healthy agent counts, and a per-agent card grid with a call-volume
+ sparkline.
+ - `#/agents`: sortable Agent Health table (call volume, p50/p95 latency,
+ error rate, cost, status badge).
+ - `#/agents/<agent_id>`: agent detail — metrics, sparkline, recent traces,
+ and a handoff form (acknowledge / needs investigation).
+ - `#/traces/<trace_id>`: trace detail — ordered step timeline with the
+ chain-break step visually flagged, and a handoff form.
+ - `#/handoffs`: history of recorded handoffs.
+ - `#/settings`: data provider and status-threshold summary.
+
## Demo Mode
- `?demo=1` opens a deterministic, fully offline mock fleet (8 agents, 48h of
- hourly buckets, ~16 traces per agent) for documentation and screenshots,
- without touching `app/.data/fleet.json`.
- - `?demo=agents` and `?demo=trace` select named mock scenes (agent health table,
- a broken trace).
+ hourly buckets, 16 traces per agent) for documentation and screenshots. It
+ never reads or writes Busabase and never claims a real connection; demo
+ handoff submissions only update the in-memory list already rendered.
+ - `?demo=agents` and `?demo=trace` select named mock scenes (agent health
+ table, a broken trace).
- `lang=en` or `lang=zh` forces UI chrome language for screenshots.
- - Demo API responses never read or write local files.
UI language: support English and Chinese chrome with `Auto` default.
- ## Data Model
+ ## Fleet Generation Model
- Read `references/fleet-schema.md` before editing the app, scripts, or the
- generator. Primary local files:
+ `app/app/js/fleet-model.js` (`generateFleetData`/`summarizeFleet`) is ported
+ verbatim from the retired `lib/generate.ts`:
- - `app/.data/fleet.json`: generated mock telemetry (agents, per-agent metrics,
- hourly buckets, traces). Regenerated by `scripts/generate_fleet_data.ts`.
- - `app/.data/handoffs.jsonl`: append-only human-in-the-loop handoff log written
- by `POST /api/handoffs`.
+ - **Per-agent profile** — a fixed "personality" (volume, latency base/jitter,
+ error tendency, cost per call) per agent archetype, so the fleet reads as
+ varied, not random noise.
+ - **Hourly buckets** — 48 hours of calls/errors per agent on a
+ business-hours-ish curve, seeded per agent.
+ - **Status** — `statusFor(error_rate_pct, p95_latency_ms)`: `critical` at
+ error rate ≥ 8% or p95 ≥ 8000ms, `degraded` at ≥ 3% or ≥ 4000ms, else
+ `healthy`.
+ - **Traces** — an ordered tool-call chain per trace; a trace can "break" mid
+ chain (`status: error`, `broke_at_step_id` set to the last step), with the
+ break probability tied to the agent's error tendency.
- ## Views
+ Same seed + same "now" + same `tracesPerAgent` always produces bit-identical
+ output — this is the property `scripts/generate_fleet_data.mjs` and the demo
+ provider both rely on.
- - `#/overview`: total calls (24h), total cost (today), degraded/critical/
- healthy agent counts, and a per-agent card grid with a call-volume sparkline.
- - `#/agents`: sortable Agent Health table (call volume, p50/p95 latency, error
- rate, cost, status badge).
- - `#/agents/<agent_id>`: agent detail — metrics, sparkline, recent traces, and
- a handoff form (acknowledge / needs investigation).
- - `#/traces/<trace_id>`: trace detail — ordered step timeline with the
- chain-break step visually flagged, and a handoff form.
- - `#/handoffs`: history of recorded handoffs.
- - `#/settings`: data provider and status-threshold summary.
+ ## Handoffs
+ The one human action — acknowledge or flag "needs investigation" on an agent
+ or trace, with a free-text note — writes a brand-new row into the `handoffs`
+ Base via `busabase-sdk`, never a field update on the agent/trace record
+ itself:
+
+ - From a standalone local preview the write merges immediately (trusted
+ operator).
+ - From the deployed AirApp it creates a pending ChangeRequest for the
+ trusted process to merge, per the AirApp boundary in
+ `$busabase-app-creator`.
+
## Safety
- - Never invent a real incident or real company data; this is illustrative mock
- telemetry only.
- - Keep all writes local and file-based (`app/.data/handoffs.jsonl`). No
- external network calls anywhere in this skill.
+ - Never invent a real incident or real company data; this is illustrative
+ mock telemetry only.
+ - Keep all writes scoped to the `handoffs` Base. No external network calls
+ anywhere in this skill.
- Redact nothing sensitive is collected here — there are no credentials in
this skill.
+
+ ## Useful Commands
+
+ ```bash
+ node skills/kelly-agent-observability/scripts/generate_fleet_data.mjs --apply
+ pnpm --dir skills/kelly-agent-observability/app dev
+ ```
+
## Execution reports
Re-read the active provider's decisions immediately before any approved execution. Record each concrete operation, target, status, timestamp, and error in the provider-backed execution report; keep app actions local-only.