AGENTS.md · diff
git:20260616.8e7c931 to git:20260924.4305938
204 added, 34 removed. Audit A to A.
# AGENTS.md
- This repository contains AI agent skills for building with LiveKit. The skills follow the [Agent Skills](https://agentskills.io) format.
+ This repository publishes [Agent Skills](https://agentskills.io) that teach coding agents to build,
+ test, and ship voice agents with LiveKit. This file is for agents working *in this repo*, where the
+ job is writing and evaluating skills rather than building a LiveKit agent.
+ [README.md](README.md) is for the people who install the skills.
- ## Repository Structure
+ ## Repository map
```
agent-skills/
- ├── README.md # User-facing documentation
- ├── AGENTS.md # This file (for AI agents)
- ├── CLAUDE.md # Points to AGENTS.md
- └── skills/
- ├── livekit-agents/ # LiveKit Agents skill
- │ ├── SKILL.md # The skill content
- │ └── references/ # Supporting documentation
- └── livekit-simulations/ # LiveKit Simulations skill
- ├── SKILL.md # The skill content
- ├── references/ # Supporting documentation
- └── scripts/ # Supporting scripts
+ ├── README.md # for users: what the skills do, how to install
+ ├── AGENTS.md # for agents maintaining the repo (this file)
+ ├── CONTRIBUTING.md # for human contributors: process
+ ├── skills/ # the published skills; this layout is the install contract
+ │ ├── reading-livekit-docs/
+ │ ├── building-livekit-agents/
+ │ ├── debugging-livekit-agents/
+ │ ├── testing-livekit-agents/
+ │ ├── writing-livekit-scenarios/
+ │ │ └── references/ # scenario craft, risk coverage, connecting the agent
+ │ ├── running-livekit-simulations/
+ │ └── operating-livekit-agents/
+ └── evals/ # tooling for checking the skills — never under skills/
+ ├── validate.py # structural checks on every SKILL.md
+ ├── trigger/ # does the right skill fire for a request?
+ └── output/ # does following a skill produce better work?
```
- ## Contributing Guidelines
+ ## How the skill set is meant to work
- ### Skill Content Principles
+ Each skill owns one job and names its siblings for adjacent jobs. Routing happens in the
+ `description` fields, which are the only part of a skill loaded before it fires. A request should
+ land on exactly one skill, and that skill's body should cover everything needed for its job.
- All skills in this repository must follow the "freeze forever" principle:
+ | Skill | Owns | Hands off to |
+ |---|---|---|
+ | reading-livekit-docs | Looking up any LiveKit fact | — (every other skill loads it first) |
+ | building-livekit-agents | Architecture and voice-specific design | debugging, testing |
+ | debugging-livekit-agents | Live-testing during development (`lk agent debugger`) | testing (to pin a bug), running (for speech issues) |
+ | testing-livekit-agents | Turn-level tests in pytest/Vitest | debugging (to find the bug), running (whole conversations) |
+ | writing-livekit-scenarios | Authoring scenarios, and the agent-side code that consumes them | running |
+ | running-livekit-simulations | Running simulations, CI, triage | writing (bad scenario), testing (repeat failure) |
+ | operating-livekit-agents | Deploying to LiveKit Cloud and running in production: worker model, prewarm, shutdown, upgrades, observability | debugging (reproduce), testing (pin), running (verify a deployed agent) |
- 1. **Encode behavior, not knowledge** - Skills teach *how to approach* problems, not API specifics
- 2. **Direct to MCP for facts** - All factual information must come from live documentation
- 3. **Require testing** - Every agent implementation must include tests
- 4. **Stay under 500 lines** - Keep skills concise for context efficiency
+ A bare "test my agent" goes to **debugging**. That's the cheap, local path; running simulations
+ spends cloud resources.
- ### SKILL.md Format
+ ## Authoring rules
- Each skill requires YAML frontmatter:
+ The skills need to stay correct without maintenance, and each rule below exists because this repo
+ broke it at least once.
+ **Encode behavior, not knowledge.** A skill teaches how to approach a job (what to check, what goes
+ wrong, which tool fits), not the API surface. You can include one simple example command with a
+ `# see --help` comment. Don't include a flag roster, a schema field list, a list of built-in
+ helpers, a JSON output shape, version numbers, minimum versions, or "as of writing". Those go stale
+ within months. Instead, the skill should say it doesn't restate them and point at `--help` and
+ `reading-livekit-docs`. When you're about to type a backticked identifier, check whether the
+ sentence would still work as a concept.
+
+ **No time-relative phrasing.** Phrases like "newer CLIs", "is landing", "still moving", and
+ "recently" stop being true. Write capability checks instead: "where the CLI supports X, …", "if the
+ command offers to Y, accept."
+
+ **Stay atomic.** One job per skill. Cross-link by name instead of restating. Two skills that both
+ explain simulations compete for the same triggers, and both get weaker.
+
+ **Descriptions are written as a set.** The description is the entire trigger mechanism. Each one
+ needs:
+
+ - third person ("Runs simulations…", never "Run simulations…" or "I can run…"; the platform docs
+ warn that other forms break discovery)
+ - the phrases a user types for *that* job
+ - the sibling to use for adjacent jobs
+ - a negative clause where two skills border each other ("Not the default for a bare 'test my
+ agent', which goes to debugging-livekit-agents")
+ - at most 1,024 characters
+
+ Lean toward pushy, since models tend to under-trigger skills. When you change one description,
+ re-read the others, then run the trigger eval.
+
+ **Names are gerunds**: `building-livekit-agents`, `running-livekit-simulations`. Lowercase, hyphens,
+ under 64 characters, no "claude" or "anthropic". The platform docs recommend this form, but
+ consistency across the set matters more.
+
+ **Bodies stay under 500 lines; references are one level deep.** Put anything long or conditional in
+ `references/`, linked directly from SKILL.md with a sentence saying when to read it. A reference
+ over 100 lines starts with a `## Contents` list so a partial read still shows its scope.
+
+ **Frontmatter:**
+
```yaml
---
- name: skill-name-kebab-case
- description: Trigger phrases and brief description. Include phrases like "build a voice agent" that help agents recognize when to use this skill.
+ name: running-livekit-simulations
+ description: 'Runs LiveKit agent simulations and acts on the results. Use when the user says "run my simulations", … For authoring the scenarios use writing-livekit-scenarios.'
license: MIT
metadata:
author: livekit
- version: "X.Y.Z"
+ version: "1.0.0"
---
```
- ### File Naming
+ **Say "self-hosted" only about the agent.** A self-hosted agent runs on the user's own servers
+ instead of LiveKit Cloud's agent hosting, and it can still use LiveKit Cloud features such as
+ LiveKit Inference. When the user runs the LiveKit server themselves, call it LiveKit OSS, as
+ opposed to LiveKit Cloud. Using "self-hosted" for both leads agents to wrongly rule out Cloud
+ features for anyone who hosts their own agent.
- - Skill directories use `kebab-case`
- - `SKILL.md` is the only uppercase filename
- - Supporting docs go in `references/` subdirectory
+ **Explain why.** Capable models read these skills, and they respond better to a reason than to a
+ bare command. "Restart after every edit — a running session holds the old code" works better than
+ "ALWAYS restart". If you find yourself writing a capitalized MUST, replace it with the reason.
- ## LiveKit MCP Server
+ ## Verifying facts while you write
- These skills are designed to work with the LiveKit Docs MCP server. If you're working on this repository and need to test MCP integration, make sure it is installed. Installation instructions are available at https://docs.livekit.io/intro/mcp-server/
+ Even a conceptual skill depends on facts: that a subcommand exists, that a callback can only fail a
+ run, that generation uploads source. Verify each one during the session, in this order:
+ 1. **`--help` on the installed CLI.** This outranks every document.
+ 2. **`lk docs` or the Docs MCP server** for the published behavior.
+ 3. **The public repos for unreleased work**: [livekit/livekit-cli](https://github.com/livekit/livekit-cli)
+ and [livekit/agents](https://github.com/livekit/agents), including open PRs. The CLI and SDK
+ often ship ahead of the docs and behind the source. Skills written from the live docs alone have
+ been wrong on arrival here before.
+ 4. **The examples in livekit/agents** for how things are used in practice (the front-desk and hotel
+ receptionist examples are the canonical scenario files).
+ Then write down the *shape* of what you verified, not the specific surface. If a fact only holds
+ for one version, it doesn't belong in a skill.
+
+ ## Evaluating skills
+
+ Run these before opening a PR. The first is always required. The second is required when any
+ `description` changed. The third is for new or substantially rewritten skills.
+
+ ### 1. Structure — `python3 evals/validate.py`
+
+ Checks every `skills/*/SKILL.md` for: frontmatter fields, name matching the directory, description
+ length and third person, body length, references that exist and have a TOC when long, no dangling
+ cross-references to skills that don't exist, and no time-relative phrasing. Exits non-zero on any
+ failure. It's cheap, so run it often.
+
+ ### 2. Triggering — `evals/trigger/`
+
+ This checks whether the right skill fires, which is what atomic skills most often get wrong. The
+ harness installs the whole set into a scratch copy of a testbed project, runs each query in
+ `queries.json` through `claude -p`, and records which skill was invoked first. Each query is
+ labeled with the acceptable skill(s), or with none for near-misses that shouldn't trigger anything.
+ Output is a per-group confusion table and a list of misses.
+
+ ```bash
+ python3 evals/trigger/run.py # fixture: agent-starter-python, cloned fresh
+ python3 evals/trigger/run.py --template node # the skills serve both languages; run both
+ ```
+
+ See `evals/trigger/README.md` for options (subset by index, runs per query, model). When you find a
+ request that routed wrong, add it as a query. Near-miss cases are the most valuable ones.
+
+ **Fixtures and the user's LiveKit project.** Evals build their fixture by shallow-cloning a public
+ `livekit-examples` starter. They don't use `lk agent init` or `lk app create`, because those resolve
+ a LiveKit Cloud project first and write its credentials into the directory. The harness strips
+ `.env*` files and injects dummy `LIVEKIT_*` variables into every subprocess. `lk` and the SDKs
+ honour those over the user's configured default project, so an eval can't spend inference or upload
+ source against whichever project the user last selected. The model making the decisions in every
+ eval is the user's Claude. LiveKit Inference is only involved if an agent executes the agent under
+ test, and the guards prevent that.
+
+ Trigger runs only see the repo's skills. That's what you want for measuring collisions within the
+ set, but users will have other skills installed, so it's worth occasionally doing a run with a
+ normal setup.
+
+ ### 3. Output — `evals/output/`
+
+ This checks whether following the skill produces better work than not following it. Run the same
+ realistic prompt twice, once with the skill path given to a fresh agent and once without, against a
+ fixture built the same way as the trigger harness's (a credential-stripped clone of a starter
+ template). Then grade the outputs.
+
+ `evals/output/grade_scenarios.py <outputs> --agent <src/agent.py|src/agent.ts>` grades scenario
+ files in two layers.
+
+ - **Structural** checks are exact: YAML parses, keys match the CLI's scenario struct, required
+ fields, group names, quick/full split.
+ - **Judged** checks are one `claude -p` call with no tools. It reads the agent's source and the
+ scenarios and returns per-assertion verdicts with quoted evidence. The assertions: every
+ constraint in the agent's instructions is exercised, refusals count as the pass, no invented
+ capabilities, no dates that will go stale, expectations are decidable, and the simulated user
+ never plays the agent.
+
+ These questions need judgment, so they get a judge. An earlier regex version gave false negatives
+ on good work and was replaced. Verdicts vary a little between runs, so use `--judge-runs 3` (an odd count, so there is always a majority)
+ and read the evidence instead of counting passes. `evals/output/README.md` has the details and the
+ safety rules for output-eval agents, which need write tools.
+
+ Two lessons from past runs:
+
+ - **Testbeds confound.** Both starter templates ship a `scenarios.yaml` that already uses the
+ instructions template the skill teaches. A baseline run copies it and looks nearly as good as the
+ skill run. Check what the fixture already contains before concluding the skill adds nothing.
+ - **Never let an eval agent run `lk agent simulate`** without a scenario file. It uploads the
+ fixture's source to LiveKit Cloud and spends inference. Eval prompts must say so, and the dummy
+ `LIVEKIT_*` environment must be set for any agent with shell access.
+
+ ### Definition of done for a skill change
+
+ - `validate.py` passes.
+ - Descriptions changed → trigger eval run, no new misses, and the run's summary pasted in the PR.
+ - New or rewritten skill → an output eval with at least three prompts, results summarized in the PR.
+ - Nothing in the body would be wrong if the CLI added a flag or renamed a helper tomorrow.
+
+ ## Removing or renaming a skill
+
+ Delete the directory. To rename, create the new directory and delete the old one. Don't leave a
+ stub behind: a stub's description is loaded into every user's context on every request, forever,
+ to say "don't use me" — the opposite of what the set is trying to do for context.
+
+ Skills are installed from this repository — `npx skills add livekit/agent-skills` today, and the
+ LiveKit CLI. Installers take `skills/<name>/SKILL.md` plus that skill's `references/`, so keep that
+ layout stable: a renamed directory is a removed skill and a new one, and nothing here can reach a
+ user's existing local copy. The README tells users of a removed skill to reinstall.
+
+ ## Things not to do here
+
+ - Don't put evals, tooling, or scratch files under `skills/`. Everything there is published; installers take the whole directory.
+ - Don't add a skill because a topic exists. Add one when agents do that job badly without it
+ and you can say what the skill changes.
+ - Don't tune a description to fix one failing query. Figure out what class of request it represents.
+
## Links
- - [LiveKit Documentation](https://docs.livekit.io)
- - [LiveKit Agents SDK](https://github.com/livekit/agents)
- - [Agent Skills Format](https://agentskills.io)
+ - [Agent Skills format](https://agentskills.io) · [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices)
+ - [LiveKit docs](https://docs.livekit.io) · [Docs MCP server](https://docs.livekit.io/intro/mcp-server/)
+ - [livekit/agents](https://github.com/livekit/agents) · [livekit/livekit-cli](https://github.com/livekit/livekit-cli)