subwave-llm-bench · diff
git:20261013.42e5382 to git:20261015.d7e9c73
5 added, 3 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: subwave-llm-bench
description: >-
Benchmark and compare LLM models for SUB/WAVE's on-air calls — track picks,
segments, listener requests, DJ scripts, banter, and programme beats — in
- both candidate-pool and agent modes, using controller/scripts/llm-bench (the
+ pool, agent and Track Shortlist modes, using controller/scripts/llm-bench (the
matrix harness) or the legacy picker-only controller/scripts/picker-test.mjs.
Use this skill whenever the user wants to assess, benchmark, compare, or test
which LLM model to run the station on — phrases like "which model should I
use", "benchmark the picker / the DJ / this model", "test these models", "is
<model> good enough for the radio", "compare the ollama models", "will a
small model work", "run llm-bench", "run picker-test", or when diagnosing
slow/failing djAgentPick / djAgentSegment / generate* calls and the model
choice is suspect. Trigger it even if the user doesn't name a harness — any
request to evaluate model reliability or choose a model for SUB/WAVE belongs
here. This skill only measures and recommends; it does NOT change the live
station's configured model.
---
# SUB/WAVE LLM model benchmark
Two harnesses, one job: measure how well a **provider + model** handles the
station's real LLM calls before trusting it on air.
- **`llm-bench`** (`controller/scripts/llm-bench/`, `npm run llm-bench`) — the
primary harness. A matrix runner over **every on-air call kind**: pool picks
(`pickNextTrack`), agent picks (`djAgentPick`), Track Shortlist picks
(`djShortlistPick`), pool + agent segments
(`generateSegment` / `djAgentSegment`), request matching (`matchRequest` /
`djAgentRequest`), the free-text scripts (intro, link, station ID, hourly),
banter, and the programme family (plan, beats, exchanges). Scores
reliability + deterministic rule checks, prints a per-model comparison
table, writes a diffable JSON report.
- **`picker-test.mjs`** — the legacy picker-only deep-dive. Still useful for
high-iteration picker runs comparable with historical results, and it has a
bundled orchestration script (see the legacy section at the end).
Both import the **live prompts and schemas from `src/`** (never copies), fake
everything external (library, tools, weather/news data), and override
provider/model **only inside their own short-lived process** — the live
controller's configured model is never touched.
## The two things that surprise people
1. **Routing matters as much as the model.** The *same* model can pass through
one provider and fail through another, because each `@ai-sdk/*` provider
translates tools / structured output differently. Canonical case:
`deepseek-v4-flash` scored **0/4 via the `deepseek` direct provider** but
**4/4 via `openrouter`**. Always benchmark through the routing you'll
actually deploy — "is model X good?" is the wrong question; "is
provider+X good?" is the right one.
2. **The stress scenarios are where truth lives.** Easy cells make every model
look fine. The verdict cells are: `djAgentPick/long-context` (full prompt +
tool loop), `generateSegment/dull-weather` (the model must *decline* to air
— small models botch the silence encoding; a huge wall-clock here means the
structured-output retry rescued a failed first attempt), and the
multi-voice JSON kinds (`generateBanter`, `generateProgrammeExchange`).
## When NOT to use this
- Changing the live model — that's an admin Settings change, not this skill.
- Diagnosing general runtime behaviour — `subwave-log-analysis`.
- Starting/stopping the stack — `subwave-control`.
## Running llm-bench
Runs on the **host from a repo clone** (it imports TS source via `tsx`; prod
containers ship only compiled `dist/`). The running stack is not required —
keys resolve from `state/secrets.env` and `controller/.env` automatically
(existing env vars win). If a key exists only inside a prod container's env,
copy it out first: `export OPENROUTER_API_KEY=$(docker exec sub-wave-controller printenv OPENROUTER_API_KEY)`.
```bash
cd <repo>/controller
npm run llm-bench -- \
--models openrouter:google/gemma-4-31b-it,ollama:qwen3:8b \
--iterations 3
```
Flags:
- `--models` (required) — comma list of `provider:model`, split on the FIRST
colon, so Ollama tags keep theirs (`ollama:qwen3:8b`). Providers: ollama |
openai-compatible | anthropic | openai | google | deepseek | openrouter |
requesty | gateway.
- `--kinds` — groups (`pick,segment,request,scripts,banter,programme`) or
exact kind names. Default: all.
- `--modes` — `pool,agent,shortlist` (default all three). Mode-independent
kinds run once either way. `shortlist` adds only the `djShortlistPick` kind:
the live source plan run over the synthetic tools, then the one bounded
choice. Match the modes to the station's routes (`llm.trackSelection`,
`requestMatching`, `segmentRuntime`): **comparing the two pickers is
`--kinds pick --modes agent,shortlist`**, and cells for a route the operator
never uses just add noise and cost.
- `--iterations` — per scenario. 1 to smoke, 3 to screen (default), 5+ to
confirm a winner. Full matrix ≈ 32 scenario cells, so ~100 calls per model
at 3 iterations.
- `--out` — report path (default `scripts/llm-bench/reports/<ts>.json`,
gitignored).
A full run takes many minutes (every call is a real model call, agent cells
can take 45 s each). Run it in the background with a long timeout and don't
poll; `OLLAMA_URL=http://localhost:11434` translates the container-internal
Ollama address for host runs.
## Reading llm-bench results
Each run is `ok`, `violation` (call succeeded, named rule checks failed), or
`thrown` (bucketed: `no-object-generated`, `timeout`, `unreachable`,
`thrown`). The summary prints pass% + p50 latency per cell, then per-model
histograms of rule failures and thrown buckets.
Judge in this order:
1. **Thrown rate** — a model that can't produce the schema at all is out.
`no-object-generated` on structured kinds = weak structured output;
everything landing at ~45 s = too slow for the loop, not incapable.
`unreachable` = provider/config problem, not the model — fix and re-run.
2. **Violations** — the named rules tell you the failure *character*:
`hallucinated-id` (invents track ids — dangerous), `banned-phrase:*` /
`stage-direction:*` / `wrapping-quotes` (would be read aloud by TTS),
`digits-in-spoken-time` / `clock-leak` (hourly/link discipline),
`opener-repeat` (ignores anti-repeat lists), `unusable-exchange` /
`single-voice` (can't hold multi-voice JSON), `variety:same-artist`
(editorial-pressure miss — informative, not disqualifying).
3. **Latency** — p50 per cell; a pick has a whole track of slack so even 20 s
works, but segments and links air sooner. Watch for retry-rescued cells
(pass with 100×-median wall clock).
4. **Shortlist/pool vs agent gap** — a model that's clean in shortlist or
pool cells but falls apart in agent cells is a Track Shortlist model;
that's a valid deployment, just say so in the recommendation. Shortlist
cells also flag `reason-unavailable` / `reason-rejected` (the Booth reason
was missing or failed the controller's safety check) and `near-miss-id`
(an id slip the live `nearestId` repair would save, unlike
`hallucinated-id`). Compare tokens per pick from the report JSON: one
Shortlist call carries no tool loop or session window.
Reports are JSON with every run record — keep old ones to diff a model across
prompt changes or across model updates.
## Producing the recommendation
1. A compact table: one row per provider+model, columns for thrown rate,
violation rate, p50 latency, and the dominant failure.
2. A clear call naming the best **provider + model + mode** ("gemma via
- OpenRouter, pool mode — clean structured output, agent cells time out").
+ OpenRouter, Track Shortlist — clean structured output, agent cells time
+ out"). Pool cells exercise the fallback picker every station keeps; they
+ are not a selectable route, so recommend Shortlist or Agentic Tools.
3. Call out disqualifiers and deferrals (5xx throughout = re-test later).
Do **not** apply the change. The operator sets `llm.provider` + `llm.model`
- (and the Agentic picker toggle) in admin Settings — those are global, their
+ (and the Music selection route, Agentic Tools or Track Shortlist) in admin Settings — those are global, their
call.
## Legacy: picker-test.mjs deep-dive
For picker-only, high-iteration runs comparable with historical results:
```bash
# DEV stack (src bind-mounted in container):
docker exec sub-wave-controller npx tsx scripts/picker-test.mjs <provider> <model> [iterations] [short|long]
# Host (works regardless of stack):
( cd <repo>/controller && STATE_DIR=<repo>/state OLLAMA_URL=http://localhost:11434 \
npx tsx scripts/picker-test.mjs <provider> <model> [iterations] [short|long] )
```
`long` is the verdict mode (realistic session window); `short` is a sanity
check. The bundled `scripts/assess-models.sh` (in this skill's directory) runs
one or more models in both modes, auto-detects dev/prod, and summarises event-
log failures: `assess-models.sh <provider> [iterations] <model>...` — with
`ollama` and no models it discovers and tests everything installed.
Failure strings glossary (from the event log, `kind: pickerTest`):
- `agent did not call the done tool before stopping` — model ignored the
tool protocol. Check routing before rejecting the model (see surprise #1).
- Every failure at ~`agentTimeoutMs` (45000) with `tools=0` — a latency
failure, not incapability; the deadline aborted it mid-loop.
- `Failed after N attempts … Internal Server Error` — provider outage;
re-run later, don't reject the model on this.
- `hallucinated-id` / `no-object-generated` — weak structured output.
Cross-check live behaviour in `/admin/debug` → `llm.recentCalls`: the `via`
field shows which path produced each result (`ai-sdk:agent:native` = clean
native path; `…:recovery` = rescued by the done-tool recovery; a thrown
`djAgentPick` falls back to the pool picker, so the station never goes
silent).
## Notes
- Both harnesses append telemetry rows to the controller event log
(`state/logs/events-*.jsonl`) and, if `LLM_DEBUG_RAW` is on, raw request
bodies to `state/logs/llm-debug.log`. Harmless, filterable.
- The docs page mapping every call kind to its code lives at
`docs/llm-calls.md`; the harness design spec at
`docs/superpowers/specs/2026-07-09-llm-bench-design.md`.