mivas-run · diff
git:20260911.7a442e0 to git:20260911.3f4ba64
91 added, 53 removed. Audit A to A.
---
name: mivas-run
description: >-
- Runs MIVAS Bench end to end from a consumer seat: prerequisites (Bluejay API key,
- provider key), a harness (existing or self-built), a Kubernetes deployment (local,
- EKS, Baseten, or any cluster), then a gated ladder of smoke tests, three Bluejay
- smoke calls, harness + benchmark integrity, and finally a full industry run.
- Use when someone says /mivas-run, "run MIVAS myself", "set up mivas-bench",
- "deploy a harness for MIVAS", "smoke my harness on Bluejay", "benchmark my voice
- model on healthcare/legal/customer-support", or asks how to plug their own voice
- agent into MIVAS.
- argument-hint: "family/runtime industry [target: local|eks|baseten|k8s]"
+ Walks a consumer through running MIVAS Bench end to end, interview first: what model
+ they are benchmarking, where the model is served (vendor API, or their own endpoint on
+ an inference provider), and where the Docker container runs (local, any Kubernetes, or
+ any container hosting platform). Then a gated ladder: preflight, deploy, three Bluejay
+ smoke calls, harness + benchmark integrity, full industry run. Use when someone says /mivas-run, "run MIVAS myself",
+ "set up mivas-bench", "deploy a harness for MIVAS", "smoke my harness on Bluejay",
+ "benchmark my voice model on healthcare/legal/customer-support", or asks how to plug
+ their own voice agent or their own model into MIVAS.
+ argument-hint: "[model or family/runtime] [industry] [where to host]"
---
# MIVAS run (consumer)
- Everything here uses the public Bluejay REST API with **your** `BLUEJAY_API_KEY`
- and **your** cluster. No Bluejay internals, no shared infrastructure.
+ Opinionated walkthrough. You interview the user, pick for them when they are unsure, and
+ refuse setups that cannot work. Everything uses the public Bluejay REST API with **their**
+ `BLUEJAY_API_KEY`, their inference, their hosting. Nothing internal to Bluejay.
- ## Prerequisites (stop if any is missing)
+ ## Step 0: interview. Do not run any command before it is done.
- 1. Bluejay account + API key (Bluejay app → API Keys) → `BLUEJAY_API_KEY` in `.env`.
- 2. Provider key for the harness (`OPENAI_API_KEY`, `GOOGLE_API_KEY`, …). Table: [HARNESS.md](HARNESS.md#existing-harnesses).
- 3. A harness you fully control: an existing `voice-agent-harnesses/<family>/<runtime>/`, or your own built per [HARNESS.md](HARNESS.md#build-your-own).
- 4. Somewhere to run it that Bluejay can dial over public `wss://`: [DEPLOY.md](DEPLOY.md) (local + tunnel, EKS, Baseten, any Kubernetes).
- 5. A snapshot store for per-call final state, or accept that state scoring is skipped: [DEPLOY.md → Storage](DEPLOY.md#storage).
+ Ask with AskUserQuestion, one question at a time, in this order. Bracketed text is the
+ default you pick if they say "you choose" or do not know. Never silently assume anything else.
- ```bash
- git clone https://github.com/bluejay-ai-dev/mivas-bench && cd mivas-bench && uv sync && cp .env.example .env
+ 1. **What model are you benchmarking?**
+ *(a)* a model already wired up — OpenAI Realtime 2.1 / 2.1-mini, GPT-Live 1, Gemini Live,
+ Nova Sonic 2, Grok Voice, Qwen Audio Realtime, or the LiveKit cascaded baseline
+ → vendor API. *(b)* **your own or an open-weight model** → it needs an inference endpoint
+ you stand up first; only the cascaded harnesses can point at one. **[openai/realtime-2.1]**
+ Map the answer to a `family/runtime` from [HARNESS.md](HARNESS.md); refuse a model with no harness
+ and offer to build one.
+ 2. **Where is that model served?** Vendor API (they hold a provider key) or their own
+ endpoint on an inference provider. Route to [INFERENCE.md](INFERENCE.md) either way; case (b)
+ above is a **separate setup step that must finish and be curl-verified before any hosting work**.
+ 3. **Where will the container run?** Ask before deciding anything else about hosting, and
+ take whatever they already use. MIVAS ships as a Docker image, so the only hard requirement
+ is a platform that can **run a Linux/amd64 container** and give it **one public TLS endpoint
+ that proxies WebSockets** to container port 8765. That includes their laptop, their own
+ Kubernetes cluster, a managed Kubernetes service, a container platform like Railway, Fly,
+ Render, Cloud Run or ECS, or a plain VM with Docker. **[Railway]** when they have no
+ preference — it returns a `wss://` URL with TLS and no DNS work. Confirm the platform meets
+ the two requirements in [HOSTING.md](HOSTING.md#what-any-host-must-provide); if it cannot, say which
+ requirement fails and offer the nearest option that works.
+ 4. **Which industry?** `control-industry` first, always, whatever they answer — it is the
+ wiring smoke. Then `healthcare`, `legal`, or `customer-support` for scored work. **[healthcare]**
+ 5. **How far this session?** Three smoke calls / smoke + integrity / all the way to a full
+ industry run. **[smoke + integrity]** Say the cost of a full run before they pick it:
+ 72 tasks × k calls, several hours, and it needs snapshot storage.
+
+ Then confirm the plan back in four lines — model, inference, hosting, industry — and check
+ they hold: a Bluejay API key, the provider key or endpoint, a container registry if hosting
+ is Railway or AWS, and a snapshot store. Missing Bluejay key or provider credential is a
+ **stop**; missing snapshot store only downgrades scoring, so say so and continue.
+
+ ## Two planes, set up in order
+
```
+ inference plane hosting plane Bluejay
+ where the MODEL runs where the CONTAINER runs
+ vendor API ──────────┐ any platform that runs
+ or your endpoint on ├──▶ a Docker container ──wss──▶ smoke calls → scoring
+ an inference provider┘ (one pair per service)
+ ```
- ## Ladder (each rung gates the next)
+ Never collapse them. The inference endpoint must be reachable **from the container**, so a
+ model on the user's laptop is only valid when the container is also local.
- Track it:
+ ## Ladder — each rung gates the next, no skipping
```
- - [ ] 0 preflight scripts/preflight.py (keys, pair on disk, --check, store)
- - [ ] 1 deploy DEPLOY.md target → pods Ready → preflight --k8s / --url (wss 101)
- - [ ] 2 smoke calls scripts/bluejay.py smoke → 3 calls, rubric PASS each
- - [ ] 3 integrity scripts/verify_integrity.py → INTEGRITY PASS
- - [ ] 4 full industry scripts/bluejay.py full → verify_task_run + CSV
+ - [ ] 0 preflight scripts/preflight.py keys, pair on disk, --check, store
+ - [ ] 1 inference INFERENCE.md vendor key, or endpoint + curl proof
+ - [ ] 2 hosting HOSTING.md container up, public wss, 101 probe
+ - [ ] 3 smoke calls scripts/bluejay.py smoke 3 calls, six-check rubric each
+ - [ ] 4 integrity scripts/verify_integrity.py INTEGRITY PASS
+ - [ ] 5 full industry scripts/bluejay.py full verify_task_run + CSV
```
- Commands (from the repo root; `H=family/runtime`, `I=industry`):
+ From the repo root, with `H=family/runtime` and `I=industry`:
```bash
uv run python .agents/skills/mivas-run/scripts/preflight.py --harness $H --industry $I
- uv run python run.py --harness $H --industry $I --build --apply --no-logs # or per DEPLOY.md
- uv run python .agents/skills/mivas-run/scripts/preflight.py --harness $H --industry $I --k8s # or --url wss://…
+ # rungs 1-2 per INFERENCE.md then HOSTING.md, ending in a public wss:// URL
+ uv run python .agents/skills/mivas-run/scripts/preflight.py --harness $H --industry $I --url wss://HOST
uv run python .agents/skills/mivas-run/scripts/bluejay.py smoke --harness $H --industry $I --url wss://HOST
- uv run python .agents/skills/mivas-run/scripts/verify_integrity.py --harness $H --industry $I --smoke-dir verify-out/smoke/<slug>/<run> --k8s
+ uv run python .agents/skills/mivas-run/scripts/verify_integrity.py --harness $H --industry $I --smoke-dir verify-out/smoke/<slug>/<run>
uv run python .agents/skills/mivas-run/scripts/bluejay.py full --harness $H --industry healthcare --runs 5
```
## Rules
- - **Rung 0 first, always.** Never `--apply` with a missing provider key or `BLUEJAY_API_KEY`: the pod syncs `mivas-secrets` from `.env`, and a missing Bluejay key means no traces, which fails the rubric.
- - **Start with `control-industry`.** It is the wiring smoke (reception → scheduler → `schedule_appointment`). Only when it passes rung 2 do you point the same harness at `healthcare`, `legal`, or `customer-support`.
- - **Rubric, not `goal_success`.** A call passes when all six checks pass: utterances, actual tool calls, trace ids, dead air ≤ 25 s, zero dropouts/clipping, first agent audio ≤ 3 s. `goal_success` is informational. Details and the fix loop: [BLUEJAY.md → Rubric](BLUEJAY.md#rubric).
- - **Read one transcript** per smoke even when the rubric passes (greeting complete, handoff continues the intent, booking date plausible).
- - **Never redeploy mid-run.** In-flight CHIRP sockets die on pod restart. Wait for the run to be final.
- - **Concurrency** is `max_concurrent ≤ MIVAS_REPLICAS × 2–4` for CHIRP families. Smoke defaults to 1.
- - **Storage before rung 4.** Without `MIVAS_SNAPSHOT_BUCKET` (AWS S3 or any S3-compatible endpoint), `verify_task_run.py` skips the final-state check and Pass@1 / Pass^k are not reportable.
- - **Secrets stay out of the tree.** Report ids and URLs in chat; never commit `.env`, hostnames, or keys.
+ - **Interview before commands.** A wrong hosting choice costs an hour of rework.
+ - **`control-industry` before any scored industry.** It proves greeting, handoff, tool call and state write in one two-minute call.
+ - **The rubric is the gate, never `goal_success`.** Six checks: utterances, actual tool calls, trace ids, dead air ≤ 25 s, zero dropouts and clipping, first agent audio ≤ 3 s. See [BLUEJAY.md](BLUEJAY.md#rubric).
+ - **Read one transcript per smoke** even when the rubric passes.
+ - **Never redeploy mid-run.** Live CHIRP sockets die with the container.
+ - **Concurrency** ≈ replicas × 2–4 calls. Smoke uses 1.
+ - **Storage before a full run.** Without a snapshot store, final-state scoring is skipped and Pass@1 / Pass^k are not reportable.
+ - **A slow inference endpoint fails the rubric, not the model.** Put inference in the same region as the container.
+ - **Secrets stay out of the tree.** Report ids and URLs in chat; never commit `.env` or keys.
- ## Rung 4: full industry
+ ## Rung 5: full industry
- `bluejay.py full` converts `industries/<I>/tasks/*/task.json` to digital humans on a fresh
- simulation (via `scripts/tasks_to_digital_humans.py --push`), queues `k` runs per task, and
- prints the verifier + CSV commands. Budget: 72 tasks × k calls of up to 8 minutes; with
- `--max-concurrent 3` and `k=5` expect several hours. `--no-wait` returns the run id;
- resume with `bluejay.py poll --run RUN`. Score with `verifiers/verify_task_run.py` (tools ∧
- handoff ∧ final state), export with `scripts/bluejay_run_to_csv.py`. Do not quote a number
- from a run with void results (`verifiers/verify_run.py` exit 1).
+ `bluejay.py full` turns `industries/<I>/tasks/*/task.json` into digital humans on a fresh
+ simulation, queues `k` runs per task, and prints the scoring commands. Budget 72 tasks × k
+ calls of up to 8 minutes. `--no-wait` returns the run id; resume with `bluejay.py poll --run RUN`.
+ Score with `verifiers/verify_task_run.py` (tools ∧ handoff ∧ final state), export with
+ `scripts/bluejay_run_to_csv.py`. Never quote a number from a run with void results
+ (`verifiers/verify_run.py` exits 1).
## Files
| File | When |
|---|---|
- | [DEPLOY.md](DEPLOY.md) | rung 1: storage model, registry, local/EKS/Baseten/any-k8s recipes, DNS+TLS |
- | [HARNESS.md](HARNESS.md) | picking a harness, env keys, building and registering your own |
- | [BLUEJAY.md](BLUEJAY.md) | REST calls the scripts make, API quirks, rubric, reading results, fix loop |
- | `scripts/preflight.py` | rung 0/1 gate |
+ | [INFERENCE.md](INFERENCE.md) | rung 1: vendor keys per family, or your own model on an inference provider |
+ | [HOSTING.md](HOSTING.md) | rung 2: the host contract, worked recipes, storage |
+ | [HARNESS.md](HARNESS.md) | harness table, and building your own |
+ | [BLUEJAY.md](BLUEJAY.md) | REST calls, API quirks, the rubric, reading results |
+ | `scripts/preflight.py` | rung 0 and 2 gate |
| `scripts/bluejay.py` | `ensure-agent` · `smoke` · `queue` · `poll` · `results` · `score` · `full` |
- | `scripts/verify_integrity.py` | rung 3 |
- | `scripts/score_rubric.py` | six-check rubric (stdlib; `--self-test`) |
- | `assets/` | `minio.yaml` (in-cluster S3), `baseten/config.yaml`, `baseten/auth-proxy-worker.js` |
+ | `scripts/verify_integrity.py` | rung 4 |
+ | `scripts/pair_env.py` | prints the exact `railway variables` command for a pair |
+ | `scripts/score_rubric.py` | six-check rubric (stdlib, `--self-test`) |
+ | `assets/minio.yaml` | in-cluster S3 for local or self-managed Kubernetes |