openrouter-provider-ranking · diff
v1.0.0 to v1.0.0
160 added, 87 removed. Audit A to A.
---
name: openrouter-provider-ranking
- description: "Use this skill when a user asks to rank, compare, benchmark, prioritize, or generate routing for OpenRouter provider endpoints (provider.order, provider.only, :exacto) by TPS/throughput, TTFT/latency, effective price, uptime, cache hit rate, tool-call/Exacto quality, quantization, context, privacy, or fallback diversity; also for requests about «сортировка/приоритет провайдеров OpenRouter». Do not use for broad model-family selection unless endpoint-level provider routing is required."
+ description: "Use this skill when a user asks to rank, compare, benchmark, prioritize, or generate routing for OpenRouter provider endpoints (provider.order, provider.only, :exacto) by TPS/throughput, TTFT/latency, effective price, uptime, cache hit rate, tool-call/Exacto quality, quantization, context, privacy, or fallback diversity; also for requests about OpenRouter provider sorting or prioritization. Do not use for broad model-family selection unless endpoint-level provider routing is required."
compatibility: "Requires Python 3.10+; live endpoint discovery requires outbound HTTPS and a management-capable OpenRouter key in OPENROUTER_API_KEY. Offline endpoint JSON is supported. The bundled ranker uses only the Python standard library."
metadata:
version: "1.0.0"
domain: "openrouter-routing"
---
# OpenRouter Provider Ranking
- Ранжируй **endpoint-провайдеров одного OpenRouter model slug** под конкретный workload. Не называй порядок «глобально оптимальным»: он оптимален только относительно входных ограничений, весов, профиля токенов и доступной телеметрии.
+ Rank the **endpoint providers of a single OpenRouter model slug** for a
+ specific workload. Never call the order "globally optimal": it is optimal only
+ against the stated constraints, weights, token profile and available telemetry.
- ## Обязательные правила
+ ## Mandatory rules
- 1. Сначала применяй hard constraints, затем считай score. Никогда не компенсируй несовместимость высоким TPS или низкой ценой.
- 2. Для tool-calling по умолчанию используй `native-exacto`: модель с суффиксом `:exacto`, без `provider.sort` и без `provider.order`. Собственный score в этом режиме — диагностический, а не замена закрытой телеметрии OpenRouter.
- 3. Используй `manual` только когда пользователь требует детерминированный порядок, собственная production-телеметрия важнее Exacto, либо нужен явно зафиксированный failover chain.
- 4. Не сочетай `:exacto` с `provider.sort`: явный sort имеет приоритет. Не добавляй `provider.order` в режиме `native-exacto`.
- 5. Учитывай цену **на реальном профиле токенов**, а не только headline input/output price. Включай cache read/write, per-request fee, conditional pricing overrides и наблюдаемый cache hit rate.
- 6. Не выдумывай отсутствующие Exacto/benchmark/cache/tool-success метрики. Применяй conservative prior и uncertainty penalty; явно отмечай пробелы.
- 7. Для multi-turn workload передавай стабильный `session_id`. Помни: ручной `provider.order` отключает OpenRouter sticky provider routing; `session_id` не отменяет это ограничение.
- 8. Не сохраняй API key в skill, конфиг, лог или итоговый JSON. Читай его только из `OPENROUTER_API_KEY` либо указанной environment variable.
+ 1. Apply hard constraints first, score second. Never let high TPS or a low price compensate for an incompatibility.
+ 2. For tool calling default to `native-exacto`: the `:exacto` model suffix, no `provider.sort`, no `provider.order`. Your own score in that mode is diagnostic, not a replacement for OpenRouter's private telemetry.
+ 3. Use `manual` only when the user requires a deterministic order, when your own production telemetry outweighs Exacto, or when a failover chain has to be pinned explicitly.
+ 4. Never combine `:exacto` with `provider.sort` — an explicit sort wins. Never add `provider.order` in `native-exacto` mode.
+ 5. Price **on the real token profile**, not on headline input/output rates. Include cache read/write, per-request fees, conditional pricing overrides and the observed cache hit rate.
+ 6. Do not invent missing Exacto, benchmark, cache or tool-success metrics. Apply a conservative prior and an uncertainty penalty, and name the gaps explicitly.
+ 7. For multi-turn workloads pass a stable `session_id`. Remember that a manual `provider.order` disables OpenRouter's sticky provider routing, and `session_id` does not restore it.
+ 8. Never store the API key in the skill, a config, a log or the output JSON. Read it only from `OPENROUTER_API_KEY` or the environment variable the user names.
+ 9. **A catalogue field is a hypothesis; a measurement is a fact.** Do not hand back a ranking as the answer: always propose the minimal verification of the top candidates against the constraints that were asked for (step 5), and say plainly whether it was run.
+ 10. The comparison row is `endpoint × reasoning effort`, not the endpoint. Changing effort moves price, latency and quality more than changing provider does — and `"none"` means "omit the field", not "no reasoning".
+ 11. Compute price from components on the token profile (`in`, `out`, `cache_read`, `cache_write`, per-request fee). Headline and effective price swap places: the most expensive rate card produced the lowest cost per step at a 99.996 % cache hit.
+ 12. Measure cache, latency and reliability on the profile that will ship. Cache hit on a repeated identical prompt (99 %) and in an agent run (68 %) are different numbers; TTFT on a short prompt is 4-6x optimistic; and fitness for a single request does not predict behaviour under sequential load.
- ## Входные данные
+ ## Inputs
- Собери или оцени:
+ Collect or estimate:
- - model slug, например `deepseek/deepseek-v4-flash-0731`;
- - `uses_tools`, streaming, требуемые параметры, context и output limits;
- - ожидаемые prompt/completion tokens и requests per session;
+ - the model slug, e.g. `deepseek/deepseek-v4-flash-0731`;
+ - `uses_tools`, streaming, required parameters, context and output limits;
+ - expected prompt/completion tokens and requests per session;
- cacheable prompt fraction, token-level cache read/write rate, response-cache hit rate;
- - hard caps: цена, latency, TPS, uptime, quantization, moderation, ZDR/data policy;
- - цель: quality, balanced, interactive latency, cost или batch throughput;
- - собственную telemetry по provider tag, если она есть.
+ - hard caps: price, latency, TPS, uptime, quantization, moderation, ZDR/data policy;
+ - the goal: quality, balanced, interactive latency, cost or batch throughput;
+ - your own telemetry by provider tag, if any exists.
- Когда вход неполный, используй профиль `agentic-balanced`. Для tool-calling включай `uses_tools=true`. Не задавай ненулевой cache hit rate без наблюдений или обоснованной workload-модели.
+ When the input is incomplete, use the `agentic-balanced` profile. For tool calling set `uses_tools=true`. Do not assume a non-zero cache hit rate without observations or a defensible workload model.
- ## Процедура
+ ## Procedure
- ### 1. Классифицируй workload
+ ### 1. Classify the workload
- Выбери один профиль:
+ Pick one profile:
- - `agentic-balanced` — default для агентов и B2B SaaS;
- - `agentic-quality` — tool correctness и reliability важнее цены;
- - `interactive` — минимизация TTFT и end-to-end latency;
- - `cost` — минимизация ожидаемой стоимости при SLO;
- - `batch` — throughput/cost для длинных completion.
+ - `agentic-balanced` — the default for agents and B2B SaaS;
+ - `agentic-quality` — tool correctness and reliability outweigh price;
+ - `interactive` — minimise TTFT and end-to-end latency;
+ - `cost` — minimise expected cost under an SLO;
+ - `batch` — throughput and cost for long completions.
- Для нестандартной цели переопредели `weights`, но сумма после нормализации должна быть положительной. Подробная формула: [references/scoring.md](references/scoring.md).
+ For a goal outside these, override `weights`; the sum after normalisation must be positive. The formula: [references/scoring.md](references/scoring.md).
- ### 2. Получи свежие endpoint-метрики
+ ### 2. Fetch fresh endpoint metrics
- Предпочтительный путь — OpenRouter Endpoints API через bundled script:
+ Preferred path — the OpenRouter Endpoints API through the bundled script:
```bash
python3 scripts/rank_providers.py \
--model deepseek/deepseek-v4-flash-0731 \
--config assets/config.example.json \
--format markdown \
--output recommendation.md
```
- Для воспроизводимого/offline анализа:
+ For a reproducible or offline analysis:
```bash
python3 scripts/rank_providers.py \
--endpoints-file endpoints.json \
--config config.json \
--observations telemetry.jsonl \
--previous-ranking previous-result.json \
--output result.json
```
- Endpoint API дает provider tag, pricing, quantization, context/output limits, supported parameters, uptime, latency и throughput percentiles. Если анализируется performance page, перенеси видимые provider-specific Auto Exacto/benchmark значения в observations по точному `tag`; не пытайся угадывать соответствие по display name при наличии tag.
+ The Endpoints API gives the provider tag, pricing, quantization, context and output limits, supported parameters, uptime, and latency and throughput percentiles. When reading a performance page, carry any provider-specific Auto Exacto or benchmark values into observations by exact `tag`; never guess the match from a display name when a tag exists.
- Форматы входа: [references/input-formats.md](references/input-formats.md).
+ Input formats: [references/input-formats.md](references/input-formats.md).
- ### 3. Добавь production-телеметрию
+ **Read the metrics as they are defined, not as they read.** `latency_*` arrives
+ in **milliseconds** — `1624.8` is 1.6 s. A p90 *throughput* is the fast tail and
+ a p90 *latency* is the slow one, so rank on `p50` and report both. The
+ storefront's own numbers are collected on short prompts; see step 5 and
+ [references/storefront-traps.md](references/storefront-traps.md).
- Приоритет сигналов качества:
+ ### 3. Add production telemetry
- 1. собственный tool-call/schema success по тому же model, prompt class и provider tag;
- 2. provider-specific Exacto/benchmark значения с performance page;
- 3. endpoint performance/uptime API;
- 4. conservative prior при отсутствии данных.
+ Quality signals, in order of priority:
- Собирай минимум: `provider_name` или `provider_tag`, success, tool success, prompt/completion/cached/cache-write tokens, TTFT, generation time/TPS и total cost. Для долей с малой выборкой используй Wilson lower bound, а не raw percentage.
+ 1. your own tool-call/schema success for the same model, prompt class and provider tag;
+ 2. provider-specific Exacto or benchmark values from the performance page;
+ 3. the endpoint performance/uptime API;
+ 4. a conservative prior when nothing is available.
- ### 4. Запусти ranking и проверь результат
+ Collect at least: `provider_name` or `provider_tag`, success, tool success, prompt/completion/cached/cache-write tokens, TTFT, generation time or TPS, and total cost. For rates from a small sample use a Wilson lower bound, not the raw percentage.
- Скрипт должен:
+ ### 4. Run the ranking and check the result
- - исключить неактивные/несовместимые endpoints;
- - разрешить pricing overrides на момент запроса;
- - посчитать expected cost для workload;
- - смешать OpenRouter percentiles с собственными наблюдениями по sample confidence;
- - нормализовать cost/TPS/TTFT/E2E относительно текущего eligible pool;
- - применить quality, reliability, cache, fidelity и uncertainty components;
- - стабилизировать порядок предыдущим результатом;
- - выбрать fallback chain с provider-family diversity, если разрыв score приемлем.
+ The script should:
- Проверь exit code. При `4` не ослабляй ограничения молча: покажи, какие constraints исключили все endpoints, и предложи минимальное ослабление.
+ - drop inactive and incompatible endpoints;
+ - resolve pricing overrides as of the request;
+ - compute the expected cost for the workload;
+ - blend OpenRouter percentiles with your own observations by sample confidence;
+ - normalise cost, TPS, TTFT and E2E against the current eligible pool;
+ - apply the quality, reliability, cache, fidelity and uncertainty components;
+ - stabilise the order against a previous result;
+ - pick a fallback chain with provider-family diversity when the score gap allows.
- ### 5. Выбери routing mode
+ Check the exit code. On `4` do not relax anything silently: show which constraints eliminated every endpoint and propose the smallest relaxation.
- | Условие | Режим | Что отправлять |
+ ### 5. Verify the candidates with real requests — mandatory
+
+ A ranking is built from catalogue fields, and several of them are wrong often
+ enough to change the answer: `supports_implicit_caching` reads `false` on an
+ endpoint that caches 99 % of the prefix; storefront TTFT is taken on short
+ prompts and is 4-6x optimistic; and the cheapest endpoint on a bench may serve
+ no request at all under sequential load. The full list, each with its
+ reproduction: [references/storefront-traps.md](references/storefront-traps.md).
+
+ So **never hand back a ranking as the final answer**. Always propose the
+ minimal verification of the top candidates against the same hard constraints
+ the score used, and run it where you can:
+
+ ```bash
+ python3 scripts/probe_endpoints.py \
+ --model deepseek/deepseek-v4-flash-0731 \
+ --providers coreweave/fp8,fireworks,siliconflow/fp8 \
+ --prompt-file real-prompt.txt \
+ --max-tokens 600 --runs 3 \
+ --min-tps 50 --max-ttft 6 --require-cache --require-max-tokens
+ ```
+
+ The script sends `--runs` identical requests per endpoint and reports the
+ measured TTFT, TPS, cost, cache hit share, completion tokens and error class,
+ then judges each against the thresholds you pass. Exit code `4` means none
+ survived — the ranker's own convention; do not relax the thresholds silently.
+
+ What must be passed:
+
+ - `--prompt-file` with **your own** prompt once a decision is actually being
+ made. The `--prompt-tokens` filler is fine for a rough cut, but TTFT depends
+ on size and contract compliance depends on content.
+ - `--runs >= 2`, but do not read a cache miss from filler as a verdict: a fresh
+ prefix does not become cacheable inside a short probe. The same endpoint gave
+ 99 % over two calls on a prompt that was already in use and 0 % over four on a
+ new one. A hit proves caching; a miss proves it only with `--prompt-file`.
+ - Thresholds taken from the hard constraints, not invented: the check must
+ answer the user's question, not a generic one.
+
+ What the check does **not** do: it does not measure exact throughput and it does
+ not certify fitness. A failure is a conclusion; a pass is "on this profile, in
+ this window, over N calls". For an agentic workload add a run of sequential
+ calls — fitness for a single request does not predict it.
+
+ A separate one-request check of the data policy, whenever the constraints carry
+ `data_collection: deny` or `zdr`: `max_tokens: 1`, `allow_fallbacks: false`, one
+ request per candidate. Distinguish a policy refusal (`404 No endpoints found
+ matching your data policy`) from a provider 429 — different causes, different
+ conclusions.
+
+ ### 6. Choose the routing mode
+
+ | Condition | Mode | What to send |
|---|---|---|
- | Tool calls, нет требования фиксированного порядка | `native-exacto` | `model: <slug>:exacto`, filters/preferences, без `sort/order` |
- | Фиксированный failover chain или сильная собственная telemetry | `manual` | `provider.order` из ranking |
- | Только одна простая цель без custom score | native OpenRouter | `provider.sort: price/throughput/latency`; bundled ranker не обязателен |
+ | Tool calls, no fixed order required | `native-exacto` | `model: <slug>:exacto`, filters/preferences, no `sort`/`order` |
+ | A pinned failover chain, or strong first-party telemetry | `manual` | `provider.order` from the ranking |
+ | One simple goal with no custom score | native OpenRouter | `provider.sort: price/throughput/latency`; the bundled ranker is optional |
- OpenRouter-specific взаимодействия и ограничения: [references/openrouter-routing.md](references/openrouter-routing.md).
+ OpenRouter-specific interactions and limits: [references/openrouter-routing.md](references/openrouter-routing.md).
- ## Формат ответа пользователю
+ ## Response format
- Всегда возвращай:
+ Always return:
- 1. выбранный mode и краткое обоснование;
- 2. таблицу eligible ranking: provider tag, score, expected cost/request, TPS percentile, TTFT, E2E, uptime, quality confidence и cache hit rate;
- 3. готовый JSON request fragment;
- 4. excluded providers с причинами;
- 5. coverage/warnings и список отсутствующих сигналов;
- 6. правило обновления ranking: пересчитывать после заметного drift, изменения цен/endpoints или достаточного прироста telemetry; не фиксировать бессрочно.
+ 1. the chosen mode and a short justification;
+ 2. the eligible ranking table: provider tag, score, expected cost per request, TPS percentile, TTFT, E2E, uptime, quality confidence and cache hit rate;
+ 3. a ready JSON request fragment;
+ 4. the excluded providers, each with its reason;
+ 5. coverage and warnings, and the list of missing signals;
+ 6. **the minimal verification plan** — a ready `probe_endpoints.py` command with
+ thresholds taken from the hard constraints, and its result if it was run.
+ Call the ranking a hypothesis until that check has been made;
+ 7. the refresh rule: recompute after noticeable drift, a price or endpoint
+ change, or enough new telemetry. Never pin an order indefinitely.
- В `native-exacto` четко разделяй **diagnostic ranking** и **authoritative runtime ordering by Exacto**.
+ In `native-exacto`, keep **the diagnostic ranking** and **the authoritative runtime ordering by Exacto** clearly apart.
## Validation checklist
- Перед выдачей результата проверь:
+ Before returning a result, check that:
- - provider tags взяты из свежего API/файла, а не из памяти;
- - hard constraints применены до score;
- - цены переведены из USD/token в понятные USD/M и workload cost без двойного пересчета;
- - p90 throughput трактуется как нижняя граница, достигаемая примерно 90% запросов, а p90 latency — как верхняя граница примерно для 90% запросов;
- - `provider.max_price` передан в USD per million tokens, тогда как endpoint pricing хранится в USD per token;
- - `provider.order` отсутствует в `native-exacto`;
- - service-tier tags (`/fast`, `/flex`) не включены случайно;
- - при cache-heavy сессиях указан stable `session_id`, а последствия manual order отмечены;
- - низкая выборка не переоценивается;
- - итоговый request JSON синтаксически валиден.
+ - provider tags come from a fresh API call or file, not from memory;
+ - hard constraints were applied before the score;
+ - prices were converted from USD/token to readable USD/M and to workload cost without double conversion;
+ - **`p90` throughput is read as the fast tail — the value about 10 % of requests exceed — and `p90` latency as the slow one. Rank on `p50`;** a provider whose p90 is 57 and p50 is 33 is not the fastest, and reading p90 as a floor already put one first;
+ - `latency_*` is read as milliseconds, while `provider.max_price` is passed in USD per million tokens and endpoint pricing is stored in USD per token;
+ - `provider.order` is absent in `native-exacto`;
+ - service-tier tags (`/fast`, `/flex`) were not included by accident;
+ - cache-heavy sessions carry a stable `session_id`, and the consequences of a manual order are noted;
+ - a small sample is not over-read;
+ - the final request JSON is syntactically valid;
+ - the minimal verification of the top candidates was proposed — and run where possible — and a ranking without it was called a hypothesis;
+ - `supports_implicit_caching`, TTFT and throughput were not taken on faith wherever the conclusion depends on them.
## Bundled resources
- - `scripts/rank_providers.py` — автономный ranker и генератор routing fragment.
- - `scripts/validate_skill.py` — self-check frontmatter, resources, syntax и unit tests.
- - `assets/config.example.json` — конфиг для tool-using agent workload.
- - `assets/observations.example.json` — агрегированные provider observations.
- - `assets/telemetry.example.jsonl` — сырые request-level observations.
- - `tests/trigger-evals.json` — positive/negative activation queries для description.
- - `references/scoring.md` — формула, нормализация и профили.
- - `references/input-formats.md` — схема config/telemetry.
+ - `scripts/rank_providers.py` — the standalone ranker and routing-fragment generator.
+ - `scripts/probe_endpoints.py` — the minimal verification of candidates with real
+ requests: TTFT, TPS, cost, cache hit, `max_tokens` compliance and error class;
+ judges each against the thresholds passed, exit code `4` when none survives.
+ - `scripts/validate_skill.py` — self-check of frontmatter, resources, syntax and unit tests.
+ - `assets/config.example.json` — a config for a tool-using agent workload.
+ - `assets/observations.example.json` — aggregated provider observations.
+ - `assets/telemetry.example.jsonl` — raw request-level observations.
+ - `tests/trigger-evals.json` — positive and negative activation queries for the description.
+ - `references/scoring.md` — the formula, normalisation and profiles.
+ - `references/input-formats.md` — the config and telemetry schema.
- `references/openrouter-routing.md` — semantics OpenRouter routing/caching.
+ - `references/storefront-traps.md` — reproduced disagreements between catalogue
+ fields and endpoint behaviour; read it before ranking.
- Для проверки пакета запусти:
+ To validate the package, run:
```bash
python3 scripts/validate_skill.py
```