vllm-input-modalities · git:20260915.581525a · 2026-09-15 · sha256 93f2a3d243e6bebc
vllm-input-modalities git:20260915.581525aA

Name: vllm-input-modalities
Author: air-gapped
Immutable. This exact content is served forever at /api/v1/blob/93f2a3d243e6bebc.
---
name: vllm-input-modalities
description: |-
  vLLM non-chat inference surfaces — text embeddings (`/v1/embeddings`, `/v2/embed`), reranking/scoring (`/rerank`, `/score`), speech-to-text (`/v1/audio/transcriptions`, `/v1/audio/translations`), document OCR via VLMs. Covers 2026 `--runner pooling` (replacing `--task embed`), v0.20 deprecations (`score`→`classify`, multitask pooling, `encode`→`token_embed`+`token_classify`), Matryoshka/MRL, ColBERT/ColPali/ColQwen late-interaction MaxSim, Cohere v2 `/v2/embed`, Jina v3/v4/v5 quirks, cross-encoder score templates, Whisper large-v3-turbo quants, DeepSeek-OCR recipe (NGramPerReqLogitsProcessor, no prefix cache, GUNDAM mode).
when_to_use: |-
  Trigger on any non-chat vLLM endpoint or non-`/v1/chat/completions` model. Covers `--runner pooling`, `--convert embed|classify`, `--pooler-config`, `--hf-overrides is_matryoshka`, CLS/LAST/MEAN/ALL pooling; BGE-M3, Qwen3-Embedding, Jina v3/v4/v5, Qwen3-Reranker, BGE-reranker, mxbai-rerank, ColBERT, ColPali, ColQwen3/3.5, jina-reranker-v3/m0; Whisper, Voxtral, Qwen3-ASR/Omni, Kimi-Audio, Ultravox, FunASR, GLM-ASR; DeepSeek-OCR, dots-OCR, GLM-OCR, Nemotron-Parse. Narrow phrasings — "is --task embed still supported", "which Whisper quant fits 24 GB", "how to serve ColPali". Also implicit — "serve embed model", "deploy Whisper", "embed endpoint for {model}", "rerank endpoint", "OCR pipeline vLLM", "audit embedder", "deploy-memo for embedding", "spec-study embed/rerank/ASR/OCR".
---

# vLLM — embeddings, reranking, speech-to-text, OCR

Target audience: operators who need vLLM's non-chat-completion surfaces. Four
capabilities bundled here because they share operator-facing concepts
(`--runner` flag, pooling configuration, scoring API, multimodal preprocessing)
even though two run on the pooling runner (embedding, reranking) and two run
on the generate runner (STT, OCR).

## The mental model — one flag rules the surface

vLLM decides what a model *does* from the combination of three flags:

```
--runner {auto|generate|pooling|draft}      # what kind of workload
--convert {auto|none|embed|classify}        # adapt a generative LM to a pooler
--pooler-config '{...}'                     # override pool type, dimensions, etc.
```

The pair `(runner, convert)` has replaced the old `--task {generate|embed|
score|classify|reward|...}` flag. `--task` is **gone**, not deprecated —
there is no `task` field in `ModelConfig` and no `--task` in
`vllm/engine/arg_utils.py` at v0.27.0, so passing it is an
unrecognized-argument error. Canonical today:

| Workload | Command | Runner | Notes |
|---|---|---|---|
| Chat / completion | `vllm serve <model>` | `generate` (auto) | default |
| Embedding | `vllm serve <model> --runner pooling` | `pooling` | auto-detects CLS/LAST/MEAN from config |
| Embedding from a causal LM | `vllm serve <model> --runner pooling --convert embed` | `pooling` | adapts `*ForCausalLM` checkpoints |
| Classification | `vllm serve <model> --runner pooling --convert classify` | `pooling` | also how `score`/`rerank` comes online |
| Speech-to-text | `vllm serve <model>` | `generate` | works on any `SupportsTranscription` model |
| OCR (VLM generate) | `vllm serve <model>` | `generate` | standard chat-completion + image input |

**Scoring API is automatic.** There is no `--enable-scoring-api`. The
`/score` + `/rerank` endpoints light up whenever the loaded model's
`Pooler.get_supported_tasks()` includes `classify` (with `num_labels==1`),
`embed`, or `token_embed` (late-interaction). Nothing for the operator to
toggle.

## Quick-answer router

| Question class | File |
|---|---|
| "Which pooling type? Matryoshka? `/v2/embed`? BGE-M3, Qwen3, Jina?" | `references/embedding.md` |
| "Cross-encoder vs ColBERT? Qwen3-Reranker? BGE-reranker? Score templates?" | `references/reranking.md` |
| "Whisper-turbo? Voxtral? Qwen3-ASR? Chunking? Quants?" | `references/stt.md` |
| "DeepSeek-OCR recipe? dots-OCR? VLM document parsing?" | `references/ocr.md` |
| "Is `--task embed` gone? What replaces `encode`?" | `references/runner-flags.md` |

`scripts/probe-endpoint.sh` checks a running vLLM whether it exposes
`/v1/embeddings`, `/rerank`, `/v1/audio/transcriptions`, etc., so an operator
can confirm the right endpoints are live before pointing a client at it.

## Operator cheat sheet — the common cases inline

### Embedding

```bash
# Qwen3-Embedding (causal LM, last-token pooling — auto-detected)
vllm serve Qwen/Qwen3-Embedding-0.6B --runner pooling

# BGE-M3 (XLM-Roberta, CLS pooling — native embedding model)
vllm serve BAAI/bge-m3 --runner pooling

# Jina v3 (needs trust-remote-code; only text-matching LoRA is merged)
vllm serve jinaai/jina-embeddings-v3 --runner pooling --trust-remote-code

# Jina v4 — use the pre-merged retrieval variant
vllm serve jinaai/jina-embeddings-v4-vllm-retrieval --runner pooling \
  --pooler-config '{"pooling_type":"ALL"}' --dtype float16
# Normalization happens client-side (vector is multi-vector per token).

# Mean-pool override (Sentence-Transformers config is broken for this model)
vllm serve ssmits/Qwen2-7B-Instruct-embed-base --runner pooling \
  --pooler-config '{"pooling_type":"MEAN"}'
```

Client request format is standard OpenAI: `client.embeddings.create(...)`.

**Matryoshka dimensions.** Gated on `is_matryoshka: true` in the model's
`config.json` (or `matryoshka_dimensions`). If the config is missing it,
force-enable:

```bash
--hf-overrides '{"is_matryoshka": true}'
# or pin specific dimensions:
--hf-overrides '{"matryoshka_dimensions":[256,512,768]}'
```

Request-side: `client.embeddings.create(model=..., input=..., dimensions=512)`.
Passing `dimensions` to a non-MRL model (BGE-M3, older BGE) returns a 400 by
design — not a bug.

**`dimensions` above the model's `hidden_size` is rejected as of v0.24.0**
(#46313). Before that, an MRL model with no explicit `matryoshka_dimensions`
list validated only `dimensions >= 1` and then sliced `[..., :d]` — so an
oversized value **silently returned a `hidden_size`-length vector** instead
of erroring. If a client has been over-asking, it was never getting the
width it requested; upgrading turns that into a visible `ValueError`. Pin
the list explicitly via `--hf-overrides` to make the valid set unambiguous.

**`/v2/embed` (Cohere v2 compat)** adds `input_type` prompt prefixing,
`output_dimension` (server-side MRL), `truncate=END|START|NONE`, and
`embedding_types=["float","binary","ubinary","base64"]`. Use it when a client
expects Cohere v2's shape.

### Reranking / scoring

Three serving modes; same endpoints, picked automatically:

```bash
# Cross-encoder (classify, num_labels==1)
vllm serve BAAI/bge-reranker-v2-m3 --runner pooling

# Cross-encoder with instruction-aware score template
vllm serve Qwen/Qwen3-Reranker-0.6B --runner pooling --convert classify \
  --chat-template examples/templates/qwen3_reranker.jinja \
  --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],
                    "classifier_from_token":["no","yes"],
                    "is_original_qwen3_reranker":true}'

# Late-interaction (ColBERT family) — MaxSim over token embeddings
vllm serve jinaai/jina-colbert-v2 --runner pooling --trust-remote-code

# Multimodal reranker (ColPali / ColQwen)
vllm serve vidore/colpali-v1.3-hf --runner pooling
```

Client:

```python
# /rerank (Cohere + Jina compat)
resp = requests.post("http://localhost:8000/rerank", json={
    "query": "what is vLLM",
    "documents": ["text 1", "text 2"],
    "top_n": 3,
    "max_tokens_per_doc": 512,  # added v0.20.0 (PR #38827)
})

# /score (bi-encoder cosine, or cross-encoder logit)
resp = requests.post("http://localhost:8000/score", json={
    "text_1": ["query"],
    "text_2": ["doc A", "doc B"],
})
```

Three `score_type`s served through the same routes:

| Score type | Mechanism | Models |
|---|---|---|
| **cross-encoder** | joint query+doc forward → single logit | BGE-reranker-v2-m3/gemma, Qwen3-Reranker, mxbai-rerank-v2, nvidia/llama-nemotron-rerank |
| **late-interaction** | per-token embeddings + MaxSim | ColBERT, ColModernBERT, jina-colbert-v2, ColPali, ColQwen3/3.5, ColModernVBert |
| **bi-encoder** | cosine over `/embeddings` | any embedding model (auto) |

`jinaai/jina-reranker-v3` is listwise ("last but not late interaction") —
`JinaForRanking`, not MaxSim.

### Speech-to-text

```bash
# Whisper large-v3-turbo (base)
vllm serve openai/whisper-large-v3-turbo

# Red Hat production quants (fit on smaller cards, validated)
vllm serve RedHatAI/whisper-large-v3-turbo-FP8-dynamic
vllm serve RedHatAI/whisper-large-v3-turbo-quantized.w8a8
vllm serve RedHatAI/whisper-large-v3-turbo-quantized.w4a16

# Voxtral (Mistral)
vllm serve mistralai/Voxtral-Mini-3B-2507
```

Client:

```bash
curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "file=@audio.wav" \
  -F "model=openai/whisper-large-v3-turbo" \
  -F "language=en"
```

Chunking >30 s audio is server-side (energy-aware split at
`min_energy_split_window_size`). Beam-search transcription arrived in v0.18.

**OOM on 24 GB with Whisper** (issue #15216) is a known sharp edge — Whisper
allocates aggressively for its encoder KV, despite the 1.6 GB checkpoint.
Production path is one of the RedHatAI quants above, or raising
`--gpu-memory-utilization` past 0.9 with eager mode if memory is truly tight.

### OCR (DeepSeek-OCR)

Canonical recipe from `docs.vllm.ai/projects/recipes`:

```bash
vllm serve deepseek-ai/DeepSeek-OCR \
  --logits-processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \
  --no-enable-prefix-caching \
  --mm-processor-cache-gb 0
```

Three non-obvious flags:
- `NGramPerReqLogitsProcessor` is required — without it, table-token
  generation degrades. Enforces `whitelist_token_ids={128821,128822}`,
  `ngram_size=30`, `window_size=90`.
- **Disable prefix caching.** OCR per-request inputs don't share prefixes;
  the cache bookkeeping is pure overhead.
- **`--mm-processor-cache-gb 0`** — the multimodal processor cache isn't
  useful for one-off document images.

DeepSeek reports ~2500 tok/s per A100-40 GB, ~200 k pages/day per GPU. Mode
is hard-coded to GUNDAM (base=1024, image=640, crop=True); Tiny/Small/Base/
Large aren't exposed via env vars yet (tracked issue, as of early 2026).

Invocation is still plain `/v1/chat/completions` with image URLs — there is
no dedicated `/ocr` endpoint.

## Top pitfalls

1. **`--task embed` is dead, not deprecated.** The flag no longer exists —
   passing it fails argument parsing. Use `--runner pooling`. The `score`
   and `encode` *pooling task names* are removed too and raise
   `VLLMValidationError` (`vllm/tasks.py`): use `--convert classify` on a
   `num_labels==1` model to light up `/score` + `/rerank`, and
   `token_embed` / `token_classify` in place of `encode`.

2. **Pooling runs on PIECEWISE CUDA graphs, not full graphs.** That's
   deliberate (pooling models have variable-shape outputs). Don't force
   `--enforce-eager` for production as older cheat sheets suggest — you lose
   the piecewise graph win without gaining anything.

3. **Jina v4 base checkpoint is not vLLM-compatible.** Use
   `jinaai/jina-embeddings-v4-vllm-retrieval` (pre-merged retrieval adapter).
   Serve with `--pooler-config '{"pooling_type":"ALL"}' --dtype float16` and
   normalize client-side — output is multi-vector per token.

4. **Matryoshka without config.** If a model documents MRL support but
   `config.json` lacks `is_matryoshka` / `matryoshka_dimensions`, the server
   returns 400 for any `dimensions` param. Fix: `--hf-overrides
   '{"is_matryoshka":true}'` at serve time. Don't confuse with BGE-M3, which
   genuinely doesn't support MRL.

5. **Qwen3-Reranker needs a score template AND hf-overrides.** It's an
   instruction-tuned causal LM masquerading as a cross-encoder — skipping
   any of the three extras (see the reranking cheat-sheet command above)
   gives random-looking scores, not errors. Full recipe: `references/reranking.md` §2.
   **Second cause of the same symptom, below v0.26.0:** #48901 — any
   LAST-pooling model returns wrong scores once a query+doc pair is long
   enough to be chunk-prefilled, under `torch.compile` (the default). Rule
   out the version before re-checking the overrides.

6. **DeepSeek-OCR with prefix caching on.** It doesn't crash — it just
   wastes time and memory. Same for `--mm-processor-cache-gb > 0` for pure
   OCR traffic. Both defaults are wrong for this workload.

7. **Whisper OOM on 24 GB.** Not a bug. Use a Red Hat quant, or accept that
   large-v3 / large-v3-turbo wants ≥32 GB for comfortable batch sizes.

8. **Late-interaction kernel regression sniff test.** ColBERT / ColPali
   throughput jumped ~14% in v0.17–0.19 from MaxSim optimisations. If those
   models feel slow, check `--enable-flash-late-interaction` (default true)
   wasn't disabled by an old config.

## Landed in v0.20.0 (released 2026-04-27) — verify your deployment

The deprecations previously flagged as "scheduled for v0.20" have now shipped.
Callouts from the v0.20.0 release notes (Breaking Changes + API sections):

- **`logit_bias` / `logit_scale` → `logit_mean` / `logit_sigma`** in
  `PoolerConfig` — explicit breaking change, PR #39530. Old names still
  accepted with deprecation warning.
- **Async scheduling default OFF for pooling models** (PR #39592) — explicit
  breaking change. Pooling throughput should be marginally lower but
  stability improves; re-enable case-by-case if you measured a win on v0.19.
- `--task` flag — still accepted with deprecation warning; `--runner` +
  `--convert` is canonical.
- `score` pooling task — replaced by `classify` + `num_labels==1`.
- Pooling **multitask** — pick a task explicitly via `PoolerConfig(task=...)`
  or `--pooler-config.task <task>`; automatic multitasking is gone.
- `encode` task — split into `token_embed` and `token_classify`.
- `normalize` in `PoolingParams` — removed; use `use_activation`.

Two performance wins also landed in v0.20.0 for pooling:

- **#38559** — mean-pooling optimisation via `index_add` (+5.9% on mean-pool models).
- **#39113** — redundant-sync removal for pooling (+3.7% throughput).

Also landed: `jina-reranker-v3` (#38800), **Jina Embeddings v5** (#39575),
`max_tokens_per_doc` in `/rerank` (#38827), **Generative Scoring** (#34539),
ASR multi-chunk spacing fix (#39116).

### v0.21.0 → v0.25.1

**The v0.20.0 migration above is still the canonical runner surface** —
nothing in v0.21–v0.25 changed `--runner` / `--convert` / `PoolerConfig`.
But two request-validation changes in v0.24.0 will turn requests that used
to succeed into 400s, so they are the ones to check before upgrading.

**Two silent-success → hard-error changes (v0.24.0):**

- **#46313 — matryoshka `dimensions` above `hidden_size` is now rejected.**
  For an MRL model with no explicit `matryoshka_dimensions` list, the old
  code only checked `dimensions >= 1` and then sliced `[..., :d]`, so an
  oversized request *silently returned a `hidden_size`-length vector*. It
  now raises. A client that has been asking for e.g. `dimensions=2048`
  against a 1024-hidden model was already getting 1024 floats back and will
  now get an error instead — the error is the fix, but it surfaces at
  upgrade time.
- **#46119 — rerank `top_n` must be non-negative.** `top_n=-1` was silently
  treated as `top_n=0`. `top_n=0` still means "return all results", and
  values larger than the document count are still accepted.

**New capability worth adopting (v0.24.0):**

- **#45173 — `/v1/embeddings` accepts message-shaped input and
  `chat_template_kwargs`.** Previously message-shaped input to
  `/v1/embeddings` was rejected at validation and `chat_template_kwargs`
  never reached the renderer; only the top-level messages extension worked.
  This is the supported path for instruction-style embedding prompts.
- **#45640** Cohere `/v2/embed` input-exclusivity validation;
  **#44999 / #45210** ColBERT `AutoWeightsLoader` plus a query/document
  embedding io-processor.

**Not applicable despite the release-note wording:** v0.22.0 #43260 "add
truncation side to OpenAI endpoints" covers `/v1/completions` and
`/v1/chat/completions` only — it does **not** add `truncation_side` to
`/v1/embeddings`.

**Perf / internals, no action required:** #41163 `AllPool.forward` +51% and
#41433 GPU↔CPU pooling sync elimination (v0.21.0); #42267 pooling offline
API split into `PoolingOfflineMixin`, #42370/#42274 Speech-to-Text
entrypoint + test consolidation (refactor, no endpoint change) (v0.22.0);
#44593 proper pooling exceptions, #44410 LoRA-adapter-name pooling fix
(v0.23.0); #44612 ASR CPU preprocessing 2.5× faster (v0.24.0); #46762
realtime embeddings under Model Runner V2 and #47071/#47437 pooled-Whisper
sliding-window KV sizing — the latter had been over-reserving encoder KV
blocks by roughly `block_pool_size`× (v0.25.0).

**New architectures since v0.21.0:** Qianfan-OCR (#40136, v0.21.0),
Unlimited OCR (#46564 + Triton R-SWA backend #47102, v0.25.0),
MOSS-Transcribe-Diarize (#47729, v0.25.0 — long-form transcription with
timestamped *speaker labels*, Whisper-style encoder into a Qwen3 decoder),
LLaVA-OneVision-2 (#44785). See `references/ocr.md` §2 and
`references/stt.md`.

**Ecosystem removals that can strand a deployment:** v0.25.0 deleted
PagedAttention entirely (#47361); v0.24.0 deprecated Transformers v4
support (#45161) and removed several model families outright (ERNIE,
Xverse, Dots1, Bamba, Mono-InternVL); v0.25.0 removed Baichuan, Aquila,
Grok, Tarsier/Tarsier2, AyaVision/MusicFlamingo, Mantis. None are pooling
or STT models, but check before pinning a newer image for an unrelated
reason.

### v0.26.0 + v0.27.0 (baseline v0.27.0, released 2026-08-10)

*Latest release is v0.27.1 (2026-08-11), a one-change patch — "quantized
DSpark Markov heads" (#50424) — with nothing on this skill's surface. The
claims below were verified against the `v0.27.0` tag.*

**A pooling correctness bug, fixed in v0.26.0 — the highest-value item on
this page.** #48901: LAST-pooling models (the PR names
`Qwen/Qwen3-Reranker-0.6B`) returned **wrong relevance scores** whenever a
query+document pair was long enough for its prefill to be split into chunks,
but only under `torch.compile` (the default). A pair scoring ~0.83 unchunked
collapsed to ~0.01–0.36 and varied run to run. `--enforce-eager` was always
correct and the offline batch path happened not to chunk, which is why it hid
so easily. **Any ranking a pre-v0.26.0 engine produced on long documents is
suspect** — this is a re-score, not just an upgrade. `references/reranking.md` §2.

**Model Runner V2 is the pooling default from v0.29.0. Opt-in at v0.28.0 and below.**
v0.26.0/v0.27.0 landed encoder-only attention (#49331), sequence
`embed`/`classify` pooling (#48791), `token_classify` (#50293), `token_embed`
(#50574) and BGE-M3's `embed&token_classify` (#50661) on MRV2. The
default-enable PR ([#48290](https://github.com/vllm-project/vllm/pull/48290)) merged 2026-08-19 — after
the v0.28.0 cut, so it first ships in **v0.29.0**. Upgrading a pooling
deployment to v0.29.0 therefore changes its runner without a config change; pin
`VLLM_USE_V2_MODEL_RUNNER=0` to keep V1. Details and the escape hatch:
`references/runner-flags.md` §11.

**STT request surface moved forward (v0.27.0):** `diarized_json` response
format (#48543), cumulative long-form chunk timestamps (#41131), extra
sampling params on the *translation* API (#45839), MOSS-TD max audio duration
raised to 90 minutes (#49403). The STT entrypoint package also moved on disk —
`references/stt.md` §2.

**New pooling architectures:** `jina-embeddings-v5-text-nano` (#50688,
v0.27.0 — EuroBERT *encoder* backbone under the existing
`JinaEmbeddingsV5Model`); `BertForMaskedLM` (#48463),
`RobertaForTokenClassification` / `XLMRobertaForTokenClassification` (#47991),
LongCat-Flash-Lite n-gram embedding (#47857), all v0.26.0.

**More removals, none on this surface:** TeleChat (#47989), Persimmon and Fuyu
(#48096) in v0.26.0; Plamo2 (#49729) and Ouro (#49786) in v0.27.0. Unrelated
but image-bump relevant: `max_num_partial_prefills` and
`max_long_partial_prefills` were removed in v0.27.0 (#49244).

## Paired skills

- `vllm-configuration` → environment variables, cache paths, telemetry opt-out.
- `vllm-observability` → metrics exposition, Prometheus endpoints.
- `vllm-nvidia-hardware` → SM-level platform support for pooling + FP8 paths.

## Source and refresh policy

- First-party: vLLM docs at
  <https://docs.vllm.ai/en/stable/models/pooling_models/> (README + embed /
  scoring / token_embed / specific_models subpages), and
  `docs/contributing/model/transcription.md` in the repo.
- Production STT canonical reference: Red Hat Developer blog for Whisper +
  RHAIIS (link in `references/stt.md`).
- DeepSeek-OCR canonical reference: vLLM recipes page (link in
  `references/ocr.md`).
- Refresh triggers: any v0.30+ release, a new Jina embeddings major version, or
  a new native-multimodal reranker shipping. (MRV2 becoming the pooling default
  was a trigger; it fired — #48290 shipped in v0.29.0.) Note that three passes running have found the *runner*
  surface quiet while **request validation and pooling correctness** moved
  underneath it — grep release bodies for
  `pooling|rerank|embedding|matryoshka|top_n`, not just for runner flags.
- External-ref audit log: `references/sources.md`.

Last verified: 2026-08-11 (against vLLM v0.26.0 + v0.27.0 release notes, the
PRs behind each pooling/rerank/STT/OCR hit, and the v0.27.0 source tree for
the runner flags, task enums and MRV2 pooling gate). Headline: #48901 made
pre-v0.26.0 LAST-pooling scores untrustworthy on chunk-prefilled pairs, and
`--task` is now absent from the CLI entirely rather than deprecated.