git:20260922.fd88423 to git:20261018.1350c93

64 added, 83 removed. Audit A to A.

---

name: vllm-omni

description: |-

vLLM-Omni output-side multimodal generation — image (FLUX.1/2, Qwen-Image, GLM-Image, BAGEL, SD3.5, HunyuanImage-3.0), video (Wan2.1/2.2, LTX-2, HunyuanVideo-1.5), TTS (Qwen3-TTS, CosyVoice3, Voxtral-TTS), any-to-any omni (Qwen3-Omni, Qwen2.5-Omni, MiMo-Audio) via `vllm serve --omni`. Stage-based disaggregation (OmniConnector + Mooncake + RDMA), `/v1/images/generations`, async+sync `/v1/videos`, `/v1/audio/speech` with voice-upload, PCM16 WebSocket `/v1/realtime`, Ulysses/Ring SP + CFG-parallel, DiT FP8/INT8/GGUF, CUDA/ROCm/NPU/XPU/MUSA matrix, release pitfalls (v0.19.0rc1 FLUX regression, GLM-Image transformers>=5.0, Qwen3-TTS enforce-eager).

when_to_use: |-

Trigger on any vLLM deployment producing non-text output (image/video/audio) or any-to-any omni model, or model names ending `-Image`/`-TTS`/`-Omni`/`-Video`. Keywords — `vllm serve --omni`, `vllm-omni`, `/v1/images/generations`, `/v1/videos`, `/v1/audio/speech`, `/v1/audio/voices`, `/v1/realtime`, `async_chunk`, `stage_configs_path`, OmniConnector, MooncakeStore, OmniDiffusionSamplingParams, FlowUniPC, TeaCache, Cache-DiT, Sage/Ring/Ulysses, `--ulysses-degree`, `--ring-degree`, Thinker/Talker/Code2Wav, BAGEL, Wan2.2, FLUX.2-klein, ComfyUI bridge, verl RL. Narrow phrasings — "serve Qwen-Image", "Qwen3-Omni streaming audio", "async video job". Also implicit — "deploy image gen", "TTS endpoint", "video gen pipeline", "audit omni", "deploy-memo for {model}-Image/-TTS/-Video". NOT for embeddings/reranking/STT/OCR (→ `vllm-input-modalities`).

---

# vLLM-Omni — output-side multimodal serving

Target: operators who serve image / video / audio / any-to-any generation models with the vLLM-Omni fork of vLLM. vllm-omni extends upstream vLLM (same CUDA/ROCm/NPU/XPU runtime, same OpenAI-compat API server) to add non-autoregressive DiT models, multi-stage pipeline execution, diffusion schedulers, CFG plumbing, and real-time streaming audio I/O — things upstream vLLM does not ship.

This skill is a **reference**, not a tutorial. SKILL.md holds the mental model, quick-answer router, top pitfalls, and operator cheat sheet. The `references/` files hold endpoint catalogs, supported-model tables, stage-config grammar, and the diffusion/DiT details. Read only the reference file that matches the question.

- ## The one thing to know before anything else

- vllm-omni is **not a fork** — it layers on top of upstream vLLM, registers OmniModelConfig, and adds one CLI flag: `--omni`. Adding `--omni` to `vllm serve` routes the server through `vllm_omni.entrypoints`. As of v0.20.0 the old vLLM entrypoint-hijack / `patch.py` early-import mechanism was **removed** — the v0.20.0 release notes state "removal of the old vLLM entrypoint hijack, and runtime changes needed for the 0.20.0 integration path (#3232, #3082, #3352, #3393, #2306)". The omni runtime is rebased onto upstream vLLM rather than monkey-patching it — v0.20.0 via PR #3232, then forward through the v0.21/v0.22 rebases (#3530, #3891) and the v0.23.0/v0.24.0 rebases (#4286, #4709). The architectural claim is to decompose any-to-any models into a **graph of disaggregated stages** (Thinker / Talker / Code2Wav for Qwen3-Omni; AR-encoder / DiT for Qwen-Image) connected via `OmniConnector`, so each stage scales independently. The paper (arXiv:2602.02204) claims up to 91.4% JCT reduction vs an unspecified baseline — treat as an architectural argument, not a deployment benchmark.

- Version alignment is strict: vllm-omni major.minor must match upstream vLLM major.minor. **v0.28.0 (2026-08-31) is the current stable**; v0.29.0rc1 (2026-09-10) is the newest tag and has not been promoted. v0.26.0 (2026-08-03) was rebased on upstream vLLM 0.26.0 (#5443); first stable was v0.14.0 (2026-01-31). The v0.19.0rc1 FLUX.1-dev regression (#2730) is **fixed in v0.20.0 stable** (PR #2760) — no version pin needed anymore.

- **Not every minor gets a stable.** v0.21.0, v0.23.0, v0.25.0 and v0.27.0 exist

- only as `rc1` — the stable line went v0.20.0 → v0.22.0 → v0.24.0 → v0.24.1 →

- v0.26.0 → v0.28.0. Check `prerelease` before pinning a minor; roughly half the

- tags in this repo are rc-only.

- Don't infer a missing release is a withdrawn one.

- ### The v0.24.1 channel mismatch is resolved — but keep checking channels

+ ## How it plugs into vLLM

- Verified 2026-08-11. All three channels now agree on **v0.26.0**:

+ vllm-omni layers on upstream vLLM: `vllm serve <model> --omni` routes the server through `vllm_omni.entrypoints`, and omni model registrations reach worker subprocesses through the `vllm.general_plugins` entry point. There is no `patch.py` / early-import hijack on any current build. Any-to-any models run as a **graph of disaggregated stages** (Thinker / Talker / Code2Wav for Qwen3-Omni; AR-encoder / DiT for Qwen-Image) joined by `OmniConnector`, so each stage is configured and scaled on its own (`references/stage-config.md`).

- | Channel | Newest | Checked |

- |---|---|---|

- | GitHub releases | **v0.26.0** (2026-08-03), marked Latest | `gh release list` |

- | PyPI | **0.26.0** (2026-08-03) | `pypi.org/pypi/vllm-omni/json` |

- | Docker Hub | **v0.26.0** (2026-08-03); `latest` matches no versioned tag — it tracks the unversioned `vtest-nightly` build | Docker Hub v2 tags API |

+ ## Version pins — check before installing anything

- The 2026-07-21 finding — v0.24.1 existing as a GitHub tag with no wheel and no

- image — is now **moot, not fixed**: v0.24.1 was never uploaded to PyPI, but

- v0.26.0 supersedes it everywhere, and the Qwen-Image fix it carried (#5017,

- issue #4964) is in v0.26.0 by descent. Nobody needs the `git+…@v0.24.1` install

- form any more.

+ | Rule | Action |

+ |---|---|

+ | omni major.minor must equal vLLM major.minor (`vllm_omni/version.py` warns on mismatch) | Install omni **v0.30.0** (current stable) with vLLM **0.30.0** |

+ | v0.31.0rc1 is the newest tag but a pre-release with a not-fully-passing nightly | Do not deploy it as current |

+ | **No omni release, not even an rc, matches vLLM 0.32**: omni v0.30.0 pins `transformers >= 5.13.0, < 5.15`, vLLM v0.32.0 needs `>= 5.16.1, < 5.20.0` | Never install omni into a vLLM 0.32.0 env. Run omni in its own venv/image on vLLM 0.30.0 |

+ | Odd minors (v0.21, v0.23, v0.25, v0.27, v0.29) exist only as `rc1` | Check `prerelease` before pinning a minor; a missing stable is not a withdrawn one |

+ | Docker `vllm/vllm-omni:latest` tracks the newest tag, rc included (today v0.31.0rc1) | Pin `vllm/vllm-omni:v0.30.0` (`-x86_64` / `-aarch64` variants) in every manifest |

+ | ROCm images live in `vllm/vllm-omni-rocm`, newest stable **v0.28.0** | For ROCm on v0.30.0 use the wheel path (Install below) |

+ | A tag is not a wheel and not an image | Check GitHub releases, PyPI and both Docker repos before quoting a version |

- Two things carry forward. First, **`latest` on Docker Hub is not a released

- version**: as of 2026-08-18 its digests match no versioned tag at all — it

- tracks the unversioned `vtest-nightly` build. Pin the exact tag. Second, the underlying

- lesson stands — a tag is not a wheel — so check all three channels before

- quoting a version, rather than assuming this pass's parity is permanent.

+ Release-by-release history, channel evidence and the vLLM 0.32.0 compatibility probe: `references/sources.md` §Releases.

## Quick-answer router

**Serving a specific endpoint** → `references/endpoints.md`

- `/v1/images/generations`, `/v1/images/edits` (DALL·E-shape)

- - `/v1/videos` (async job) + `/v1/videos/sync` (raw MP4, 1200s timeout)

+ - `/v1/videos` (async job) + `/v1/videos/sync` (raw MP4, 600s default timeout)

- `/v1/audio/speech`, `/v1/audio/voices` (list + upload), `/v1/audio/speech/batch`, `/v1/audio/speech/stream` (WebSocket)

- - `/v1/realtime` (WebSocket PCM16 in/out for Qwen3-Omni)

+ - `/v1/realtime` (WebSocket PCM16 in/out for Qwen3-Omni; `?duplex=1` / `/v1/duplex` for duplex deployments)

- `/v1/chat/completions` with diffusion via `extra_body`

**Picking a model** → `references/models.md`

- Full supported-architecture → HuggingFace-ID table

- Per-model platform matrix (CUDA / ROCm / NPU / XPU / MUSA)

- - Known-issue flags per family

+ - Known-issue flags per family, including the pre-v0.20 FLUX / GLM-Image / Qwen3-TTS `--enforce-eager` pitfalls (all fixed on v0.20.0+)

**Writing / debugging stage configs** → `references/stage-config.md`

- OmniModelConfig + StageConfig YAML grammar

- OmniConnector types (Shared-memory / Mooncake-Store / Mooncake-Transfer-Engine / RDMA / Yuanrong)

- Pipeline edge validation, entry-point requirement

- `stage_id`, `model_stage`, `worker_type`, `engine_output_type`, `async_chunk`

**DiT-specific questions** → `references/diffusion.md`

- Schedulers (FlowUniPC + model-specific)

- CFG plumbing (dual CFG for Wan2.2, true_cfg_scale for Qwen-Image, cfg_branch_past_key_values)

- Caches: TeaCache / Cache-DiT / latent cache / noise_pred cache

- Quantization: FP8 (Flux #1640), INT8 (Z-Image/Qwen-Image #1470) per-component via `ComponentQuantizationConfig`; **GGUF moved out of tree in v0.26.0** (#4769) to `vllm-project/vllm-gguf-plugin`

- Ulysses / Ring sequence parallel, CFG-parallel merged-batch TP

- **Full-duplex realtime (experimental, v0.26.0)** — MiniCPM-o 4.5 only

- - Native `/v1/duplex` plus a Realtime-compatible `/v1/realtime?duplex=1` (#3907)

- - Streaming audio in/out with cancel, barge-in, overlap policy, playback-aware session state

- - Explicitly a **preview**: no persistent KV leases, no multi-session/multi-replica admission or recovery, no byte-for-byte OpenAI Realtime compatibility. Don't plan production capacity on it.

+ **Full-duplex realtime** — unified engine-owned duplex framework in v0.30.0 (#7413); MiniCPM-o 4.5, PersonaPlex, Qwen3-Omni (`qwen3_omni_duplex.yaml`)

+ - `/v1/realtime?duplex=1`; `/v1/duplex` is an alias of it. Needs a duplex plugin and a deploy YAML with `session_mode: duplex` — the default `qwen3_omni_moe.yaml` is a turn deployment

+ - **v0.30.0 client contract break:** `native_duplex` and the experimental client params are gone; use `auto_response` (the `full_duplex` alias was removed). Update clients written against v0.26–v0.28

+ - Streaming audio in/out with cancel, barge-in, playback-aware session state, session resume. `docs/serving/realtime_duplex_api.md` is the wire contract

**Qwen3-Omni realtime + Qwen3-TTS** → `references/realtime-tts.md`

- - PCM16 mono @ 16 kHz in / 24 kHz out, OpenAI realtime event shape

- - `async_chunk: false` requirement

+ - PCM16 mono @ 16 kHz in / 24 kHz out, OpenAI realtime event shape (non-beta names since v0.30.0)

+ - `async_chunk` no longer blocks realtime (v0.22.0+)

- Qwen3-TTS CustomVoice / VoiceDesign / Base modes, 12 Hz / 25 Hz tokenizers

- Voice-upload surface (10 MB cap, consent/ref_text/speaker_description required)

## The top operator mistakes this skill exists to prevent

- - **`/v1/realtime` with `async_chunk: true`**. The realtime WebSocket rejects at connection if `async_chunk` is enabled (api_server.py:1208). Use the default stage-config (`vllm_omni/deploy/qwen3_omni_moe.yaml`) — **not** the `...moe_async_chunk.yaml` variant — for realtime sessions. The async-chunk config is for higher-throughput non-realtime Qwen3-Omni serving.

- - **Qwen3-TTS with CUDA graphs on (v0.18 only)**. Issue #2866: on v0.18 the code2wav stage crashed when `enforce_eager: false`, so `--enforce-eager` was mandatory. **#2866 is CLOSED (2026-04-29)** and v0.20.0 ships TTS CUDA-graph capture + shared memory pools (release notes cite #2690/#2758/#2803), lifting the requirement. On v0.20.0+ keep `--trust-remote-code` but `--enforce-eager` is no longer forced — drop it to regain CUDA-graph throughput, and re-test latency.

+ - **Old realtime advice: "turn `async_chunk` off for `/v1/realtime`"**. True only on **v0.20.x and older**, where the WebSocket rejected at connection when `async_chunk` was on. PR #3614 (v0.22.0) removed that gate and added streaming audio input under async-chunk. On v0.22.0+ realtime works with the default `vllm_omni/deploy/qwen3_omni_moe.yaml`, which ships `async_chunk: true`; `--no-async-chunk` / `--async-chunk` override the YAML (v0.28.0+). `qwen3_omni_moe_async_chunk.yaml` no longer exists — do not look for it.

- - **Running the v0.19.0rc1 FLUX artifacts**. Issue #2730: FLUX.1-dev generated incorrect images in v0.19.0rc1 (T5 text-encoder bug). **Fixed in v0.20.0 stable** (PR #2760, merged 2026-04-24). The v0.19.0rc1 tag artifacts are still broken, so do not deploy that specific tag — use v0.20.0+ for any FLUX deployment.

+ - **Realtime clients parsing beta event names on v0.30.0+**. v0.30.0 emits the current OpenAI names (#7339, #7426): `response.output_audio.delta` / `response.output_audio.done` replace `response.audio.delta` / `response.audio.done`. A client keyed on the old names receives nothing and writes silence. Transcripts arrive as `transcription.delta` / `transcription.done`.

- - **GLM-Image on v0.18 without `transformers>=5.0`** — historical only, resolved. On v0.18 GLM-Image needed a manual `pip install 'transformers>=5.0'` because the wheel pinned transformers below 5.0 and the model silently failed to load. v0.20.0 shipped Transformers 5.x compat fixes, and **v0.26.0's own `requirements/common.txt` floors transformers at `>= 5.5.3`** (kept deliberately aligned with the upstream vLLM constraint). No manual upgrade on any current build.

+ - **v0.30.0 config removals that fail at load.** `diffusion_batch_size` is gone — use `max_num_seqs` (#6484). Unknown diffusion config fields now raise with the field name and stage id instead of being silently dropped (#5172) — a stale or misspelled key that used to be ignored now stops the server. First-party Dynin-Omni and dots.tts support was removed (#7655).

- **PCM format on `/v1/realtime`**. Qwen3-Omni realtime hard-expects **16-bit PCM mono @ 16 kHz input**, outputs PCM at 24 kHz. Stereo, 8 kHz, 24-bit, or WAV-with-header inputs produce garbage or silent failures. Use the reference client in `examples/online_serving/qwen3_omni/openai_realtime_client.py` as a template.

- **`guidance_scale=0` was silently ignored below v0.26.0**. `OmniDiffusionSamplingParams.guidance_scale` used `0.0` as its "unset" default, so an explicit `0` could not be distinguished from an omitted value and the pipeline's own default was substituted instead — HunyuanImage-3.0's `5.0`, for example, which *enables* CFG. Not model-specific: the collision was in the shared input layer, so every pipeline that substitutes a default was affected. **Fixed in v0.26.0** (#4999, fixes #4998) by making `None` the sentinel and testing presence by identity. On v0.26.0+ `guidance_scale=0` means what it says. On older builds, omit the field to disable CFG rather than sending `0`. The CFG gate is still `guidance_scale > 1.0`, so a value like `0.5` disables CFG on every version.

- - **Prefix caching on a stage that emits latents**. Any stage with `engine_output_type: latent` (thinker stages producing hidden states) must set `enable_prefix_caching: false` in its `engine_args`. Prefix cache reuses token-level blocks, which makes no sense for latent outputs — leaving it on surfaces as intermittent stale responses.

+ - **Prefix caching on a stage that emits latents**. Any stage with `engine_output_type: latent` (thinker stages producing hidden states) must set `enable_prefix_caching: false` on that stage (a flat per-stage key in the deploy YAML; the shipped Qwen3-Omni YAML sets it on every stage). Prefix cache reuses token-level blocks, which makes no sense for latent outputs — leaving it on surfaces as intermittent stale responses.

- - **`/v1/videos/sync` for long jobs**. The sync endpoint has a hardcoded `VIDEO_SYNC_TIMEOUT_S` (default ~1200s) and returns 504 past that. Long Wan2.2 / HunyuanVideo-1.5 jobs should use `POST /v1/videos` (async), then poll `GET /v1/videos/{id}` and fetch `/content`.

+ - **`/v1/videos/sync` for long jobs**. The sync endpoint times out at `VIDEO_SYNC_TIMEOUT_S` — **600 s** by default, overridable with `VLLM_OMNI_VIDEO_SYNC_TIMEOUT` (v0.24.0+; a fixed 600 s before that) — and returns 504 past it. Long Wan2.2 / HunyuanVideo-1.5 jobs should use `POST /v1/videos` (async), then poll `GET /v1/videos/{id}` and fetch `/content`.

- **Orphan processes after a Wan2.2 crash**. Issue #2768: killing one Wan2.2 worker leaves sibling stage processes alive. Wrap launches in a process group + `pkill -9` sweep on failure, or use `systemd`'s `KillMode=control-group`. **#2768 now reads CLOSED/COMPLETED (2026-05-16) but keep the mitigation** — the last comment on the thread (2026-05-12, four days before closure) is a *fresh reproduction* by a different reporter, with no fix PR referenced. Treat the closure as bookkeeping, not as a fix.

- - **Two v0.26.0 breaking changes that rename or remove things you may be using.** (1) **LTX pipeline registry names changed** (#5148): `LTX2Pipeline` is now the single one-stage entry for LTX-2 *and* LTX-2.3, T2V *and* I2V — checkpoint metadata picks the version, `image=` picks I2V. `LTX2ImageToVideoPipeline`, `LTX23Pipeline` and `LTX23ImageToVideoPipeline` were removed; the distilled two-stage path is now `LTX2DistilledPipeline`. Expect higher compute too: the official guidance recipe runs up to four transformer passes per denoise step (cond / uncond / STG perturbation / cross-modality). (2) **GGUF diffusion support moved out of tree** (#4769) to `vllm-project/vllm-gguf-plugin` — a GGUF DiT deployment now needs that plugin installed, and the in-core path is gone.

+ - **Two v0.26.0 breaking changes that rename or remove things you may be using.** (1) **LTX pipeline registry names changed** (#5148): `LTX2Pipeline` is now the single one-stage entry for LTX-2 *and* LTX-2.3, T2V *and* I2V — checkpoint metadata picks the version, `image=` picks I2V. `LTX2ImageToVideoPipeline`, `LTX23Pipeline` and `LTX23ImageToVideoPipeline` were removed. On v0.30.0 the supported list names `LTX2TwoStagePipeline`, `LTX2DistilledOneStagePipeline` and `LTX2DistilledTwoStagePipeline` (`LTX2DistilledPipeline` is still registered). Expect higher compute too: the official guidance recipe runs up to four transformer passes per denoise step (cond / uncond / STG perturbation / cross-modality). (2) **GGUF diffusion support moved out of tree** (#4769) to `vllm-project/vllm-gguf-plugin` — a GGUF DiT deployment now needs that plugin installed, and the in-core path is gone.

- **Qwen3-TTS `max_model_len` validation error**. Issue #2595 (closed 2026-04-28): serving fails when `max_model_len` exceeds the derived maximum. The recorded workaround is `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1` (see PR #2508). The thread closed on that workaround rather than on a root-cause fix, so expect it to still be needed.

- **Assuming vllm-omni serves text-only models**. If the model has no multimodal output, use stock vLLM — vllm-omni adds overhead for features a text-only model won't exercise, and the community skill explicitly recommends against it. The decision rule: output modality is non-text OR the model name ends `-Omni`/`-Image`/`-TTS`/`-Video` → vllm-omni; otherwise stock vLLM.

## Operator cheat sheet

### Install

```bash

uv venv --python 3.12 --seed

source .venv/bin/activate

# CUDA — pin upstream vLLM to the matching minor:

- uv pip install vllm==0.26.0 --torch-backend=auto

+ uv pip install vllm==0.30.0 --torch-backend=auto

- # ROCm — note the index moved to rocm723:

- uv pip install vllm==0.26.0+rocm723 \

- --extra-index-url https://wheels.vllm.ai/rocm/0.26.0/rocm723

+ # ROCm (vLLM wheel is cp312-only):

+ uv pip install vllm==0.30.0+rocm723 \

+ --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723

# Then the omni package (prebuilt wheel OR editable clone):

- uv pip install vllm-omni==0.26.0

- # OR: git clone https://github.com/vllm-project/vllm-omni && cd vllm-omni && uv pip install -e .

+ uv pip install vllm-omni==0.30.0

+ # OR: git clone -b v0.30.0 https://github.com/vllm-project/vllm-omni && cd vllm-omni && uv pip install -e .

```

- **Do not reach for upstream vLLM 0.27.x here.** vllm-omni v0.26.0 is rebased on

- vLLM **0.26.0** and the minor must match, so this stack is on **PyTorch 2.11.0

- and FlashInfer 0.6.14** (vLLM v0.26.0 `requirements/cuda.txt`) — not the 2.13.0

- / 0.6.16 pair that upstream v0.27.0 moved to. Transformers floor is

- `>= 5.5.3`, kept deliberately aligned with upstream (vllm-omni

- `requirements/common.txt`), alongside `diffusers==0.38.0`.

+ **Do not reach for upstream vLLM 0.31.x / 0.32.x here.** vllm-omni v0.30.0 is

+ rebased on vLLM **0.30.0** and the minor must match, so this stack is on

+ **PyTorch 2.13.0, FlashInfer 0.6.18.post1, torchvision 0.28.0** (vLLM v0.30.0

+ `requirements/cuda.txt`) — vLLM v0.31.0/v0.32.0 moved to FlashInfer 0.7.0.post1.

+ vllm-omni `requirements/common.txt` pins `transformers >= 5.13.0, < 5.15` (an

+ upper bound: 5.15 has a reported model-construction regression) and

+ `diffusers==0.40.0`.

- Python **3.12 is required** (3.11 is not supported). Docker image:

- `vllm/vllm-omni:v0.26.0` (`-x86_64` / `-aarch64` variants published per tag);

- ROCm images live in a **separate** repo, `vllm/vllm-omni-rocm:v0.26.0`. Pin the

- exact tag: `latest` tracks an unversioned nightly, not a release. Model-specific

- builds also appear on the tag list (`cosmos3`, `minimax-h3`, `minimax-h3-cu129`)

- — those are not releases.

+ Python: the package declares **3.10–3.13** (`requires-python >=3.10,<3.14`);

+ the quickstart uses 3.12 and the ROCm vLLM wheel is cp312-only, so default to

+ 3.12. Docker image: `vllm/vllm-omni:v0.30.0` (`-x86_64` / `-aarch64` variants

+ per tag). ROCm images live in a **separate** repo, `vllm/vllm-omni-rocm`, whose

+ newest stable is **v0.28.0** — a ROCm container on v0.30.0 does not exist; use

+ the wheel path. Pin the exact tag: `latest` tracks the newest tag, rc included.

+ Model-specific tags (`minimax-h3`, `qwen-image21`, `flux-3-image-klein`, …) and

+ rolling `nightly` / `nightly-<commit>` tags are not releases.

### Serving canonical forms

```bash

# Text-to-image (default Z-Image-Turbo quickstart):

vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091

# Qwen-Image with tensor parallelism:

vllm serve Qwen/Qwen-Image --omni --tensor-parallel-size 2 --port 8091

- # Qwen3-Omni realtime (default stage config, async_chunk OFF):

- vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \

- --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --port 8091

+ # Qwen3-Omni (registry loads vllm_omni/deploy/qwen3_omni_moe.yaml: stage 0 on

+ # cuda:0, stages 1+2 share cuda:1, async_chunk ON; serves chat + /v1/realtime).

+ # No global --gpu-memory-utilization: it applies to every stage and

+ # oversubscribes the shared GPU — tune per stage with --stage-overrides.

+ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

- # Qwen3-Omni high-throughput non-realtime (async_chunk ON):

- vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \

- --stage-configs-path vllm_omni/deploy/qwen3_omni_moe_async_chunk.yaml

+ # Same, with chunk-level stage overlap off (overrides the YAML):

+ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --no-async-chunk

- # Qwen3-TTS (trust-remote-code; --enforce-eager only required on v0.18, lifted by TTS CUDA-graph capture in v0.20.0+):

+ # Qwen3-TTS (--trust-remote-code required; no --enforce-eager, CUDA graphs work):

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \

--trust-remote-code --task-type CustomVoice

# Wan2.2 T2V with Ulysses sequence parallel:

vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni \

--ulysses-degree 4 --ulysses-mode strict --port 8091

```

### Common extra flags

| Flag | Purpose |

|---|---|

| `--omni` | Enable vllm-omni entrypoint (load-bearing) |

- | `--deploy-config` | Stage-config YAML. **`--stage-configs-path` and the legacy `stage_args` loader were REMOVED at v0.28.0** — with them went `vllm_omni/deploy/qwen3_omni_moe_async_chunk.yaml`. Use `--deploy-config` plus the `--async-chunk` bool override. |

+ | `--deploy-config` | Stage-config YAML. **`--stage-configs-path` and the legacy `stage_args` loader were REMOVED at v0.28.0** — with them went `vllm_omni/deploy/qwen3_omni_moe_async_chunk.yaml`. Use `--deploy-config` plus the `--async-chunk` / `--no-async-chunk` override. |

+ | `--stage-overrides` | Per-stage JSON, e.g. `'{"0": {"gpu_memory_utilization": 0.8}}'` — use instead of global engine flags on multi-stage models |

| `--ulysses-degree` / `--usp` | Ulysses sequence parallelism for DiT |

| `--ulysses-mode` | `strict` (divisibility) \| `advanced_uaa` (uneven shapes) |

| `--ring-degree` | Ring-based parallelism |

| `--num-gpus` | GPUs allocated to diffusion pipeline |

| `--omni-master-address` / `-oma` | Orchestrator hostname (multi-node) |

| `--omni-master-port` / `-omp` | Orchestrator port |

| `--stage-id` | Single-stage mode (requires master address) |

| `--worker-backend` | `multi_process` \| `ray` |

| `--model-class-name` | Override diffusion pipeline class |

### Key numbers to memorize

| Metric | Value |

|---|---|

- | Current stable, all channels | **v0.26.0** (2026-08-03), rebased on vLLM 0.26.0 (#5443) |

- | Docker `latest` resolves to | the unversioned `vtest-nightly` build — matches no release, pin the exact tag |

- | Latest pre-release | v0.26.0rc1 (2026-07-28) |

- | Stables in the line | v0.14.0, v0.16.0, v0.18.0, v0.20.0, v0.22.0, v0.24.0, v0.24.1, v0.26.0, v0.28.0 (v0.21/v0.23/v0.25/v0.27 are rc1-only) |

- | First stable | v0.14.0 (2026-01-31) |

- | Minimum Python | 3.12 |

- | Runtime pins (via vLLM 0.26.0) | PyTorch 2.11.0, FlashInfer 0.6.14, transformers >= 5.5.3, diffusers 0.38.0 |

- | transformers pin on current main | **`transformers >= 5.13.0, < 5.15`** — an upper bound, not just a floor. `requirements/common.txt` gives the reasons inline: unified multimodal model classes below the floor, a `transformers_keys_to_ignore_compat` change in 5.9, and `hub_kernels.LayerRepository` needing 5.13.0 (read 2026-09-15) |

+ | Python | 3.10–3.13 declared; use 3.12 (ROCm wheel is cp312-only) |

+ | Runtime pins (via vLLM 0.30.0) | PyTorch 2.13.0, FlashInfer 0.6.18.post1, `transformers >= 5.13.0, < 5.15`, diffusers 0.40.0 |

| `/v1/realtime` input | PCM16 mono @ 16 kHz |

| Qwen3-Omni audio output rate | 24 kHz |

| Qwen3-TTS tokenizer rate | 12 Hz or 25 Hz |

- | `/v1/videos/sync` timeout | ~1200s (hard) |

+ | `/v1/videos/sync` timeout | 600 s default; `VLLM_OMNI_VIDEO_SYNC_TIMEOUT` overrides (v0.24.0+) |

| Voice upload size cap | 10 MB |

| Paper claim | up to 91.4% JCT reduction vs "baseline" (unspecified) |

| Qwen3-TTS published RTF (v0.16) | 0.22–0.45 |

| MiMo-Audio published RTF (v0.16) | ~0.2 (11× baseline) |

## Paired skills

- **`vllm-input-modalities`** — the complement: text embeddings, reranking, STT (Whisper/Voxtral-STT/Qwen3-ASR), OCR (DeepSeek-OCR). Trigger together when the deployment does both input and output non-text modalities.

- **`vllm-nvidia-hardware`** — for sizing GB300/NVL72/Rubin capacity for diffusion + CFG-parallel + Ulysses footprints.

- **`vllm-caching`** — OmniConnector borrows Mooncake from upstream vLLM; the caching skill has the connector-config surface.

- **`vllm-observability`** — vllm-omni inherits upstream `/metrics`; profiler hooks (`OmniTorchProfilerWrapper`) add stage_id + rank awareness to trace files.

## Source policy

- All claims are cited with file:line, release-note PR refs, or issue IDs. Full anchor list + community channels + third-party plugin catalog in `references/sources.md`. Compiled 2026-04-18 against v0.18.0; freshened 2026-05-28 (v0.20.0) and 2026-07-21 (v0.24.0/v0.24.1, channel mismatch, six issue closures re-classified). **Last freshened 2026-08-11** — rebased to v0.26.0, recorded its two breaking changes (LTX registry renames, GGUF out of tree), and closed the long-open `guidance_scale=0` and GLM-Image `transformers` questions against source.

+ All claims are cited with file:line, release-note PR refs, or issue IDs. Full anchor list, version history, community channels and third-party plugin catalog in `references/sources.md`.

- **Known gap, now larger:** the model roster in `references/models.md` has not been re-synced against `docs/models/supported_models.md` since 2026-04-18 — five minors. On top of the v0.22.0/v0.24.0 additions (Cosmos3, DreamZero, Higgs Audio V3, IndexTTS2, Step-Audio2, SDXL, GR00T-N1.7, MiniCPM-o 4.5), v0.26.0 adds MiniMax H3 (joint video+audio), Krea 2, Boogu Image 0.1, Nemotron Audex, LingBot Video, MammothModa2-Dev, Cosmos3 Edge/Distilled and MOSS-TTS-Local v1.5. Treat that file as a floor, not a complete list.

+ **Known gap:** the model tables in `references/models.md` are a floor, not a complete list — check `docs/models/supported_models.md` at the deployed tag before calling a model unsupported.