git:20260811.74582f8 to git:20260915.517712e

2 added, 0 removed. Audit A to A.

---

name: vllm-chat-templates

description: |-

vLLM chat-template (prompt-side Jinja) operator reference. Template resolution precedence (`--chat-template` → AutoProcessor → tokenizer default → bundled fallback), `chat_template_kwargs` allowlist silently dropping `add_generation_prompt`/`enable_thinking`/custom kwargs (PR 27622 fix), 26 shipped `tool_chat_template_*.jinja` files, known template-layer bugs for Qwen3/Qwen3-Coder, DeepSeek-R1/V3/V3.1/V3.2, GPT-OSS, Kimi-K2, Llama-4, Mistral (HF vs mistral mode), Gemma-3/4, Phi-4, GLM. Prompt side only — output parsing lives in sibling skills.

when_to_use: |-

Trigger on "chat template broken", "chat template no effect", "--chat-template ignored", "jinja template error vllm", "apply_chat_template error", "ChatTemplateResolutionError", "tool_chat_template", "which tool_chat_template", "chat_template_kwargs not respected", "add_generation_prompt dropped", "enable_thinking ignored", "continue_final_message", "--chat-template-content-format auto", "tokenizer-mode mistral silent", "harmony format gpt-oss template", "template resolution order", "AutoProcessor chat template skipped when tools", "prompt doesn't match model card", "/v1/chat/completions vs /v1/responses template". TGI/SGLang/TensorRT-LLM-to-vLLM migration where Jinja differs. Also implicit — "audit chat template for {model}", "deploy-memo prompt format", "why does model emit wrong prompt", "jinja check". NOT for output parsing (→ `vllm-reasoning-parsers` / `vllm-tool-parsers`), NOT for KV caching (→ `vllm-caching`).

---

# vLLM chat templates — operator triage

Target audience: operators deploying vLLM in production. Assumes OpenAI-API-compatible frontend (`/v1/chat/completions` or `/v1/responses`), multi-GPU, mid-2024 through 2026 model families.

## Scope — three sibling skills, three layers

Chat-template bugs span three layers. This skill owns the **prompt-rendering** layer; route to a sibling when the problem is output-side.

|---|---|---|---|

| **Chat template (Jinja)** | messages → prompt string | **this skill** | template precedence, `--chat-template`, `chat_template_kwargs`, `examples/tool_chat_template_*.jinja`, bundled fallbacks |

| **Reasoning parser** | model output → `reasoning_content` + `content` | `vllm-reasoning-parsers` | `--reasoning-parser`, `extract_reasoning`, `is_reasoning_end`, `<think>` splitting |

When in doubt: if the operator complains about what the **server receives**, it's this skill; if about what the **client receives**, it's one of the parser skills. Template and parser bugs often present the same symptom ("reasoning_content is null"), so all three skills name each other in diagnostics. This file stays on the Jinja side.

## Why this matters

Chat templates are the **Jinja layer between structured messages and the raw prompt string the model sees**. Tool-calling and reasoning models are extremely sensitive to this layer — a one-character whitespace difference in a template can send tool-call success rate from 99% to 0%, silently.

The hardest class of bug in vLLM deployments is **template silent failure**: the server starts, requests return 200s, but the model emits garbled tool calls, or `reasoning_content` is `null`, or `enable_thinking=false` has no effect. No exception, no log line, just wrong outputs. Operators often chase parser bugs when the real fix is a template override (or vice versa — override a template when the real fix is a parser flag).

This skill's job: given a symptom, identify whether it's a template, a parser, a kwarg, or an API-endpoint issue, and point at the known fix.

## Template resolution precedence (memorize this)

vLLM picks a chat template in this order. First hit wins. Source: `vllm/renderers/hf.py:96-145` (`resolve_chat_template()`).

1. **`--chat-template <path-or-name>` CLI flag** — highest. Path to a `.jinja` file, literal Jinja string, or a named template (`"tool_use"`) that the tokenizer knows.

2. **AutoProcessor chat template** — only consulted when the request has `tools=None`. Multimodal models often expose their template via the processor, not the tokenizer. Source: `hf.py:110-116`.

3. **AutoTokenizer `chat_template`** — the one shipped in `tokenizer_config.json`. Source: `hf.py:119-126`.

4. **Bundled fallback in `vllm/transformers_utils/chat_templates/`** — at v0.27.0, **13 model types** mapping onto 7 shipped `.jinja` files: blip-2, chameleon, clip, colpali, deepseek_ocr, deepseek_ocr2, unlimited-ocr, deepseek_vl_v2, minicpmv, minicpmv4_6, paligemma, siglip, siglip2. **`fuyu` was removed** (the Fuyu model itself was dropped in v0.26.0, #48096) and `template_fuyu.jinja` is gone; there has never been a `qwen` entry. Source: `registry.py:26-39` (`_MODEL_TYPE_TO_CHAT_TEMPLATE_FALLBACK`).

If none resolves, **`ChatTemplateResolutionError`** raised at `hf.py:718` with: *"As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one."*

> **Do not "modernize" that v4.44.** It is vLLM's own error string, verbatim and unchanged on v0.27.0 — it refers to the *transformers release that removed default templates*, not to the version vLLM needs. vLLM v0.27.0's actual runtime floor is **`transformers >= 5.5.3`** (`requirements/common.txt`, unchanged since v0.25.1). Two different facts; don't collapse them.

+ >

+ > **The pin has since moved, and it is still below the security floor.** v0.28.0 is unchanged at `>= 5.5.3`; **v0.29.0 and `main` raise it to `>= 5.10.4`** (read 2026-09-15). Neither reaches **5.15.0**, which is the floor that clears all three 2026 fixes in the tokenizer-loading path — including one where a hostile repo's **chat-template names** become filenames and `save_pretrained` writes outside the target directory. So satisfying vLLM's requirement is not the same as being patched: pin `transformers` yourself. Detail and the per-fix table: `transformers-config-tokenizers-expert` § "Loading a tokenizer is a trust decision".

### Consequences operators miss

- **Passing `--chat-template` does NOT override the tokenizer's template if the path is invalid** — vLLM raises *"The supplied chat template ... appears path-like, but doesn't exist!"* (`chat_utils.py:1105-1109`). Check `ls` first.

- **Tool-calling disables the AutoProcessor step.** If a multimodal model has a processor template but the request has `tools`, vLLM skips it and falls through to tokenizer. This is *intentional* — tool templates must be explicit.

- **`--tokenizer-mode mistral` silently ignores `--chat-template`.** Warning only; see Mistral section in `references/model-families.md`. Issue vllm-project/vllm#25401 — closed as COMPLETED 2025-10-09. Behavior is at least louder in current vLLM; verify on your version before assuming silent-drop. Verified 2026-04-24.

- **Some templates (e.g. gpt-oss) appear hard-coded** — operator-supplied `--chat-template` has no effect (vllm-project/vllm#23015, closed NOT_PLANNED 2026-04-20). Intentional: gpt-oss goes through the Harmony path via `/v1/responses`; `--chat-template` is not the intended escape hatch. Verified 2026-04-24.

## Triage: symptom → suspect

Use this table first. Load `references/model-families.md` for the specific model bug + flags.

| Symptom | First suspect | Second suspect |

|---|---|---|

| `reasoning_content` is `null` or missing from response | `--reasoning-parser` flag missing/wrong | Template strips opening `<think>` before model generates it (DeepSeek-R1) |

| `tool_calls` empty array `[]` but model emitted text | `--tool-call-parser` wrong for model family | `tool_choice="auto"` (try `"required"` or a named tool) |

| Tool calls have malformed JSON / `arguments` empty | Wrong tool-call parser (Hermes vs pythonic vs XML vs model-specific) | Template doesn't inject `tools` correctly |

| `enable_thinking=False` has no effect | Kwarg dropped by vLLM allowlist (need PR #27622 or ≥v0.11.1) | Downstream harness not forwarding `chat_template_kwargs` |

| Tool calling works on `/v1/completions` but not `/v1/chat/completions` | Chat template doesn't render tool result (GLM-5.1 #39611, fixed 2026-04-12) | Endpoint-specific parser path |

| Prompt in server log doesn't match model card | Wrong template resolving — check precedence order above | `--chat-template` pointing at file that doesn't exist (silent for some tokenizer modes) |

| `<think>` content leaks into final output | `--reasoning-parser` missing | `skip_special_tokens=true` stripping control tokens before parser (#38855) |

| Garbled prompt with repeated `<image>` placeholders | Template has placeholder AND multimodal parser adds one | `interleave_strings` mismatch — see `chat_utils.py:1186-1241` |

| `IndexError` / Jinja exception on multi-turn tool calls | Tool-call ID format regex too strict (Kimi-K2, Mistral) | Historical `tool_calls=[]` in assistant msg crashes template (Llama #13978) |

| Server starts, first request works, subsequent requests fail | State-ful template bug: leading whitespace accumulating (DeepSeek V3.1 #28804) | Concurrent-request regression (Gemma-4 `<pad>` tokens #39392) |

| "this model is reasoning but reasoning_effort=none" | gpt-oss #37909 — `"none"` not honored | Wrong reasoning parser |

| Works on SGLang / TGI, broken on vLLM | vLLM applies template differently (HF precedence + kwarg allowlist) | Check `--tokenizer-mode` differences |

Template, parser (output), and kwarg (`chat_template_kwargs`) are orthogonal layers — diagnose one at a time, don't change multiple at once.

## The flag matrix (common case)

Per-family flag matrix lives in `references/flags-matrix.md`. Quick cheat-sheet for the production-critical ones:

|---|---|---|---|

Commit-gated models (Kimi-K2) require minimum HF revision — check `references/model-families.md` before recommending.

## Top 15 patterns that silently break production

Consult this list for any "broken chat template" symptom. Most reports reduce to one of these.

1. **Chat template silently ignored.** Mistral in `--tokenizer-mode mistral` (#25401), reportedly gpt-oss (#23015). Warning-only. Symptom: edit template, nothing changes. Fix: switch to HF tokenizer mode, or accept that the built-in Mistral template is canonical.

2. **`skip_special_tokens=true` strips control tokens before reasoning parser.** Qwen3 reasoning, gpt-oss harmony — and formerly Gemma-4 (#38855, **fixed; closed 2026-06-15**). Fix: `extra_body={"skip_special_tokens": false}` per request. For Gemma-4 the real fix is two-sided — see the §Gemma-4 note in `model-families.md` before assuming a vLLM upgrade is enough.

3. **`<think>` opening tag prepended to prompt → never appears in output → `reasoning_content=null`.** DeepSeek-R1 (HF #144, vLLM #12999). Template forces thinking by inserting `<think>\n` in the prefill; parser sees only `</think>`. Fix: post-process prepend `<think>`, or use patched template.

4. **Empty string `''` wrapped to `[{'type':'text','text':''}]`.** Broke Kimi-K2 templates expecting strings. Any template that doesn't type-check `message.content` is at risk. Fix: updated chat template or vLLM version.

5. **Tool-call ID regex mismatch.** Mistral requires 9 digits; Kimi-K2 required `functions.X:idx` exactly. vLLM generates longer IDs → `IndexError` → all tool calls silently dropped. Fix: Kimi-K2 fixed in PR #27565; Mistral template truncates to last 9 digits.

6. **`add_generation_prompt` dropped by kwarg allowlist.** PR #25794 restricted `**kwargs` forwarding for security; tokenizers relying on non-standard params broke. Fix: PR #27622 whitelists standard HF params — requires vLLM ≥ v0.11.1.

7. **Jinja template from disk triggers `_try_extract_ast()` exception.** #14884: `\n` escape sequences double-converted when shell-interpolated. Fix: pass the file path, not the contents; don't shell-escape.

8. **`tool_calls=[]` in historical assistant message crashes template.** #13978 (Llama). Templates assume nonempty if present. vLLM now drops empty tool_calls (`chat_utils.py:1586-1589`).

9. **`tool_choice="auto"` vs `"required"` divergence.** `auto` = free-form generation + regex parse (schema-free, brittle). `required`/named = structured-outputs FSM (schema-valid). Many "tool calling is broken" reports vanish when switching to `required`.

10. **Multi-turn whitespace accumulation.** DeepSeek V3.1 (#28804, closed NOT_PLANNED 2026-03-23) — leading whitespace grows each turn. Look for this when assistant replies get progressively indented. Upstream not planning a fix; apply client-side mitigation (strip leading whitespace on each turn) or use `deepseek_v31` parser commits post-issue report. Verified 2026-04-24.

11. **`reasoning_effort="none"` silently breaks output.** gpt-oss #37909. Fix: omit or use `"low"`.

12. **Parallel *tool-call* requests → `<pad>` tokens.** Gemma-4 #39392, **still open and reproducing on `vllm/vllm-openai:v0.25.1` as of 2026-07-20** — so "fix: version bump" is no longer available advice. The reporter narrows the trigger: Gemma-4 is stable under load until *parallel tool-call* requests arrive; serialising those avoids it (Ampere, RTX 3090/A6000). Workaround `--max-num-seqs 1`, or serialise only the tool-calling path rather than throttling the whole server.

13. **`--chat-template-content-format auto` misroutes tool results.** GLM-5.1 #39614 (CLOSED/COMPLETED 2026-04-25): tool result as `{"type":"text","text":"..."}` hits an "else" branch in a template checking `.name` → renders `<tools></tools>` instead of result. Fixed upstream; the `--chat-template-content-format openai`/`string` workaround is only needed on vLLM before the fix.

14. **`/v1/chat/completions` vs `/v1/responses` divergence.** gpt-oss tool calling works only on `/v1/responses` (#22578). GLM-5.1 (#39611, CLOSED/COMPLETED 2026-04-12) — tool results were ignored on chat-completions but work on completions; fixed upstream, so on patched vLLM tool results now render on `/v1/chat/completions`.

15. **API field name mismatch.** The **response** field on current vLLM is `reasoning`, *not* `reasoning_content` — verified at v0.27.0: `ChatMessage.reasoning` (`chat_completion/protocol.py:71`), `DeltaMessage.reasoning` (`engine/protocol.py:397`). The **request** side still accepts `reasoning_content` and normalizes it (RFC #27755). A client reading `reasoning_content` off a response sees `null` every time even when the parser ran perfectly — this looks exactly like a parser failure. Check with `jq '.choices[0].message | keys'` before debugging parsers. (An earlier revision of this skill claimed vLLM "settled on `reasoning_content`"; that was backwards.)

## What `chat_template_kwargs` accepts

Threaded as `extra_body={"chat_template_kwargs": {...}}` in OpenAI client. Source: `vllm/renderers/params.py:70-125`, `hf.py:407-435`.

- **Any Jinja variable** referenced in the resolved template — detected via AST parse at `hf.py:429`.

- **HF `apply_chat_template` params** auto-detected from tokenizer signature (`hf.py:387-404`).

- `add_generation_prompt` (bool)

- `continue_final_message` (bool) — mutually exclusive with `add_generation_prompt=True` (enforced in `chat_completion/protocol.py:964`). **Frontend-dependent:** almost no HF chat template actually reads this variable, so on the Python renderer it is close to a no-op. The **Rust frontend** implements Transformers-v5 semantics properly ([#47844](https://github.com/vllm-project/vllm/pull/47844), v0.26.0) — it appends a sentinel to the last message and strips it from the rendered output, so the template never needs to know. The Rust Harmony renderer rejects it outright ("Harmony renderer does not support continue_final_message"). If it silently does nothing, check which frontend you are on before editing the template.

- `documents` (list of `{title, contents}`) for RAG templates

- `enable_thinking` (Qwen3 and some others)

- **Reserved (will raise `ValueError`):** `chat_template`, `tokenize` (`hf.py:416-421`).

- **Unknown kwargs silently filtered** (`hf.py:434-435`) — if `enable_thinking` doesn't appear in the detected Jinja vars, it's dropped without warning. This is a confusing failure mode: operator sets it, sees no effect, can't find an error.

Merging: server defaults (serving.py:111) ← request kwargs (params.py:28-40) — request wins.

## Reasoning parser vs chat template (they're confused a lot)

- **Chat template** (this skill) controls the *prompt* — where `<think>` tags go going *in* to the model, and how tool definitions get rendered.

- **Reasoning parser** (`vllm-reasoning-parsers`) extracts `<think>` content from *output* into `reasoning_content`. Template-side causes of null `reasoning_content` are pattern #3 above (opening `<think>` prepended in prefill); all other root causes live in the parser skill.

- **Tool-call parser** (`vllm-tool-parsers`) extracts `tool_calls[]` from *output*. Template-side causes of empty `tool_calls` are wrong tool template or `tool_choice="auto"` vs `"required"` (pattern #9); parser-selection and streaming state-machine debugging live in that skill.

Raw-output capture (`--disable-log-requests=False` + no `--tool-call-parser`/`--reasoning-parser`) determines which layer is at fault: if the model emits the expected tags/JSON, the prompt is fine and the problem is parser-side; if not, the template is misrendering.

## Multimodal template notes

Models use model-specific placeholders (`<image>`, `<|image_start|>`, etc.) — vLLM's internal placeholder is `<##IMAGE##>` / `<##AUDIO##>` / `<##VIDEO##>` (`chat_utils.py:96-100`).

- If request has images but template has no placeholder, vLLM **prepends** them to final prompt (`chat_utils.py:1233-1240`). Some models tolerate this; others produce garbage.

- If template has more placeholders than media items: `ValueError: "Found more '<image>' placeholders in input prompt than actual multimodal data items."` (`chat_utils.py:1226-1229`).

- `interleave_strings=True` (via `multimodal_config.interleave_mm_strings`) enables in-order substitution; default is prepend-all.

For Pixtral, Mistral-Small-3.1 multimodal, and other MM-tool-calling combinations, see `references/model-families.md`.

## When nothing works — full debugging workflow

Load `references/debugging.md`. The short version:

1. **Log the actual prompt.** `--disable-log-requests=False` + log level DEBUG. Compare against model card example. If they differ, template is wrong.

2. **Bypass vLLM.** Load the same tokenizer in a Python REPL, call `tokenizer.apply_chat_template(messages, add_generation_prompt=True)`, compare. If same bad output, template is wrong. If different, vLLM's kwarg forwarding is eating something.

3. **Capture raw model output.** Set `--tool-call-parser` to none, read what the model actually emits. Then work backward to pick the right parser.

4. **Pin versions.** Record `vllm --version`, `transformers.__version__`, model revision SHA. Many template bugs track a specific `tokenizer_config.json` commit — mention in bug reports.

## Code locations (for reading vLLM source)

Line numbers below re-probed at **v0.27.0**; they drift every release, so treat them as hints and grep the symbol.

- `vllm/entrypoints/chat_utils.py` — message parsing, multimodal placeholder injection, `ChatTemplateResolutionError`.

- `vllm/renderers/hf.py:257` — `resolve_chat_template()` precedence (was cited as 96-145).

- `vllm/renderers/hf.py:665-728` — `safe_apply_chat_template()` overloads + error surface; the `ChatTemplateResolutionError` raise is at **718** (was cited as 477, which is now unrelated system-message code).

- `vllm/renderers/hf.py:633` — `resolve_chat_template_kwargs()` allowlist (was cited as 407-435).

- `vllm/renderers/hf.py:558` — `resolve_chat_template_content_format()`.

- `vllm/renderers/params.py` — `ChatParams` dataclass.

- `vllm/reasoning/__init__.py` — registered reasoning parsers (29 names at v0.27.0).

- `vllm/transformers_utils/chat_templates/registry.py:26-39` — the `_MODEL_TYPE_TO_CHAT_TEMPLATE_FALLBACK` dict; `get_chat_template_fallback_path()` at 57-68 (was cited as 64-75).

- **`vllm/entrypoints/openai/serving_chat.py` no longer exists.** Parser instantiation is in `vllm/entrypoints/openai/chat_completion/serving.py`.

- `examples/tool_chat_template_*.jinja` (**26** files at both v0.25.1 and v0.27.0 — the "27" previously claimed here was never right) — ready-made tool templates per model family.

## Per-family deep-dive

When the operator names a model family, load `references/model-families.md` — it has per-family flag recipes, minimum commit SHAs, known bugs with issue numbers, and recovery strategies.

## Further reference material

- `references/model-families.md` — Qwen3, DeepSeek, GPT-OSS, Kimi-K2, Llama-4, Mistral, Gemma, Phi-4, GLM, Hermes/Granite/Hunyuan — with inline issue URLs.

- `references/debugging.md` — stepwise debugging workflow + full error-message catalog.

- `references/flags-matrix.md` — quick CLI-flag lookup per model.

- `references/sources.md` — verification log for external URLs/issues/PRs cited in this skill, with `Last verified` dates.

Last verified: 2026-08-11 against vLLM **v0.27.0** (see `references/sources.md` for per-ref status).