Your current environment

• vLLM version: v0.19.0 (vllm/vllm-openai:v0.19.0-x86_64-cu130-ubuntu2404 with transformers>=5.5.0)
• GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (96GB GDDR7, SM 12.0)
• OS: Linux 5.15.0-171-generic
• CUDA: 13.0
• Python: 3.12

Model

google/gemma-4-E4B-it (dense 4B model, FP8 quantization)

Also applies to any Gemma 4 variant (google/gemma-4-26B-A4B-it, google/gemma-4-31B-it) — and likely any model whose reasoning parser uses channel-style delimiters not present in the prompt.

Command

vllm serve google/gemma-4-E4B-it \
  --quantization fp8 \
  --max-model-len 25600 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --structured-outputs-config '{"backend":"xgrammar"}'

🐛 Describe the bug

When --reasoning-parser gemma4 is specified together with --default-chat-template-kwargs '{"enable_thinking": false}', the xgrammar structured output engine is completely bypassed for every request. Grammar constraints (JSON schema, BNF, etc.) are never enforced. The model generates unconstrained text that happens to look valid because it's well-trained, but the grammar FSM never runs.

This manifests as a dramatic performance difference that led to discovering the bug:

Benchmark data (single GPU, google/gemma-4-E4B-it FP8)

The "with parser" numbers are faster because xgrammar bitmask computation and FSM advancement are silently skipped on every decode step.

Root cause

The bug is in vllm/v1/structured_output/__init__.py, in the interaction between should_fill_bitmask() / should_advance() and the reasoning parser.

Step 1: is_reasoning_end() returns False for prompts without thinking tokens

Gemma4ReasoningParser inherits from BaseThinkingReasoningParser, which scans the prompt backward for <|channel> (start, token 100) or <channel|> (end, token 101):

# vllm/reasoning/basic_parsers.py
def is_reasoning_end(self, input_ids):
    for i in range(len(input_ids) - 1, -1, -1):
        if input_ids[i] == start_token_id:
            return False
        if input_ids[i] == end_token_id:
            return True
    return False  # ← neither found → "reasoning has NOT ended"

With enable_thinking: false, the prompt contains neither <|channel> nor <channel|>. The method returns False — meaning "reasoning has not ended yet."

Step 2: Structured output engine skips grammar enforcement

# vllm/v1/structured_output/__init__.py
def should_fill_bitmask(self, request):
    if self.reasoner is not None:
        if self.enable_in_reasoning:  # default: False
            return True
        return request.structured_output_request.reasoning_ended  # ← False!
    return True  # ← no parser: always fill

def should_advance(self, request):
    if self.reasoner is None:
        return True  # ← no parser: always advance
    if self.enable_in_reasoning:
        return True
    if structured_req.reasoning_ended:  # ← False!
        return True
    # ... checks is_reasoning_end_streaming() for <channel|> in delta ...
    return False  # ← never advances

Step 3: Model never generates <channel|>, so reasoning_ended stays False forever

Since thinking is disabled, the model never outputs the <channel|> end token. The is_reasoning_end_streaming() check on each decode step never finds it. Grammar enforcement is permanently disabled for the entire generation.

Summary

Suggested fix

is_reasoning_end() should return True (not False) when no reasoning tokens are found in the input. If thinking was never started, there is no reasoning to wait for — the model is already in "content" mode.

# Fix in basic_parsers.py
def is_reasoning_end(self, input_ids):
    for i in range(len(input_ids) - 1, -1, -1):
        if input_ids[i] == start_token_id:
            return False
        if input_ids[i] == end_token_id:
            return True
    return True  # ← no thinking tokens found → treat as "not in reasoning"

Alternatively, the structured output engine could handle the "no reasoning tokens in prompt" case explicitly, treating it as reasoning_ended = True.

Related issues

• #37359 — Same root cause for GPT-OSS models with guidance backend (offline path)
• #37362 — Same root cause for Nemotron V3 reasoning parser
• #34650 — Speculative decoding causes </think> detection failure in structured output
• ollama/ollama#15260 — Identical bug for Gemma 4 in Ollama (think=false + format silently ignored)
• ollama/ollama#14645 — Same bug for Qwen 3.5 in Ollama

Impact

Silent correctness issue: Users who configure --reasoning-parser gemma4 with enable_thinking: false (a common production setup to get reasoning parsing without thinking overhead) get zero grammar enforcement on structured output. The output appears correct because the model is well-trained, but the safety guarantee of grammar-constrained decoding is completely lost. There are no warnings or errors logged.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.