vllm - 💡(How to fix) Fix [Bug]: Gemma-4 31B with DFlash speculator produces gibberish/repetitive token loop [1 participants]

Your current environment

 vLLM: 0.20.1rc1.dev64+g5560cac7e (built from source, main branch at commit 5560cac7e2)
  PyTorch: 2.11.0+cu130
  CUDA: 13.0
  Python: 3.10.18
  OS: Linux 5.14.0-620.el9.x86_64 (RHEL 9)
  GPUs: 8x NVIDIA H100 80GB HBM3 (using 2 with -tp 2)
  Driver: 590.48.01

🐛 Describe the bug

Model

RedHatAI/gemma-4-31B-it-speculator.dflash (resolves to google/gemma-4-31B-it with a DFlash speculative decoding head)

How to reproduce

vllm serve RedHatAI/gemma-4-31B-it-speculator.dflash -tp 2
# In a separate terminal:
vllm chat

Then send any message, e.g. "Hello".

Description

When serving RedHatAI/gemma-4-31B-it-speculator.dflash with tensor parallelism 2, responses frequently (but not always) start coherent and then degrade into repetitive token loops / gibberish. Short, simple prompts seem most affected.

Example output

> Hello
Hello! How can I help1 ay1/saj dey-//-o-1sCj?C-o la laay1sCj?C-o laCj?C-oenCjC-oCjC-oCjC-o
CjC-1sCj?C-1sCj1sCj1sCj1Cj1Cj1Cj1Cj1Cj1Cj1Cj1Cj1Cj1Cj1Cj1Cj1Cj1Cj1C1y1y/C/C/C/C/C/C
/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C
/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C/C
...

The output starts with a coherent fragment ("Hello! How can I help") then immediately corrupts into garbled characters and eventually collapses into an infinite /C repetition loop until max tokens or manual interrupt.

What works (narrowing the issue)

• RedHatAI/gemma-4-31B-it-speculator.eagle3 with the same -tp 2 setup: no issue — outputs are coherent. This suggests the problem is specific to the DFlash speculator, not Gemma-4 31B or the speculative decoding pipeline in general.
• z-lab/Qwen3.5-27B-DFlash with the same setup: no issue — outputs are coherent. This rules out a general DFlash bug and points to something specific to the Gemma-4 + DFlash combination.

Notes

• The issue is intermittent — not every response is corrupted, but the majority are.
• Tested via vllm chat; did not test whether the issue also reproduces via the /v1/chat/completions REST API (initial API tests with simple prompts returned correct results, but the intermittent nature makes it hard to rule out).
• The corruption pattern (coherent start → garbled tokens → repetitive loop) suggests the speculative decoding verification/correction step may be accepting incorrect draft tokens.
• No existing open issue appears to cover this specific combination (DFlash speculator + Gemma-4 31B + gibberish output).
• Possibly related to #39071 (Gemma 4 31B weird character output), though that issue is about structured outputs specifically.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.