vllm - 💡(How to fix) Fix GDN attention backend crashes with ngram speculative decoding on mixed decode batches [3 comments, 2 participants]

Description

The GDN (Gated Delta Network) attention backend in vllm/v1/attention/backends/gdn_attn.py crashes with an AssertionError when ngram speculative decoding produces a batch containing both regular decode tokens and speculative decode tokens.

Environment

• vLLM version: 0.17.2.dev0+g95c0f928c.d20260313 (nightly)
• GPU: NVIDIA GH200 480GB
• Model: Qwen3.5 9B (uses GDN linear attention)
• Config: FP8 online quantization, ngram speculative decoding (128 tokens, prompt lookup)

Error

File ".../vllm/v1/attention/backends/gdn_attn.py", line 310, in build
    assert not (num_decodes > 0 and num_spec_decodes > 0), (
AssertionError: num_decodes: 1, num_spec_decodes: 1

Root Cause

Line 310 in gdn_attn.py:

# Function code counted on either presency non-spec decode or spec decode,
# but not both.
assert not (num_decodes > 0 and num_spec_decodes > 0), (
    f"num_decodes: {num_decodes}, num_spec_decodes: {num_spec_decodes}"
)

When ngram speculative decoding rejects some tokens, vLLM creates a batch with both:

• Regular decode tokens (from rejected/verified sequences)
• Speculative decode tokens (new speculative proposals)

The GDN attention builder assumes these are mutually exclusive, but they're not when spec decoding has partial rejections.

The CUDA graph preparation code below the assertion also assumes mutual exclusivity — there are separate branches for num_decodes == 0 (spec-only) and num_spec_decodes == 0 (decode-only), with no handling for mixed batches.

Behavior

• First 1-2 requests succeed (all spec tokens accepted → pure spec decode batches)
• Subsequent requests crash when a token rejection creates a mixed batch
• Engine dies permanently after the assertion failure

Steps to Reproduce

from vllm import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen3.5-9B",  # Any Qwen3.5 model with GDN layers
    trust_remote_code=True,
    quantization="fp8",
    max_model_len=32768,
    speculative_config={
        "method": "ngram",
        "num_speculative_tokens": 128,
        "prompt_lookup_max": 12,
        "prompt_lookup_min": 2,
    },
)

# Run multiple requests — will crash after 1-2 succeed
for i in range(5):
    output = llm.generate("def hello():\n", SamplingParams(max_tokens=100))

Impact

This effectively makes speculative decoding unusable with Qwen3.5 models (and any model using GDN attention layers), which is a significant performance regression — ngram spec decoding provides signifcant speedups

Expected Behavior

The GDN attention builder should handle mixed batches where both num_decodes > 0 and num_spec_decodes > 0, similar to how the standard attention backends handle this case.