vllm - 💡(How to fix) Fix [Bug] DFlash speculative decoding fundamentally incompatible with all KV cache quantization (fp8, turboquant) due to non-causal attention requirement [1 participants]

Summary

DFlash speculative decoding (introduced in v0.20.0 via DFlashProposer / DFlashQwen3ForCausalLM) requires non-causal cross-attention for the draft model. We have empirically verified that every attention backend in v0.20.0 either rejects non-causal attention entirely OR rejects KV-quant dtypes when non-causal is set. This means DFlash spec decode cannot compose with any of fp8_e5m2, fp8_e4m3, or turboquant_4bit_nc — it is locked to bfloat16 KV cache. On a 24 GiB RTX 3090 with a 27B-class target model, this halves the available KV pool (~28K → ~14K tokens), making DFlash impractical for production long-context use.

Verified compatibility matrix on v0.20.0

Confirmed from source inspection inside the running container (kubectl exec against harbor.focuscell.org/infra/vllm-openai:v0.20.0-tq-hybrid-v2).

Source citations:

• vllm/v1/spec_decode/dflash.py:192,289 — DFlash mandates causal=False for draft cross-attention
• vllm/v1/attention/backends/turboquant.py:308,319,612 — causal=True hardcoded, no non-causal path
• vllm/v1/spec_decode/dflash.py:518 — DFlash draft model init

Reproduction

# Target: Intel/Qwen3.6-27B-int4-AutoRound (Qwen3_5ForConditionalGeneration)
# Draft:  z-lab/Qwen3.6-27B-DFlash
# vLLM:   v0.20.0

python -m vllm.entrypoints.openai.api_server \
  --model Intel/Qwen3.6-27B-int4-AutoRound \
  --quantization auto_round \
  --kv-cache-dtype fp8_e5m2 \
  --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-27B-DFlash","num_speculative_tokens":16}' \
  --max-model-len 96000

Observed: Engine init fails. All fp8_e5m2 / fp8_e4m3 / turboquant_4bit_nc variants crash with backend rejection errors citing non-causal incompatibility. --kv-cache-dtype auto (bfloat16) is the only path that allows engine init with DFlash.

Crash log excerpt (FLASH_ATTN + fp8_e5m2):

ValueError: KV cache dtype fp8_e5m2 is not supported for non-causal attention
  in vllm/v1/attention/backends/flash_attn.py

Impact

DFlash is advertised as a memory-efficient speculative decoding option (per the upstream PR). In practice, on consumer GPUs (24 GiB RTX 3090) with a 27B-class quantized target model:

• With fp8_e5m2 KV cache: ~28K token KV pool (EX004, proven in production)
• With DFlash + bfloat16 KV cache: ~14K token KV pool (50% regression)

The spec decode throughput gains from DFlash do not justify halving the KV pool for long-context workloads. DFlash as currently implemented is only practical for short-context or high-VRAM setups.

Suggested fix paths

1. Allow KV cache quantization for non-causal attention paths in FLASH_ATTN and FLEX_ATTENTION backends — the quantization math is the same regardless of causal mask
2. Add a non-causal variant of the turboquant backend that supports turboquant_4bit_nc with causal=False
3. At minimum, document this constraint in the DFlash configuration docs so users don't discover it through trial and error

Environment

• vLLM version: 0.20.0 (image harbor.focuscell.org/infra/vllm-openai:v0.20.0-tq-hybrid-v2, based on mainline)
• Hardware: NVIDIA RTX 3090 24 GiB (Ampere, SM 86)
• Target model: Intel/Qwen3.6-27B-int4-AutoRound (Qwen3_5ForConditionalGeneration)
• Draft model: z-lab/Qwen3.6-27B-DFlash
• OS: Talos Linux, CUDA 12.4

Internal references

• Our investigation and compat matrix: https://git.developerdojo.org/HardMagic/inference-tuning/-/work_items/28
• Compat matrix file: references/vllm-compat-matrix-v0.20.0.md in HardMagic/inference-tuning