vllm - 💡(How to fix) Fix [Bug]: Qwen3.5 (Qwen3_5ForConditionalGeneration) FLA linear attention tensor format mismatch causes gibberish output [2 comments, 2 participants]

Description

When running Qwen3_5ForConditionalGeneration (Qwen3.5-4B-NVFP4) with vLLM nightly, the model produces completely garbled/gibberish output — a mix of random tokens from multiple languages.

Root Cause

The following warning is emitted during every inference call, indicating a tensor format mismatch in the FLA (Flash Linear Attention) ops layer:

vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning:
Input tensor shape suggests potential format mismatch: seq_len (12) < num_heads (32).
This may indicate the inputs were passed in head-first format [B, H, T, ...]
when head_first=False was specified. Please verify your input tensor format
matches the expected shape [B, T, H, ...].

Also from torch/_dynamo/eval_frame.py:

UserWarning: Input tensor shape suggests potential format mismatch: seq_len (12) < num_heads (16).
This may indicate the inputs were passed in head-first format [B, H, T, ...]
Please verify your input tensor format matches the expected shape [B, T, H, ...].

Qwen3.5 uses a hybrid Transformer + linear attention (Mamba-style) architecture. The linear_attn.conv1d layers are present in multiple decoder layers. The FLA ops kernel appears to receive tensors in [B, H, T, ...] (head-first) format when head_first=False is expected, causing completely incorrect computation and gibberish output.

Reproduction

# docker-compose.yml
services:
  qwen3-5-4b:
    image: vllm/vllm-openai:nightly
    command: [
      "/models/Qwen3.5-4B-NVFP4",
      "--served-model-name", "Qwen3.5-4B-NVFP4",
      "--gpu-memory-utilization", "0.85",
      "--host", "0.0.0.0",
      "--port", "8000",
      "--dtype", "auto",
      "--max-model-len", "32768",
      "--trust-remote-code",
      "--attention-backend", "flashinfer"
    ]

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.5-4B-NVFP4",
    "messages": [{"role": "user", "content": "Hello, introduce yourself"}],
    "temperature": 0.6,
    "max_tokens": 100
  }'

Expected output: Normal coherent text response

Actual output: Gibberish mix of random multilingual tokens, e.g.:

一根ulaHTTPRequestdataIK_and胜利 Ký尔ula树皮 auxili紹ns/welcome生生 televis多喝在水中 ric尔ael注射گز...

Environment

• vLLM version: v0.17.0rc1.dev164+gfff3711a2
• Model: Qwen/Qwen3.5-4B (NVFP4 quantized checkpoint, Qwen3_5ForConditionalGeneration)
• GPU: NVIDIA GeForce RTX 5090 (SM 12.0 / Blackwell)
• CUDA: via Docker vllm/vllm-openai:nightly
• OS: Ubuntu (WSL2, kernel 6.6.87.2-microsoft-standard-WSL2)

Additional Notes

• The bug occurs regardless of --quantization flag, --dtype, --kv-cache-dtype, or --enforce-eager
• With --enforce-eager, the request hangs indefinitely (no response), confirming the linear attention computation is broken
• The hf_quant_config.json correctly excludes model.language_model.layers.*.linear_attn.conv1d from NVFP4 quantization, so those layers should be in BF16 — the issue is in the FLA kernel's tensor layout, not quantization
• Standard Transformer-only models (e.g., Qwen3-4B) work correctly; the issue is specific to Qwen3.5's hybrid linear attention architecture