vllm - 💡(How to fix) Fix [Bug]: Qwen3.5-397B-NVFP4 Disagg accuracy gsm8k collapses with async scheduling

vllm2026-05-09 20:30:28

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

Qwen3.5-397B-A17B-NVFP4 disagg serving has a severe correctness regression when --async-scheduling is enabled.

This is not the FP8 KV cache issue. The repro uses --kv-cache-dtype bfloat16.

The issue appears only under a larger GSM8K evaluation load. A small 32-question GSM8K sanity run is healthy, but the full 1319-question GSM8K run collapses in accuracy and produces many invalid responses under the same server configuration.

Root Cause

Without these patches, Qwen3.5 Disagg serving fails earlier because the GDN conv-state transfer path for NIXL is not fully implemented, so the FP8 KV cache correctness issue cannot be reached.

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 144 On-line CPU(s) list: 0-143 Vendor ID: ARM Model name: Neoverse-V2 Model: 0 Thread(s) per core: 1 Core(s) per socket: 72 Socket(s): 2 Stepping: r0p0 Frequency boost: disabled CPU(s) scaling MHz: 100% CPU max MHz: 3411.0000 CPU min MHz: 81.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti L1d cache: 9 MiB (144 instances) L1i cache: 9 MiB (144 instances) L2 cache: 144 MiB (144 instances) L3 cache: 228 MiB (2 instances) NUMA node(s): 34 NUMA node0 CPU(s): 0-71 NUMA node1 CPU(s): 72-143 NUMA node2 CPU(s): NUMA node3 CPU(s): NUMA node4 CPU(s): NUMA node5 CPU(s): NUMA node6 CPU(s): NUMA node7 CPU(s): NUMA node8 CPU(s): NUMA node9 CPU(s): NUMA node10 CPU(s): NUMA node11 CPU(s): NUMA node12 CPU(s): NUMA node13 CPU(s): NUMA node14 CPU(s): NUMA node15 CPU(s): NUMA node16 CPU(s): NUMA node17 CPU(s): NUMA node18 CPU(s): NUMA node19 CPU(s): NUMA node20 CPU(s): NUMA node21 CPU(s): NUMA node22 CPU(s): NUMA node23 CPU(s): NUMA node24 CPU(s): NUMA node25 CPU(s): NUMA node26 CPU(s): NUMA node27 CPU(s): NUMA node28 CPU(s): NUMA node29 CPU(s): NUMA node30 CPU(s): NUMA node31 CPU(s): NUMA node32 CPU(s): NUMA node33 CPU(s): Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Not affected Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; CSV2, but not BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected

Without these patches, Qwen3.5 Disagg serving fails earlier because the GDN conv-state transfer path for NIXL is not fully implemented, so the FP8 KV cache correctness issue cannot be reached.

Code Example

==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (aarch64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.17.0-1014-nvidia-64k-aarch64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.61
CUDA_MODULE_LOADING set to   :
GPU models and configuration :
GPU 0: NVIDIA GB200
GPU 1: NVIDIA GB200
GPU 2: NVIDIA GB200
GPU 3: NVIDIA GB200

Nvidia driver version        : Could not collect
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  144
On-line CPU(s) list:                     0-143
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   0
Thread(s) per core:                      1
Core(s) per socket:                      72
Socket(s):                               2
Stepping:                                r0p0
Frequency boost:                         disabled
CPU(s) scaling MHz:                      100%
CPU max MHz:                             3411.0000
CPU min MHz:                             81.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti
L1d cache:                               9 MiB (144 instances)
L1i cache:                               9 MiB (144 instances)
L2 cache:                                144 MiB (144 instances)
L3 cache:                                228 MiB (2 instances)
NUMA node(s):                            34
NUMA node0 CPU(s):                       0-71
NUMA node1 CPU(s):                       72-143
NUMA node2 CPU(s):
NUMA node3 CPU(s):
NUMA node4 CPU(s):
NUMA node5 CPU(s):
NUMA node6 CPU(s):
NUMA node7 CPU(s):
NUMA node8 CPU(s):
NUMA node9 CPU(s):
NUMA node10 CPU(s):
NUMA node11 CPU(s):
NUMA node12 CPU(s):
NUMA node13 CPU(s):
NUMA node14 CPU(s):
NUMA node15 CPU(s):
NUMA node16 CPU(s):
NUMA node17 CPU(s):
NUMA node18 CPU(s):
NUMA node19 CPU(s):
NUMA node20 CPU(s):
NUMA node21 CPU(s):
NUMA node22 CPU(s):
NUMA node23 CPU(s):
NUMA node24 CPU(s):
NUMA node25 CPU(s):
NUMA node26 CPU(s):
NUMA node27 CPU(s):
NUMA node28 CPU(s):
NUMA node29 CPU(s):
NUMA node30 CPU(s):
NUMA node31 CPU(s):
NUMA node32 CPU(s):
NUMA node33 CPU(s):
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Not affected
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, but not BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.6.2
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.19.2rc1.dev215+g32e45636e (git sha: 32e45636e)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-71    0              N/A
GPU1    NV18     X      NV18    NV18    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-71    0              N/A
GPU2    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    72-143  1              N/A
GPU3    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    72-143  1              N/A
NIC0    NODE    NODE    SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC1    NODE    NODE    SYS     SYS     NODE     X      NODE    NODE    SYS     SYS     SYS     SYS
NIC2    NODE    NODE    SYS     SYS     NODE    NODE     X      PIX     SYS     SYS     SYS     SYS
NIC3    NODE    NODE    SYS     SYS     NODE    NODE    PIX      X      SYS     SYS     SYS     SYS
NIC4    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC5    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC6    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      PIX
NIC7    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    PIX      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/home/aperevedents/storage/cuda/cuda128/lib64:
CUDA_HOME=/home/aperevedents/storage/cuda/cuda128
CUDA_HOME=/home/aperevedents/storage/cuda/cuda128
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_aperevedents

---

# [prefill node]: Start server
VLLM_SSM_CONV_STATE_LAYOUT=DS \
VLLM_NIXL_SIDE_CHANNEL_HOST=$(hostname) \
VLLM_NIXL_SIDE_CHANNEL_PORT=6550 \
vllm serve nvidia/Qwen3.5-397B-A17B-NVFP4 \
  --served-model-name Qwen3.5-397B-A17B-NVFP4 \
  --host 0.0.0.0 \
  --port 8100 \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --enable-expert-parallel \
  --generation-config vllm \
  --no-enable-prefix-caching \
  --kv-cache-dtype bfloat16 \
  --async-scheduling \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail"}'

# [decode node]: Start server
VLLM_SSM_CONV_STATE_LAYOUT=DS \
VLLM_NIXL_SIDE_CHANNEL_HOST=$(hostname) \
VLLM_NIXL_SIDE_CHANNEL_PORT=6551 \
vllm serve nvidia/Qwen3.5-397B-A17B-NVFP4 \
  --served-model-name Qwen3.5-397B-A17B-NVFP4 \
  --host 0.0.0.0 \
  --port 8200 \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --enable-expert-parallel \
  --generation-config vllm \
  --no-enable-prefix-caching \
  --kv-cache-dtype bfloat16 \
  --async-scheduling \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail"}'

# [localhost]: Start proxy
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
  --host 0.0.0.0 \
  --port 8000 \
  --prefiller-hosts "$PREFILL_HOST" \
  --prefiller-ports 8100 \
  --decoder-hosts "$DECODE_HOST" \
  --decoder-ports 8200

# [localhost]: Small GSM8K sanity run
python tests/evals/gsm8k/gsm8k_eval.py \
  --host http://localhost \
  --port 8000 \
  --model Qwen3.5-397B-A17B-NVFP4 \
  --max-tokens 1024 \
  --no-stop \
  --num-questions 32 \
  --save-results /tmp/qwen35_disagg_async_bf16_gsm8k_32.json \
  --save-details

---

Running GSM8K evaluation: 32 questions, 5-shot
Results:
Accuracy: 0.906
Invalid responses: 0.031
Total latency: 8.399 s
Questions per second: 3.810
Total output tokens: 6912
Output tokens per second: 822.930
Results saved to /tmp/qwen35_disagg_async_bf16_gsm8k_32.json

---

# [localhost]: Full GSM8K run
python tests/evals/gsm8k/gsm8k_eval.py \
  --host http://localhost \
  --port 8000 \
  --model Qwen3.5-397B-A17B-NVFP4 \
  --max-tokens 1024 \
  --no-stop \
  --save-results /tmp/qwen35_disagg_async_bf16_gsm8k.json \
  --save-details

---

Running GSM8K evaluation: 1319 questions, 5-shot

Results:
Accuracy: 0.078
Invalid responses: 0.442
Total latency: 146.944 s
Questions per second: 8.976
Total output tokens: 965451
Output tokens per second: 6570.211
Results saved to /tmp/qwen35_disagg_async_bf16_gsm8k.json

RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (aarch64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.17.0-1014-nvidia-64k-aarch64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.61
CUDA_MODULE_LOADING set to   :
GPU models and configuration :
GPU 0: NVIDIA GB200
GPU 1: NVIDIA GB200
GPU 2: NVIDIA GB200
GPU 3: NVIDIA GB200

Nvidia driver version        : Could not collect
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  144
On-line CPU(s) list:                     0-143
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   0
Thread(s) per core:                      1
Core(s) per socket:                      72
Socket(s):                               2
Stepping:                                r0p0
Frequency boost:                         disabled
CPU(s) scaling MHz:                      100%
CPU max MHz:                             3411.0000
CPU min MHz:                             81.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti
L1d cache:                               9 MiB (144 instances)
L1i cache:                               9 MiB (144 instances)
L2 cache:                                144 MiB (144 instances)
L3 cache:                                228 MiB (2 instances)
NUMA node(s):                            34
NUMA node0 CPU(s):                       0-71
NUMA node1 CPU(s):                       72-143
NUMA node2 CPU(s):
NUMA node3 CPU(s):
NUMA node4 CPU(s):
NUMA node5 CPU(s):
NUMA node6 CPU(s):
NUMA node7 CPU(s):
NUMA node8 CPU(s):
NUMA node9 CPU(s):
NUMA node10 CPU(s):
NUMA node11 CPU(s):
NUMA node12 CPU(s):
NUMA node13 CPU(s):
NUMA node14 CPU(s):
NUMA node15 CPU(s):
NUMA node16 CPU(s):
NUMA node17 CPU(s):
NUMA node18 CPU(s):
NUMA node19 CPU(s):
NUMA node20 CPU(s):
NUMA node21 CPU(s):
NUMA node22 CPU(s):
NUMA node23 CPU(s):
NUMA node24 CPU(s):
NUMA node25 CPU(s):
NUMA node26 CPU(s):
NUMA node27 CPU(s):
NUMA node28 CPU(s):
NUMA node29 CPU(s):
NUMA node30 CPU(s):
NUMA node31 CPU(s):
NUMA node32 CPU(s):
NUMA node33 CPU(s):
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Not affected
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, but not BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.6.2
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.19.2rc1.dev215+g32e45636e (git sha: 32e45636e)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-71    0              N/A
GPU1    NV18     X      NV18    NV18    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-71    0              N/A
GPU2    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    72-143  1              N/A
GPU3    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    72-143  1              N/A
NIC0    NODE    NODE    SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC1    NODE    NODE    SYS     SYS     NODE     X      NODE    NODE    SYS     SYS     SYS     SYS
NIC2    NODE    NODE    SYS     SYS     NODE    NODE     X      PIX     SYS     SYS     SYS     SYS
NIC3    NODE    NODE    SYS     SYS     NODE    NODE    PIX      X      SYS     SYS     SYS     SYS
NIC4    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC5    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC6    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      PIX
NIC7    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    PIX      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/home/aperevedents/storage/cuda/cuda128/lib64:
CUDA_HOME=/home/aperevedents/storage/cuda/cuda128
CUDA_HOME=/home/aperevedents/storage/cuda/cuda128
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_aperevedents

</details>

🐛 Describe the bug

IMPORTANT!!!

Before reproducing this issue, Qwen3.5/GDN NIXL disagg support needs to be applied:

#41628: Support GDN conv-state splits for NIXL
#41869: PD disagg with NIXL Connector: GDN support (Qwen3.5)

Without these patches, Qwen3.5 Disagg serving fails earlier because the GDN conv-state transfer path for NIXL is not fully implemented, so the FP8 KV cache correctness issue cannot be reached.

Summary

Qwen3.5-397B-A17B-NVFP4 disagg serving has a severe correctness regression when --async-scheduling is enabled.

This is not the FP8 KV cache issue. The repro uses --kv-cache-dtype bfloat16.

Setup

Model: nvidia/Qwen3.5-397B-A17B-NVFP4
vLLM: 0.19.2rc1.dev215+g32e45636e (git sha: 32e45636e)
Hardware: GB200
Disagg setup:
- Prefill: 1 node, 4 GPUs, TP=4, EP enabled
- Decode: 1 node, 4 GPUs, TP=4, EP enabled
KV connector: NixlConnector
KV cache dtype: bfloat16
Prefix caching: disabled
Async scheduling: enabled

Testing Done

Qwen3.5 Disagg with async scheduling enabled

# [prefill node]: Start server
VLLM_SSM_CONV_STATE_LAYOUT=DS \
VLLM_NIXL_SIDE_CHANNEL_HOST=$(hostname) \
VLLM_NIXL_SIDE_CHANNEL_PORT=6550 \
vllm serve nvidia/Qwen3.5-397B-A17B-NVFP4 \
  --served-model-name Qwen3.5-397B-A17B-NVFP4 \
  --host 0.0.0.0 \
  --port 8100 \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --enable-expert-parallel \
  --generation-config vllm \
  --no-enable-prefix-caching \
  --kv-cache-dtype bfloat16 \
  --async-scheduling \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail"}'

# [decode node]: Start server
VLLM_SSM_CONV_STATE_LAYOUT=DS \
VLLM_NIXL_SIDE_CHANNEL_HOST=$(hostname) \
VLLM_NIXL_SIDE_CHANNEL_PORT=6551 \
vllm serve nvidia/Qwen3.5-397B-A17B-NVFP4 \
  --served-model-name Qwen3.5-397B-A17B-NVFP4 \
  --host 0.0.0.0 \
  --port 8200 \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --enable-expert-parallel \
  --generation-config vllm \
  --no-enable-prefix-caching \
  --kv-cache-dtype bfloat16 \
  --async-scheduling \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail"}'

# [localhost]: Start proxy
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
  --host 0.0.0.0 \
  --port 8000 \
  --prefiller-hosts "$PREFILL_HOST" \
  --prefiller-ports 8100 \
  --decoder-hosts "$DECODE_HOST" \
  --decoder-ports 8200

# [localhost]: Small GSM8K sanity run
python tests/evals/gsm8k/gsm8k_eval.py \
  --host http://localhost \
  --port 8000 \
  --model Qwen3.5-397B-A17B-NVFP4 \
  --max-tokens 1024 \
  --no-stop \
  --num-questions 32 \
  --save-results /tmp/qwen35_disagg_async_bf16_gsm8k_32.json \
  --save-details

Output (correct):

Running GSM8K evaluation: 32 questions, 5-shot
Results:
Accuracy: 0.906
Invalid responses: 0.031
Total latency: 8.399 s
Questions per second: 3.810
Total output tokens: 6912
Output tokens per second: 822.930
Results saved to /tmp/qwen35_disagg_async_bf16_gsm8k_32.json

Full gsm8k test

# [localhost]: Full GSM8K run
python tests/evals/gsm8k/gsm8k_eval.py \
  --host http://localhost \
  --port 8000 \
  --model Qwen3.5-397B-A17B-NVFP4 \
  --max-tokens 1024 \
  --no-stop \
  --save-results /tmp/qwen35_disagg_async_bf16_gsm8k.json \
  --save-details

Output (incorrect):

Running GSM8K evaluation: 1319 questions, 5-shot

Results:
Accuracy: 0.078
Invalid responses: 0.442
Total latency: 146.944 s
Questions per second: 8.976
Total output tokens: 965451
Output tokens per second: 6570.211
Results saved to /tmp/qwen35_disagg_async_bf16_gsm8k.json

Full logs

GSM8K evaluation script note

The repro uses vLLM's tests/evals/gsm8k/gsm8k_eval.py with small local CLI additions needed for this disaggregated OpenAI-compatible endpoint:

Added --model so the request body includes "model": "Qwen3.5-397B-A17B-NVFP4". This is required by the proxy/server path; the upstream script does not send the model field.
Added --no-stop to avoid the default GSM8K stop strings ("Question", "Assistant:", "<|separator|>"). This is useful for Qwen3.5-style thinking output, where those strings can appear inside reasoning and prematurely truncate generations.
Added --save-details to store per-question raw responses for debugging.
Adjusted answer extraction to prefer the canonical GSM8K marker #### <answer> when present, instead of always using the last number in the full response.

These changes do not alter the server configuration or the request concurrency pattern. They only make the vLLM GSM8K script compatible with this served-model-name/proxy setup and make the reported accuracy easier to debug.

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#device allocation #model download #tokenizer error #environment variable #cache issue

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - 💡(How to fix) Fix [Bug]: Qwen3.5-397B-NVFP4 Disagg accuracy gsm8k collapses with async scheduling

Recommended Tools

GitHub issue graph ai analysis

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

Code Example

Your current environment

🐛 Describe the bug

IMPORTANT!!!

Summary

Setup

Testing Done

Qwen3.5 Disagg with async scheduling enabled

Output (correct):

Full gsm8k test

Output (incorrect):

Full logs

GSM8K evaluation script note

Before submitting a new issue...

Still need to ship something?

TRENDING

vllm - 💡(How to fix) Fix [Bug]: Qwen3.5-397B-NVFP4 Disagg accuracy gsm8k collapses with async scheduling

Recommended Tools

GitHub issue graph ai analysis

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

Code Example

Your current environment

🐛 Describe the bug

IMPORTANT!!!

Summary

Setup

Testing Done

Qwen3.5 Disagg with async scheduling enabled

Output (correct):

Full gsm8k test

Output (incorrect):

Full logs

GSM8K evaluation script note

Before submitting a new issue...

Still need to ship something?

RELATED_DISCOVERY

TRENDING