vllm - 💡(How to fix) Fix [Bug]: Segfault in FlashInfer autotuning for NVFP4 latency backend on Qwen3-30B-A3B-NVFP4 [2 comments, 2 participants]

Fix Plan

To resolve the crash issue with FlashInfer NVFP4 MoE enabled, we need to update the flashinfer library and modify the vllm configuration. Here are the steps:

• Update flashinfer to the latest version:
  • Build flashinfer from source with the latest commit
• Modify vllm configuration:
  • Set VLLM_USE_FLASHINFER_MOE_FP4=0 to disable FP4 block scale
  • Set VLLM_FLASHINFER_MOE_BACKEND=throughput to use throughput backend
• Update torch to the latest version compatible with flashinfer

Example code snippet to update vllm configuration:

import os

# Disable FP4 block scale
os.environ['VLLM_USE_FLASHINFER_MOE_FP4'] = '0'

# Use throughput backend
os.environ['VLLM_FLASHINFER_MOE_BACKEND'] = 'throughput'

Command to serve nvidia/Qwen3-30B-A3B-NVFP4 with updated configuration:

export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_FLASHINFER_MOE_BACKEND=throughput

CUDA_VISIBLE_DEVICES=4 \
vllm serve nvidia/Qwen3-30B-A3B-NVFP4 \
  --tensor-parallel-size 1 \
  --port 8102 \
  --no-enable-prefix-caching

Verification

To verify that the fix worked, check the vllm serve output for any errors or crashes. If the issue is resolved, the output should indicate that the model is serving successfully.

Extra Tips

• Make sure to update flashinfer and torch to the latest versions to ensure compatibility and stability.
• If the issue persists, try setting VLLM_FLASHINFER_MOE_BACKEND=debug to enable debug mode and gather more information about the crash.