vllm - 💡(How to fix) Fix [Performance]: NVFP4 MoE on SM120: no env override to select backend (FLASHINFER_CUTLASS vs MARLIN) [1 comments, 2 participants]

TL;DR

Adding an environment variable VLLM_NVFP4_MOE_BACKEND to override the NVFP4 MoE backend selection could help mitigate the performance regression.

Guidance

• Introduce the proposed environment variable VLLM_NVFP4_MOE_BACKEND to allow users to manually select the MoE backend, potentially opting for Marlin over FLASHINFER_CUTLASS for better performance in certain workloads.
• Verify the impact of this change by comparing single-user throughput with different backends, using metrics such as tokens per second.
• Consider testing with various hardware configurations and parameters (e.g., --gpu-memory-utilization, --max-num-seqs) to understand the tradeoffs between KV cache availability and throughput.
• Evaluate the effectiveness of the override by monitoring performance metrics and adjusting the backend selection as needed.

Example

No specific code snippet is provided, but the introduction of the VLLM_NVFP4_MOE_BACKEND environment variable could be implemented in a configuration file or as a command-line argument, allowing users to specify their preferred MoE backend.

Notes

The performance regression appears to be specific to the SM120 (RTX PRO 6000) hardware and the NVFP4 MoE models. The proposed solution may not apply to other hardware configurations or model types.

Recommendation

Apply workaround: Introduce the VLLM_NVFP4_MOE_BACKEND environment variable to allow users to override the default MoE backend selection, potentially improving performance in certain workloads. This approach provides flexibility and allows users to choose the best backend for their specific use case.