vllm - 💡(How to fix) Fix [Bug]: Gemma 4 26B MoE NVFP4 fails under tensor-parallel-size 2 across all available MoE backends [1 participants]

TL;DR

The issue may be resolved by modifying the expert dimensions in the 26B MoE architecture to be compatible with tensor parallel size 2.

Guidance

• Investigate the kernel paths that are causing the initialization failure to understand the specific compatibility issues with the 26B MoE architecture.
• Compare the expert dimensions and tensor parallelization settings between the community NVFP4 quantizations of google/gemma-4-26B-A4B-it and the official nvidia/Gemma-4-31B-IT-NVFP4 to identify potential differences.
• Consider testing with a smaller tensor parallel size or modifying the model architecture to accommodate the parallel size 2.
• Verify that the issue is indeed specific to the 26B MoE architecture and not a more general problem with tensor parallelization.

Notes

The solution may require significant modifications to the model architecture or the tensor parallelization settings, and may not be compatible with all use cases.

Recommendation

Apply workaround: Modify the expert dimensions in the 26B MoE architecture to be compatible with tensor parallel size 2, as the official nvidia/Gemma-4-31B-IT-NVFP4 model works correctly under TP=2.