vllm - 💡(How to fix) Fix [Bug]: R1 NVFP4 flash_infer_one_sided on prefill causes accuracy degradation during P/D

TL;DR

The accuracy drop from 96.3% to 92.9% may be related to the change in the all2all-backend to flash_infer_one_sided in the prefill.yaml configuration.

Guidance

• Review the flash_infer_one_sided configuration and its impact on the model's accuracy, especially when used for inference.
• Compare the performance of flash_infer_one_sided with other backend options to determine if the issue is specific to this configuration.
• Investigate if there are any known limitations or trade-offs with using flash_infer_one_sided that could be contributing to the accuracy drop.
• Consider testing the model with the original backend configuration to isolate the cause of the accuracy drop.

Notes

The issue seems to be specific to the flash_infer_one_sided backend and does not occur when used for decoding, suggesting that the problem may be related to the inference process.

Recommendation

Apply a workaround by testing alternative backend configurations to find one that maintains the desired accuracy level, as the current configuration change seems to be the likely cause of the issue.