vllm - 💡(How to fix) Fix [CI Failure]: Fusion E2E TP2 Quick (H100) [1 participants]

TL;DR

Review the changes introduced in the Deepseek V4 PR (#40860) to identify potential memory allocation issues causing the consistent OOM errors in the test_tp2_ar_rms_fp8_fusions test.

Guidance

• Investigate the memory tracking mechanism to understand why it's considered unsafe and how it might be improved to provide more accurate allocation information.
• Analyze the changes made in the DSV4 PR, focusing on any new allocations or memory-intensive operations that could be causing the OOM errors.
• Consider running the test with increased memory allocation or using a memory profiling tool to identify the specific allocation causing the issue.
• Review the test case test_tp2_ar_rms_fp8_fusions to see if there are any opportunities to optimize memory usage or reduce allocations.

Notes

The lack of ability to reproduce the issue locally and the known issues with memory tracking make it challenging to provide a definitive solution. Further investigation into the changes introduced by the DSV4 PR and the memory usage patterns of the test is necessary.

Recommendation

Apply workaround: Increase memory allocation for the test or optimize memory usage in the test_tp2_ar_rms_fp8_fusions test to mitigate the OOM errors until the root cause can be fully identified and addressed.