vllm - 💡(How to fix) Fix [BUGS] vLLM V1 Engine Hangs After Weight Loading on Blackwell (sm_121) Multi-Node Ray Setup (TP=2) [2 comments, 2 participants]

TL;DR

The issue can be potentially resolved by investigating the memory utilization and distribution of the model weights across the GPUs, as the hang occurs immediately after successful model loading.

Guidance

• Verify the memory utilization of each GPU node after model loading to ensure that the --gpu-memory-utilization setting of 0.7 is not causing an issue, and consider adjusting this value further if necessary.
• Check the NCCL (Network Collective Communication Library) configuration and logs for any errors or warnings that might indicate issues with the distributed setup, given that NCCL_P2P_DISABLE=1 was set.
• Investigate the impact of VLLM_DISABLE_FRONTEND_MULTIPROCESSING=1 despite the warning, as it might be related to the hanging issue, even if the variable is reported as unknown/unsupported in this version.
• Review the Ray configuration and logs for any timeout or resource allocation issues, considering the adjustments made to CGRAPH timeouts.

Example

No specific code snippet can be provided without more context on the customizations or specific configurations applied to the vLLM setup.

Notes

The solution may depend on the specific version of vLLM and its compatibility with the distributed setup and hardware. The fact that model weights load successfully but the process hangs afterward suggests a potential issue with resource allocation, memory, or communication between nodes.

Recommendation

Apply workaround: Adjust the --gpu-memory-utilization setting and monitor the system's memory and resource utilization closely to identify any bottlenecks, as this seems to be a critical point of failure given the successful loading of model weights but subsequent hang.