vllm - 💡(How to fix) Fix [Bug]: NCCL Error: unhandled cuda error [1 participants]

Can't share completely but it has 2 gpus with 40GB vram each. They run on 1 single node. When I run serve Qwen30-30B-A3B it fails and only returns this NCCL Error: unhandled cuda error. I have set NCCL_DEBUG="INFO" or "TRACE" but neither shows the error.

TL;DR

• Investigate CUDA-related issues as the likely cause of the NCCL Error: unhandled cuda error when running serve Qwen30-30B-A3B.

Guidance

• Check the CUDA version and ensure it is compatible with the NCCL version being used.
• Verify that the GPUs are properly recognized and utilized by the system, and that there are no issues with the GPU drivers.
• Consider running a simple CUDA test program to isolate if the issue is specific to the serve Qwen30-30B-A3B command or a broader CUDA problem.

Notes

• The lack of detailed error messages even with NCCL_DEBUG set to "INFO" or "TRACE" suggests that the issue might be occurring at a lower level, potentially related to CUDA or GPU interaction.

Recommendation

• Apply workaround: Try updating CUDA drivers or checking for GPU-specific configuration issues, as this might resolve the unhandled CUDA error without requiring a version upgrade of other components.