vllm - 💡(How to fix) Fix [Usage]: RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)` [2 comments, 2 participants]

im trying to serve a quantized 4 bit model on a T4 gpu, i get this error

TL;DR

• Verify that the quantized 4-bit model is compatible with the T4 GPU architecture to resolve the serving error.

Guidance

• Check the documentation for VLLM to ensure that 4-bit quantization is supported on T4 GPUs.
• Review the model's configuration to confirm that it is correctly set up for 4-bit quantization.
• Investigate if there are any specific requirements or limitations for serving quantized models on T4 GPUs in the VLLM documentation or related issues.

Notes

• The issue lacks specific error details, making it challenging to provide a more targeted solution.
• Compatibility between the model's quantization and the GPU architecture is a common potential cause for such errors.

Recommendation

• Apply workaround: Check the VLLM documentation and adjust the model's quantization settings or explore alternative serving options if 4-bit quantization is not supported on T4 GPUs.