vllm - 💡(How to fix) Fix [Bug]: DeepSeek V4 pro can not run with TP16 [1 comments, 2 participants]

can not run with this error:

TL;DR

The error may be related to a mismatch between the expected and actual size of the fp8 quantization, which could be resolved by adjusting the quantization size to match the V4 shape requirements.

Guidance

• Verify the V4 shape calculation to ensure it matches the expected output of 192, and check if the fp8 quantization size is correctly set to 128*x.
• Review the documentation and recipes provided to ensure that the correct parameters are being used for the DeepSeek-V4-Pro model.
• Check the image attachments for any additional error messages or clues that may indicate the root cause of the issue.
• Consider reaching out to the chatbot on the documentation page for further assistance with troubleshooting.

Notes

The provided information lacks specific technical details, making it challenging to provide a precise solution. Further investigation and verification of the model parameters and quantization settings are necessary to resolve the issue.

Recommendation

Apply workaround: Adjust the quantization size to match the V4 shape requirements, as this may resolve the error and allow the model to run correctly.