vllm - 💡(How to fix) Fix [Performance]: DeepSeek-V3.2 performance on 8xH20 is not match with official data [1 participants]

TL;DR

The most likely fix is to adjust the model or batch size to reduce memory usage and prevent Out-of-Memory (OOM) errors on the 8xH20 devices.

Guidance

• Review the benchmarking scripts from the provided URL to identify potential memory-intensive operations or parameters that can be adjusted.
• Consider reducing the batch size or model complexity to decrease memory usage, as the official performance tests may have been conducted on more capable hardware (8xH200).
• Verify the device's memory usage and available resources before and after adjustments to ensure the changes are effective.
• Check the documentation for any specific guidelines on running the scripts on 8xH20 devices, as there may be known limitations or workarounds.

Notes

The provided information lacks specific details about the scripts and model configurations, so the suggested adjustments are general and may require further experimentation to find the optimal solution.

Recommendation

Apply workaround: Adjust the model or batch size to reduce memory usage, as this is a more feasible and immediate solution given the potential hardware limitations.