vllm - 💡(How to fix) Fix [Bug]: vllm 0.18 kimi k2.5 way worse than h200 single node [3 comments, 2 participants]

https://github.com/SemiAnalysisAI/InferenceX/actions/runs/23637554623/job/68911904882

##H200 logs & reprod https://github.com/SemiAnalysisAI/InferenceX/commit/749096c4c5ae4dfc37a878cd9081561483282fb8 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/22600499132/job/65492428203

Fix Plan

To address the performance issue, we will focus on optimizing the vllm serve command and environment variables.

Step-by-Step Solution

1. Update Environment Variables: Ensure that VLLM_ROCM_USE_AITER is set to 1 to enable AITER MLA for Kimi TP8.
2. Optimize vllm serve Command:
   • Adjust --gpu-memory-utilization to a lower value (e.g., 0.8) to prevent memory bottlenecks.
   • Experiment with different --block-size values (e.g., 32 or 128) to find the optimal setting for your model and hardware.
   • Consider increasing --max-num-seqs to improve throughput.

Example code snippet:

export VLLM_ROCM_USE_AITER=1
vllm serve $MODEL --port $PORT \
--tensor-parallel-size=$TP \
--gpu-memory-utilization 0.8 \
--max-model-len $MAX_MODEL_LEN \
--block-size=32 \
--trust-remote-code \
--max-num-seqs 512 \
--mm-encoder-tp-mode data > $SERVER_LOG 2>&1 &

1. Verify Hardware Configuration: Ensure that the MI325 hardware is properly configured and optimized for the workload.

Verification

To verify that the fix worked, monitor the performance metrics (e.g., throughput, latency) and compare them to the previous results. You can use tools like gpu-util or nvidia-smi to monitor GPU utilization and memory usage.

Extra Tips

• Regularly update your vllm version to ensure you have the latest optimizations and bug fixes.
• Experiment with different --mm-encoder-tp-mode values (e.g., model, data) to find the optimal setting for your use case.
• Consider using tools like perf or nvprof to profile your application and identify performance bottlenecks.