vllm - 💡(How to fix) Fix [Performance]: vllm 19.0 online server测试波动偏大 [1 participants]

TL;DR

• Investigate and adjust the batch size, gpu-memory-utilization, and tensor-parallel-size parameters to stabilize the accuracy of the model.

Guidance

• Review the command used to deploy the model as a server, focusing on parameters that could impact performance and accuracy, such as batch size, gpu-memory-utilization, and tensor-parallel-size.
• Consider reducing the batch size from 32 to a smaller number to see if it improves the stability of the accuracy.
• Experiment with different values for gpu-memory-utilization and tensor-parallel-size to find an optimal balance between performance and accuracy.
• Verify the impact of these changes by running multiple tests with the same set of data to check for consistency in accuracy.

Example

No specific code example is provided due to the lack of direct code issues in the problem statement. However, an example command adjusting some of the mentioned parameters could look like:

OMP_NUM_THREADS=1 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m vllm.entrypoints.openai.api_server --model /models/Qwen3-VL-8B-Instruct --served-model-name Qwen3-VL-8B-Instruct --port=8007 --limit-mm-per-prompt.image=15 --seed 42 --gpu-memory-utilization 0.80 --trust-remote-code True --tensor-parallel-size 4 --attention-backend FLASH_ATTN

This example reduces gpu-memory-utilization to 0.80 and tensor-parallel-size to 4, which might help in stabilizing the performance.

Notes

• The actual optimal parameters may vary depending on the specific hardware and model being used.
• It's crucial to monitor the system's resources and the model's performance during these experiments to avoid overloading the GPUs.

Recommendation

• Apply workaround: Adjust the model deployment parameters to find a stable configuration that balances performance and accuracy, as the issue seems to be related to the current configuration rather than a version-specific bug.