vllm - 💡(How to fix) Fix [Performance]: llmcompressor W8A8 Inference: decoding stage speed is lower than FP16 [1 participants]

TL;DR

Investigate and potentially update the kernel used in the llmcompressor to improve performance, specifically considering the decoding stage where the W8A8 kernel is currently slower than FP16.

Guidance

• Review the performance comparison between W8A8 and FP16 kernels in the decoding stage to understand the root cause of the regression.
• Examine the SmoothQuant paper's claims about the decoding stage performance to see if there are any specific optimizations or configurations that could be applied to the llmcompressor.
• Consider testing the llmcompressor with different kernel configurations to verify if switching to a different kernel, such as FP16, improves performance as expected.
• Look into the collect_env.py output for any relevant environment or configuration details that might be contributing to the performance issue.

Example

No specific code example is provided due to the lack of direct code references in the issue, but investigating kernel configurations and their impact on the decoding stage performance is key.

Notes

The issue seems to be related to a specific performance regression in the llmcompressor, and the provided images and references suggest a comparison between different kernel performances. However, without more details on the current environment and configurations, it's challenging to provide a definitive solution.

Recommendation

Apply workaround: Investigate and potentially update the kernel used in the llmcompressor, as the current W8A8 kernel seems to be the bottleneck in the decoding stage, and there's evidence suggesting that FP16 could offer better performance.