vllm - 💡(How to fix) Fix [Feature]: Support for Asymmetric W4A16 (compressed-tensors) for MoE Architectures (Qwen3.5-122B) [1 participants]

When I tried to infer the AWQ quantized (using llmcompressor) Qwen3.5-122B model using vllm, I am getting the following error, "AssertionError: Only symmetric quantization is supported for MoE"

TL;DR

The error "AssertionError: Only symmetric quantization is supported for MoE" suggests that asymmetric quantization is not currently supported for the specified model, so a workaround or alternative quantization scheme may be necessary.

Guidance

• The error message indicates that the AWQModifier with the scheme "W4A16_ASYM" is not compatible with the MoE model, so consider using a symmetric quantization scheme instead.
• Review the AWQModifier documentation to see if there are any other quantization schemes that are supported for MoE models.
• If asymmetric quantization is required, consider reaching out to the developers or waiting for a future update that may add support for asymmetric quantization.
• Verify that the vllm version (0.19.0) and Transformers version (5.6.0.dev0) are compatible with the specified model and quantization tool.

Example

No code example is provided as the issue is related to the compatibility of the quantization scheme with the model.

Notes

The issue lacks information about the specific requirements for asymmetric quantization, so it's unclear if a workaround is possible without compromising the model's performance.

Recommendation

Apply workaround: Use a symmetric quantization scheme instead of asymmetric quantization, as it is currently supported for MoE models.