vllm - 💡(How to fix) Fix [Bug]: kimi k2.5 Int4 default `flashinfer_trtllm_mxint4_moe` not working B300 SM103

(Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 132, in _moe_forward_shared (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] return runner.forward_impl( (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] ^^^^^^^^^^^^^^^^^^^^ (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 815, in forward_impl (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] shared_output, hidden_states = self._apply_quant_method( (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 492, in _apply_quant_method (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] result = self.quant_method.apply_monolithic( (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py", line 1616, in apply_monolithic (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] return flashinfer_trtllm_mxint4_moe( (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/flashinfer_mxint4_moe.py", line 264, in flashinfer_trtllm_mxint4_moe (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] ).to(x.dtype) (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] ^^ (Worker_TP5 pid=2071791) ERROR 04-20 23:50:01 [multiproc_executor.py:949] AttributeError: 'list' object has no attribute 'to'

TL;DR

The error is likely due to an incorrect data type being passed to the flashinfer_trtllm_mxint4_moe function, which expects a tensor but receives a list instead.

Guidance

• Review the code in flashinfer_mxint4_moe.py to ensure that the input to flashinfer_trtllm_mxint4_moe is correctly typed and not a list.
• Verify that the x variable in the error message has the expected data type, and that the to method is applicable to it.
• Check the documentation for flashinfer_trtllm_mxint4_moe to confirm its input requirements and ensure that the calling code meets these requirements.
• Consider adding type checking or debugging statements to identify where the incorrect data type is being introduced.

Example

No code example is provided due to the lack of context about the surrounding code.

Notes

The error message suggests a specific line of code where the error occurs, but without more context, it's difficult to provide a precise fix. The solution will depend on the intended functionality of the code and the expected data types.

Recommendation

Apply a workaround by modifying the code to ensure that the input to flashinfer_trtllm_mxint4_moe is correctly typed, or investigate and fix the root cause of the incorrect data type being passed.