vllm - 💡(How to fix) Fix [Feature]: General LL GEMMs with PDL Support [3 comments, 3 participants]

TL;DR

Create generalized versions of the low-latency GEMMS to support multiple data types (bf16, fp8, nvfp4) and arbitrary shapes.

Guidance

• Identify the current limitations of dsv3_a_gemm and dsv3_router_gemm in terms of supported shapes and data types.
• Determine the requirements for the generalized GEMMS, including support for bf16, fp8, nvfp4, and arbitrary shapes.
• Review the PDL fix in flashinfer (issue #2887) for potential insights or solutions that can be applied to the generalized GEMMS.
• Consider the potential impact of supporting arbitrary shapes on performance and latency.

Example

No specific code snippet can be provided without more context, but the generalized GEMMS could potentially involve modifying the existing dsv3_a_gemm and dsv3_router_gemm functions to accept more flexible shape and data type parameters.

Notes

The creation of generalized GEMMS may require significant changes to the existing codebase, and careful consideration should be given to ensuring that the new implementations maintain the low-latency performance of the current specialized GEMMS.

Recommendation

Apply workaround: Create generalized versions of the low-latency GEMMS to support multiple data types and arbitrary shapes, as this will provide the most flexibility and compatibility for future use cases.