pytorch - 💡(How to fix) Fix [Inductor][NVGEMM] tile_M<128 EFC kernels broadcast incorrectly [1 participants]

TL;DR

Remove the tile_M >= 128 constraint from _include_efc_kernels_only and the -1 branch from _classify once the cutlass_api bug is fixed to enable EFC kernels with smaller tiles.

Guidance

• Identify the cutlass_api bug fix version to determine when to apply the changes.
• Remove the tile_M >= 128 constraint from _include_efc_kernels_only to allow EFC kernels with smaller tiles.
• Update the _classify callback in _add_nv_gemm_choices_impl to remove the -1 branch that drops EFC kernels with tile_M < 128.
• Verify the changes by testing EFC kernels with smaller tiles (e.g., tile_M=64) and checking for correct results.

Notes

The provided solution assumes that the cutlass_api bug has been fixed. If the bug fix version is not available, the workaround of filtering out EFC kernels with tile_M < 128 should remain in place.

Recommendation

Apply the workaround until the cutlass_api bug is fixed, then remove the constraints to enable EFC kernels with smaller tiles, as this will allow for more efficient autotuning and potentially better performance for small-M decode shapes.