vllm - 💡(How to fix) Fix [Bug]: CUTLASS block FP8 kernel failure on Blackwell GB300 [5 comments, 2 participants]

cutlass_scaled_mm_supports_block_fp8() incorrectly reports support for Blackwell (cc=10.3, to_int() = 103 ≥ 100), but the kernel raises RuntimeError: Error Internal at runtime during the profile/dummy run. The patch forces the Triton fallback for all Blackwell devices, which works correctly. However it has a perf penalty.

TL;DR

The issue can be mitigated by modifying the cutlass_block_fp8_supported function to return False for Blackwell devices with a capability of 100 or higher, forcing a fallback to Triton.

Guidance

• The cutlass_scaled_mm_supports_block_fp8 function incorrectly reports support for Blackwell devices, causing a runtime error.
• The provided patch forces the Triton fallback for all Blackwell devices, which works correctly but has a performance penalty.
• To verify the issue, run the provided vllm serve command with the specified parameters.
• The modified cutlass_block_fp8_supported function can be used as a temporary workaround to avoid the runtime error.

Example

def cutlass_block_fp8_supported() -> bool:
    capability_tuple = current_platform.get_device_capability()
    capability = -1 if capability_tuple is None else capability_tuple.to_int()
    
    # CUTLASS block FP8 kernels fail at runtime on Blackwell (B300, cc >= 100).
    # Fall back to Triton instead.
    if capability >= 100:
        return False
    
    return ops.cutlass_scaled_mm_supports_block_fp8(capability)

Notes

The provided patch is a temporary workaround, and a more permanent solution may be required to avoid the performance penalty.

Recommendation

Apply workaround: The modified cutlass_block_fp8_supported function can be used as a temporary workaround to avoid the runtime error, although it may have a performance penalty.