Your current environment

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 4.3.1
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 | packaged by Anaconda, Inc. | (main, May  6 2024, 19:46:43) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-5.15.0-124-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : False
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   : N/A
GPU models and configuration : No devices found.
Nvidia driver version        : Could not collect
cuDNN version                : Probably one of the following:
...
==============================
         vLLM Info
==============================
vLLM Version                 : 0.1.dev15639+g1472223c4.d20260416 (git sha: 1472223c4, date: 20260416)
...
==============================
     Environment Variables
==============================
OMP_NUM_THREADS=1
MKL_NUM_THREADS=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root


Note: the attached `collect_env.py` output was captured when the instance was in a no-GPU state, so it reports `Is CUDA available: False` / `No devices found`. The bug itself was reproduced earlier on H800 / SM90 with GPU available; the issue here is not that the code failed to load, but that after the SM90 swap_ab port the failing cases move from the old explicit `m must be divisible by 4` check to CUTLASS `can_implement() -> kInvalid`.

Describe the Bug

This appears to be a separate SM90 FP8 blockwise issue, independent from the decode-side performance investigation in #38697.

On SM90, FP8 blockwise currently hard-fails for cases such as m=1 or m=33 with:

m must be divisible by 4

I tried porting the SM100/SM120-style swap_ab graceful handling to SM90. That removes the original hard check, and the new path is definitely exercised, but all minimal repro cases now fail earlier in CUTLASS can_implement() with:

cutlass_gemm_caller can_implement failed with kInvalid (Invalid status, code=11)

So the issue is no longer the old explicit m % 4 == 0 guard. The failure has moved into the SM90 CUTLASS blockwise implementation itself.

Minimal Repro Cases

All of the following fail consistently in the new SM90 swap_ab path:

• m=1, n=256, k=128
• m=1, n=16384, k=1024
• m=33, n=1024, k=1024
• m=33, n=8192, k=128

Before the port, these were blocked by the explicit m must be divisible by 4 check.
After the port, they all reach ops.cutlass_scaled_mm(...), but fail in can_implement() with kInvalid.

Findings

I compared the SM90 host-side swap_ab argument construction against the SM100/SM120 implementations and did not find an obvious missing host-side field swap.

The following are all swapped consistently:

• ptr_A/B
• dA/dB
• ptr_SFA/SFB
• layout_SFA/SFB
• prob_shape
• c_stride
• LayoutC/D transpose

The most likely root cause appears to be the scale path, not the ordinary A/B pointer swap.

Under the current SM90 non-array TMA blockwise mainloop, with the swap_ab config:

• ScaleGranularity = (128, 1, 128)
• TileShape = (128, 32, 128)

the mainloop compiles into a form where SFB is loaded via TMA. Then can_implement() performs TMA alignment checks on args.layout_SFB.

For these failing cases, layout_SFB ends up with a dynamic stride that depends on the original m. When m=1 or m=33, that stride is not 4-float aligned, so check_alignment<4>(args.layout_SFB) fails and the adapter only reports kInvalid.

Why This Looks SM90-Specific

SM100/SM120 already have graceful handling for:

• m < 16
• m % 4 != 0

However, they use a different blockwise layout/mainloop family.

The current SM90 path appears to be a non-array TMA blockwise kernel family with stricter scale-layout alignment requirements, so directly porting the SM100/SM120 host-side swap_ab logic is not sufficient.

Conclusion

This no longer looks like test-only busywork or a missing host-side swap field.

It looks like the current SM90 non-array TMA FP8 blockwise path does not support the intended swap_ab semantics for small or non-4-aligned M, because the scale-layout alignment constraints fail in can_implement().

So this likely needs either:

• a different SM90 kernel family/path for this case, or
• an explicit statement that SM90 does not support this graceful swap_ab handling yet.

Old vs New Behavior

Local Investigation Patch

The investigation patch was applied in:

• csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm90_fp8_dispatch.cuh
• csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/cutlass_gemm_caller.cuh (for expanded status reporting)
• tests/kernels/quantization/test_cutlass_scaled_mm.py

Local Logs

I have local logs for the 4 minimal repro cases showing the post-port kInvalid failure path:

• blockwise_case_1_256_128_status.log
• blockwise_case_1_16384_1024_status.log
• blockwise_case_33_1024_1024_status.log
• blockwise_case_33_8192_128_status.log

I can paste the full traceback if needed.