PR #40127: fix: add SM>=89 guard for Triton block FP8 and Marlin fallback on Ampere

• Repository: vllm-project/vllm
• Author: Sandermage
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/40127

Description (problem / solution / changelog)

Summary

Triton block FP8 kernel (TritonFp8BlockScaledMMKernel) uses fp8e4nv format which requires SM >= 89 (Ada Lovelace+). On Ampere GPUs (SM 86, e.g. RTX A5000/A6000), this causes a silent failure when loading block-quantized FP8 models (e.g. block_shape=[128,128]).

Changes

1. TritonFp8BlockScaledMMKernel.is_supported() — SM capability check

• Query current_platform.get_device_capability() when compute_capability is not passed
• Return False with descriptive message when SM < 89
• Follows the same pattern used by other kernels (e.g. DeepGemmFp8BlockScaledMMKernel)

2. get_fp8_linear_kernel() — graceful fallback

• Wrap block FP8 kernel selection in try/except ValueError
• When all block FP8 kernels are unsupported, fall back to per-tensor FP8 kernels (_POSSIBLE_FP8_KERNELS) which includes Marlin
• Marlin FP8 works correctly on SM 86 with per-tensor quantization

Testing

Tested on 2× NVIDIA RTX A5000 (SM 86, Ampere) with:

• Model: Qwen3.6-35B-A3B-FP8 (MoE, block_shape=[128,128])
• vLLM: v0.19.1 nightly, V1 engine, tensor_parallel_size=2
• Result: 145+ tok/s generation speed, stable across 10 benchmark runs
• Without this fix: crash during model loading (no supported FP8 block kernel)

Related

• Closes https://github.com/vllm-project/vllm/issues/40124 (Ampere FP8 part)
• Genesis Project patches: https://github.com/Sandermage/genesis-vllm-patches

Changed files

• vllm/model_executor/kernels/linear/__init__.py (modified, +46/-17)
• vllm/model_executor/kernels/linear/scaled_mm/triton.py (modified, +9/-0)

---

PR #40128: fix: handle non-divisible page sizes in hybrid model KV cache unification

• Repository: vllm-project/vllm
• Author: Sandermage
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/40128

Description (problem / solution / changelog)

Summary

unify_kv_cache_spec_page_size() raises NotImplementedError when page sizes across different layer types are not evenly divisible. This breaks hybrid models that mix attention (with TurboQuant KV cache) and recurrent layers (Mamba/DeltaNet).

Root cause: TurboQuant k8v4 with head_dim=256 yields page_size = block_size × num_kv_heads × 388 = 12416 bytes, while DeltaNet/Mamba state is ~12.6 MiB. 12648448 % 12416 ≠ 0, triggering the error.

Changes

Replace the hard NotImplementedError with LCM-based padding:

1. Fast path preserved: when all smaller page sizes divide max_page_size evenly, behavior is unchanged
2. Slow path (new): compute LCM of all smaller page sizes, pad max_page_size UP to the nearest multiple of that LCM
3. For the padded layer, use page_size_padded via dataclasses.replace(); for layers that divide evenly, scale block_size as before
4. Memory overhead is typically <0.1% (logged at INFO level)

Testing

Tested on Qwen3.6-35B-A3B-FP8 (hybrid: 30 MoE + 10 dense layers) with TurboQuant k8v4 KV cache on 2× RTX A5000:

• Model loads successfully with unified page sizes
• 145+ tok/s, 160k context window, 10/10 stability runs

Without this fix: crash at model init with NotImplementedError: The page size of the layer is not divisible by the maximum page size.

Related

• Refs https://github.com/vllm-project/vllm/issues/40124
• Genesis Project patches: https://github.com/Sandermage/genesis-vllm-patches

Changed files

• vllm/v1/core/kv_cache_utils.py (modified, +51/-11)

---

PR #40129: perf: add tuned Triton MoE configs for RTX A5000 (E=256, N=512)

• Repository: vllm-project/vllm
• Author: Sandermage
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/40129

Description (problem / solution / changelog)

Summary

Add pre-tuned Triton kernel configurations for the NVIDIA RTX A5000 (SM 86, Ampere) with E=256 experts, N=512 intermediate size.

There are currently no tuned MoE configs for Ampere consumer/workstation GPUs. The Triton autotuner defaults work but leave ~15-20% performance on the table.

Tuning Methodology

Configs were generated using vLLM's built-in Triton autotuner on 2× RTX A5000 with:

• Model: Qwen3.6-35B-A3B-FP8 (256 experts, top_k=8, block_shape=[128,128])
• Batch sizes: 1, 2, 4, 8, 16, 32, 64, 128, 256
• Best configs selected by lowest kernel execution time across 100 iterations per batch size

Key observations for SM 86 (Ampere):

• BLOCK_SIZE_M=16 is optimal — MoE routes few tokens per expert
• BLOCK_SIZE_K varies 64-256 depending on batch size
• num_stages kept conservative (1-4) due to SM 86 shared memory constraints

Performance

File

vllm/model_executor/layers/fused_moe/configs/E=256,N=512,device_name=NVIDIA_RTX_A5000.json

Follows the same naming convention as existing configs (e.g., E=256,N=512,device_name=NVIDIA_H100_80GB_HBM3.json).

Related

• Refs https://github.com/vllm-project/vllm/issues/40124
• Genesis Project: https://github.com/Sandermage/genesis-vllm-patches

Changed files

• vllm/model_executor/layers/fused_moe/configs/E=256,N=512,device_name=NVIDIA_RTX_A5000,dtype=fp8_w8a8,block_shape=[128,128].json (added, +74/-0)