vllm - 💡(How to fix) Fix [RFC]: Unified Device Capability Abstraction for Cross-Platform Feature Detection [1 comments, 2 participants]

Motivation.

This is an AI-generated proposal, there may be some error. appreciate if you can point out.

1. Problem Summary

As raised by @tjtanaa in #39158, the current has_device_capability(int) / is_device_capability_family(int) API is fundamentally CUDA-centric and does not translate correctly to ROCm or XPU.

1.1 Problem A: device_capability is inherently a CUDA concept

torch.cuda.get_device_capability() returns (major, minor) tied to NVIDIA's SM (Streaming Multiprocessor) versioning. ROCm and XPU have completely different hardware models — any mapping to CUDA-style numbers is artificial and lossy:

1.2 Problem B: Same CUDA generation, different capability numbers across SKU tiers

Even within NVIDIA's own ecosystem, the same architecture generation has different compute capability numbers depending on the product tier (server / workstation / client):

This is particularly problematic because:

1. Same generation, different numbers: B200 (10,0) and RTX PRO 6000 (12,0) are both Blackwell, both support FP8/FP4, but have completely different capability numbers. Code using is_device_capability_family(100) to gate Blackwell features will miss RTX PRO 6000.

2. Same number, different features: RTX PRO 6000 and RTX 5090 both report (12,0), but RTX PRO supports FP8/FP4 while consumer RTX 5090 does not. Code using has_device_capability(120) to gate FP8 would be wrong on RTX 5090.

3. Maintenance burden: Every new GPU SKU tier requires auditing all numeric capability checks. The current codebase already handles is_device_capability_family(100) and is_device_capability_family(120) separately (e.g., in cutlass_moe.py), and this will only get worse.

1.3 Problem C: Cross-platform semantic mismatch

The same numeric value means completely different things on different platforms:

1.4 Scope observation: where capability checks are used

Most usage of has_device_capability / get_device_capability falls into:

This distribution suggests the migration can be prioritized: fix tests/config first (highest cross-platform impact), then gradually migrate kernel dispatch code.

2. Analysis of Current Codebase

2.1 How capability is actually used (282 call sites)

Analyzing all has_device_capability / is_device_capability* call sites across the codebase, they fall into three distinct categories:

Category A: Testing for a hardware feature (~60%)

# "Does this device support FP8?"
current_platform.has_device_capability(89)   # Actually means: supports_fp8()

# "Does this device support BF16?"
current_platform.has_device_capability(80)   # Actually means: supports_bf16()

# "Does this device support TMA / warp-group MMA?" 
current_platform.has_device_capability(90)   # Actually means: supports_hopper_features()

# "Does this device support Blackwell features (FP4, TMA v2)?"
current_platform.is_device_capability_family(100)  # Actually means: supports_blackwell_features()

These should be feature queries, not numeric comparisons.

Category B: Selecting a kernel implementation (~30%)

# "Use CUTLASS 3.x path for SM90+"
if current_platform.has_device_capability(90): use_cutlass3()

# "Use Blackwell-specific GEMM kernel"
if current_platform.is_device_capability_family(100): use_blackwell_gemm()

# "Use CUTLASS MoE for Blackwell server OR workstation"
if p.is_device_capability_family(100) or p.is_device_capability_family(120): ...

These are CUDA-kernel-specific dispatch decisions. They are inherently platform-specific and do not need cross-platform abstraction — they are always guarded behind is_cuda() check already. However, even here the Blackwell SKU split causes issues (see Section 1.2).

Category C: FP8 weight-only quantization fallback (~10%)

# fbgemm_fp8.py — Marlin WOQ fallback for non-FP8 hardware
self.use_marlin = not current_platform.has_device_capability(89)

# marlin.py — FP8 Marlin works on SM ≥ 7.5 (Turing+)
def is_fp8_marlin_supported():
    return current_platform.has_device_capability(75)

This pattern reveals that FP8 "support" is not binary — there are two distinct levels:

• Native FP8 compute: Hardware tensor cores that natively compute in FP8 (SM ≥ 89 on CUDA, gfx942/gfx950 on ROCm)
• FP8 weight-only quantization (WOQ): Kernels like Marlin that can load FP8-quantized weights and dequantize them to FP16/BF16 for compute (works on SM ≥ 75 on CUDA, i.e., Turing+)

The current supports_fp8() doesn't distinguish between these, which leads to suboptimal decisions. For example, FBGEMMFp8Config uses has_device_capability(89) to decide whether to use native FP8 compute vs Marlin fallback, but this is incorrect on RTX 5090 ((12,0)) which has has_device_capability(89) = True but may not actually have native FP8 tensor cores for all operations.

2.2 Existing feature-level APIs (the right pattern)

The codebase already has the correct abstraction for some features:

# vllm/platforms/interface.py — base class
Platform.supports_fp8()    → bool   # ✅ Cross-platform
Platform.supports_mx()     → bool   # ✅ Cross-platform
Platform.support_deep_gemm() → bool # ✅ Cross-platform
Platform.fp8_dtype()       → dtype  # ✅ Cross-platform
Platform.is_fp8_fnuz()     → bool   # ✅ Cross-platform

Each platform overrides these with the correct implementation:

• CUDA: supports_fp8() → has_device_capability(89)
• ROCm: supports_fp8() → "gfx94" in arch or "gfx95" in arch or "gfx12" in arch
• XPU: inherits False (or could override when XPU gains FP8)

This is the pattern that should be expanded — but with finer granularity (see Section 3).

2.3 Missing feature-level APIs

Proposed Change.

3. Proposal: Two-Layer Capability Model

Layer 1: Feature-Based Queries (Cross-Platform) — PRIMARY

Expand the existing Platform base class with semantic feature queries that each platform implements correctly:

# vllm/platforms/interface.py

class Platform:
    # === Existing (keep) ===
    def supports_fp8(cls) -> bool: ...      # keep for backward compat, alias to supports_fp8_native()
    def supports_mx(cls) -> bool: ...
    def support_deep_gemm(cls) -> bool: ...
    
    # === New feature queries ===
    
    @classmethod
    def supports_bf16(cls) -> bool:
        """Returns whether the current platform supports BF16 compute."""
        return False
    
    @classmethod
    def supports_fp8_native(cls) -> bool:
        """Returns whether the current platform has native FP8 tensor core compute.
        
        This means the hardware can natively perform matrix multiplication in FP8
        (e.g., NVIDIA SM ≥ 89 Ada/Hopper/Blackwell server, AMD gfx942/gfx950).
        Used for: native FP8 GEMM, FP8 KV cache, FP8 attention.
        """
        return False

    @classmethod
    def supports_fp8_woq(cls) -> bool:
        """Returns whether the platform supports FP8 weight-only quantization.
        
        This means kernels like Marlin can load FP8-quantized weights and 
        dequantize them to FP16/BF16 for compute. Works on significantly older
        hardware than native FP8 (e.g., NVIDIA SM ≥ 75 Turing+).
        Used for: Marlin FP8, fbgemm FP8 fallback path.
        """
        return False

    @classmethod
    def supports_fp4(cls) -> bool:
        """Returns whether the current platform supports FP4 quantization 
        (native compute or equivalent)."""
        return False

    @classmethod
    def supports_tma(cls) -> bool:
        """Returns whether the current platform supports 
        Tensor Memory Accelerator (or equivalent async copy engine)."""
        return False

    @classmethod
    def supports_fp8_kv_cache(cls) -> bool:
        """Returns whether the current platform supports FP8 KV cache."""
        return cls.supports_fp8_native()

    @classmethod
    def get_architecture_family(cls) -> str:
        """Returns human-readable architecture family name.
        
        Examples: 'hopper', 'blackwell', 'blackwell_consumer','cdna3', 
                  'rdna4', 'ponte_vecchio', 'unknown'
        """
        return "unknown"

Why split FP8 into native vs WOQ?

The current supports_fp8() is ambiguous. In the codebase today:

# fbgemm_fp8.py — decides whether to use native FP8 or Marlin fallback
self.use_marlin = not current_platform.has_device_capability(89)

# marlin_utils_fp8.py — Marlin FP8 works on much older hardware
def is_fp8_marlin_supported():
    return current_platform.has_device_capability(75)  # Turing+!

With the split:

# fbgemm_fp8.py — clear intent
self.use_marlin = not current_platform.supports_fp8_native()

# marlin_utils_fp8.py — explicit WOQ check
def is_fp8_marlin_supported():
    return current_platform.supports_fp8_woq()

This distinction matters for:

• Testing: A test for native FP8 GEMM should use @requires_feature("fp8_native"), while a test for Marlin FP8 WOQ should use @requires_feature("fp8_woq")
• Consumer GPUs: RTX 5090 (SM 12.0) may support FP8 WOQ via Marlin but not native FP8 tensor core compute
• ROCm: MI200 (gfx90a) has no FP8 at all, MI300 (gfx942) has native FP8-FNUZ, different Marlin support

Layer 2: Numeric Capability (Platform-Specific) — KEEP BUT DEPRECATE for cross-platform use

Keep has_device_capability() / is_device_capability() / is_device_capability_family() as-is for platform-specific kernel dispatch, but:

1. Document that these are CUDA/ROCm-specific and must always be guarded by is_cuda() / is_rocm().
2. Deprecation warning in tests when used without platform guard (enforce via lint rule).
3. XPU/CPU/TPU continue to return None / False — this is correct behavior.
4. For kernel dispatch: These remain the correct API when selecting between CUDA-specific kernel implementations (e.g., CUTLASS 2.x vs 3.x). Such code is inherently platform-specific and doesn't need cross-platform abstraction.

5. Migration Plan

Phase 0: Add feature methods to Platform (this RFC)

Phase 1: Convert feature-gated capability checks in vllm/ source

Convert ~60% of has_device_capability calls that are actually feature checks:

# Before
if current_platform.has_device_capability(89):
    # FP8 native path
elif current_platform.has_device_capability(75):
    # FP8 Marlin WOQ fallback

# After  
if current_platform.supports_fp8_native():
    # FP8 native path
elif current_platform.supports_fp8_woq():
    # FP8 Marlin WOQ fallback

Priority conversion targets (most impactful):

• has_device_capability(89) → supports_fp8_native() (~15 sites in vllm/)
• has_device_capability(75) for Marlin → supports_fp8_woq() (~3 sites)
• has_device_capability(80) → supports_bf16() (~8 sites)
• is_device_capability_family(100) → supports_fp4() or architecture check (~20 sites)
• not has_device_capability(89) for use_marlin → not supports_fp8_native() and supports_fp8_woq() (~5 sites)

Phase 2: Convert test skip patterns

Integrate with the test skip RFC (#39158) to migrate test files:

# Before (CUDA-only, broken on ROCm, wrong on consumer GPUs)
@pytest.mark.skipif(not current_platform.has_device_capability(89), reason="need fp8")

# After (cross-platform, correct on all SKUs)
@requires_feature("fp8_native")

Note: Since vLLM CI always uses NVIDIA server GPUs, the migration can be incremental — existing tests won't break, but new tests should use the feature-based API.

Phase 3: Lint enforcement

Add pre-commit hook to warn on new has_device_capability usage in test files without a is_cuda() / is_rocm() guard.

Feedback Period.

1-2 weeks.

CC List.

@tjtanaa

Any Other Things.

6. Comprehensive Device Capability Reference

6.1 NVIDIA CUDA — torch.cuda.get_device_capability()

Server / Data Center GPUs

Workstation / Pro GPUs

Consumer / GeForce GPUs

Key insight: RTX PRO 6000 and RTX 5090 both report (12, 0), but RTX PRO supports FP8/FP4 while consumer RTX 5090 does not. Capability number alone is insufficient for feature detection on SM 12.0.

6.2 AMD ROCm — GCN Architecture → Mapped Capability

Key insight: ROCm's (9, 0) = MI200 (NO FP8) vs CUDA's (9, 0) = Hopper (HAS FP8). The same number has opposite meanings.

6.3 Feature to Capability Mapping — Why Numeric Checks Fail

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.