vllm - ✅(Solved) Fix [Bug]: Misleading error message for FP8 quantization: refers to CUDA version instead of GPU compute capability [3 pull requests, 1 comments, 2 participants]

vllm2026-03-11 16:45:01

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

vllm-project/vllm#36805•Fetched 2026-04-08 00:34:32

View on GitHub

Comments

Participants

Timeline

Reactions

Author

jranaraki

Participants

github-actions[bot]

jranaraki

Timeline (top)

referenced ×5subscribed ×4cross-referenced ×3mentioned ×3

Error Message

AssertionError: Float8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+

Root Cause

The assertion originates from TorchAO:

torchao/quantization/quant_api.py

Relevant code:

    if torch.cuda.is_available():
        assert is_sm_at_least_89() or is_MI300(), (
            "Float8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+"
        )

The check calls:

is_sm_at_least_89()

which refers to GPU compute capability ≥ 8.9, not a CUDA toolkit version. Therefore, the error message:

CUDA>=8.9

is incorrect and should instead refer to SM / compute capability ≥ 8.9.

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: GenuineIntel Model name: Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz CPU family: 6 Model: 166 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 Stepping: 0 CPU(s) scaling MHz: 64% CPU max MHz: 4700.0000 CPU min MHz: 400.0000 BogoMIPS: 3199.92 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp vnmi md_clear flush_l1d arch_capabilities ibpb_exit_to_user Virtualization: VT-x L1d cache: 192 KiB (6 instances) L1i cache: 192 KiB (6 instances) L2 cache: 1.5 MiB (6 instances) L3 cache: 12 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Vulnerable Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace

PR fix notes

PR #36854: [Bugfix] Clear error message for FP8 torchao quantization on unsupported GPUs

Repository: vllm-project/vllm
Author: haosdent
State: open | merged: False
Link: https://github.com/vllm-project/vllm/pull/36854

Description (problem / solution / changelog)

Purpose

Fixes #36805. When using FP8 activation torchao configs (e.g., Float8DynamicActivationFloat8WeightConfig) on GPUs with compute capability < 8.9, torchao raises a confusing AssertionError: Float8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+. The message says "CUDA>=8.9" but actually refers to GPU compute capability, not CUDA toolkit version.

This PR adds a proactive capability check in torchao_quantize_param_data() that intercepts the error before it reaches torchao and provides a clear, actionable message:

torchao FP8 activation quantization config 'Float8DynamicActivationFloat8WeightConfig' requires GPU compute capability >= 8.9 (e.g., NVIDIA Ada Lovelace / Hopper or AMD MI300+) (current GPU compute capability: 7.5). For older GPUs, consider using a non-FP8 config such as Int8WeightOnlyConfig or Int4WeightOnlyConfig.

Test Plan

Test Result

Changed files

vllm/model_executor/layers/quantization/torchao.py (modified, +33/-0)

PR #37033: fix: correct FP8 error message to reference compute capability

Repository: vllm-project/vllm
Author: alvinttang
State: open | merged: False
Link: https://github.com/vllm-project/vllm/pull/37033

Description (problem / solution / changelog)

Summary

Fixes misleading FP8 quantization error that says "CUDA>=8.9" when the actual requirement is GPU compute capability >= 8.9
Adds an early validation check in vllm's torchao integration (torchao_quantize_param_data) that raises a clear ValueError before torchao's confusing assertion fires
The new error message tells users exactly what is needed: a GPU with compute capability >= 8.9 (e.g., Ada Lovelace or Hopper GPUs)

Fixes #36805

Details

The upstream torchao assertion in quant_api.py checks is_sm_at_least_89() but produces the error:

Float8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+

This misleads users into thinking they need a newer CUDA toolkit version. The actual requirement is GPU compute capability (SM version) >= 8.9, which is a hardware property.

This PR adds two helper functions:

_is_float8_config(): detects whether a torchao config uses Float8 quantization
_check_fp8_compute_capability(): validates GPU compute capability and raises a clear error

The check runs in torchao_quantize_param_data() before quantize_() is called, so the user sees a clear message like:

ValueError: Float8 quantization requires a GPU with compute capability >= 8.9
(e.g., Ada Lovelace or Hopper GPUs), but the current GPU has compute
capability 7.5. This is a GPU hardware limitation, not a CUDA toolkit
version issue.

Test plan

Verify on a GPU with compute capability < 8.9 (e.g., GTX 1650, T4) that the new error message appears when loading an FP8 torchao model
Verify on a GPU with compute capability >= 8.9 (e.g., RTX 4090, H100) that FP8 quantization still works normally
Verify non-FP8 torchao quantization is unaffected on older GPUs

🤖 Generated with Claude Code

Changed files

vllm/model_executor/layers/quantization/torchao.py (modified, +48/-0)

PR #37036: fix: correct misleading FP8 error message about CUDA version vs compute capability

Repository: vllm-project/vllm
Author: gambletan
State: closed | merged: False
Link: https://github.com/vllm-project/vllm/pull/37036

Description (problem / solution / changelog)

Summary

Fixes #36805: The upstream TorchAO assertion error says CUDA>=8.9 when FP8 dynamic activation quantization fails, but the actual requirement is GPU compute capability >= 8.9 (SM89+), not a CUDA toolkit version. This is confusing for users with older GPUs like the GTX 1650 (compute 7.5).
Wraps the quantize_() call in torchao_quantize_param_data with a try/except to catch this specific assertion and re-raise with a clear message mentioning compute capability (e.g., Ada Lovelace, Hopper) and AMD MI300+.

Test plan

Verify the error message text matches the upstream TorchAO assertion ("CUDA>=8.9")
Test on a GPU with compute capability < 8.9 using an FP8 TorchAO quantized model to confirm the new error message appears
Test on a GPU with compute capability >= 8.9 to confirm normal operation is unaffected

🤖 Generated with Claude Code

Changed files

vllm/model_executor/layers/quantization/torchao.py (modified, +11/-1)

Code Example

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.10.19 (main, Feb 12 2026, 00:42:18) [Clang 21.1.4 ] (64-bit runtime)
Python platform              : Linux-6.8.0-101-generic-x86_64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.0.140
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA GeForce GTX 1650 with Max-Q Design
Nvidia driver version        : 590.48.01
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           39 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  12
On-line CPU(s) list:                     0-11
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz
CPU family:                              6
Model:                                   166
Thread(s) per core:                      2
Core(s) per socket:                      6
Socket(s):                               1
Stepping:                                0
CPU(s) scaling MHz:                      64%
CPU max MHz:                             4700.0000
CPU min MHz:                             400.0000
BogoMIPS:                                3199.92
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp vnmi md_clear flush_l1d arch_capabilities ibpb_exit_to_user
Virtualization:                          VT-x
L1d cache:                               192 KiB (6 instances)
L1i cache:                               192 KiB (6 instances)
L2 cache:                                1.5 MiB (6 instances)
L3 cache:                                12 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-11
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Vulnerable
Vulnerability Itlb multihit:             KVM: Mitigation: VMX disabled
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.1
[pip3] nvidia-cutlass-dsl-libs-base==4.4.1
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pynvml==13.0.1
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchao==0.16.0
[pip3] torchaudio==2.10.0
[pip3] torchvision==0.25.0
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.16.1rc1.dev296+ga73af584f (git sha: a73af584f)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
  	GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-11	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_user

---

AssertionError: Float8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+

---

if torch.cuda.is_available():
        assert is_sm_at_least_89() or is_MI300(), (
            "Float8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+"
        )

---

Float8 dynamic activation quantization is only supported on GPUs with compute capability >= 8.9 (e.g., Ada GPUs) or AMD MI300+.

RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.10.19 (main, Feb 12 2026, 00:42:18) [Clang 21.1.4 ] (64-bit runtime)
Python platform              : Linux-6.8.0-101-generic-x86_64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.0.140
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA GeForce GTX 1650 with Max-Q Design
Nvidia driver version        : 590.48.01
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           39 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  12
On-line CPU(s) list:                     0-11
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz
CPU family:                              6
Model:                                   166
Thread(s) per core:                      2
Core(s) per socket:                      6
Socket(s):                               1
Stepping:                                0
CPU(s) scaling MHz:                      64%
CPU max MHz:                             4700.0000
CPU min MHz:                             400.0000
BogoMIPS:                                3199.92
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp vnmi md_clear flush_l1d arch_capabilities ibpb_exit_to_user
Virtualization:                          VT-x
L1d cache:                               192 KiB (6 instances)
L1i cache:                               192 KiB (6 instances)
L2 cache:                                1.5 MiB (6 instances)
L3 cache:                                12 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-11
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Vulnerable
Vulnerability Itlb multihit:             KVM: Mitigation: VMX disabled
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.1
[pip3] nvidia-cutlass-dsl-libs-base==4.4.1
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pynvml==13.0.1
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchao==0.16.0
[pip3] torchaudio==2.10.0
[pip3] torchvision==0.25.0
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.16.1rc1.dev296+ga73af584f (git sha: a73af584f)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
  	GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-11	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_user

</details>

Bug description

When running vLLM with an FP8 TorchAO quantized model, the following error occurs:

AssertionError: Float8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+

However, this message is misleading because CUDA toolkit versions are unrelated to the requirement being checked. The code path appears to check GPU compute capability (SM version) instead.

Root cause

The assertion originates from TorchAO:

torchao/quantization/quant_api.py

Relevant code:

    if torch.cuda.is_available():
        assert is_sm_at_least_89() or is_MI300(), (
            "Float8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+"
        )

The check calls:

is_sm_at_least_89()

which refers to GPU compute capability ≥ 8.9, not a CUDA toolkit version. Therefore, the error message:

CUDA>=8.9

is incorrect and should instead refer to SM / compute capability ≥ 8.9.

Expected behaviour

The message should clarify that the requirement refers to GPU compute capability, not CUDA toolkit version.

Example of clearer wording:

Float8 dynamic activation quantization is only supported on GPUs with compute capability >= 8.9 (e.g., Ada GPUs) or AMD MI300+.

🐛 Describe the bug

Incorrect error message: Float8 dynamic activation quantization refers to CUDA version instead of compute capability

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

To fix the issue, we need to update the error message in torchao/quantization/quant_api.py to reflect the correct requirement.

Update the assert statement to include a clearer error message:

if torch.cuda.is_available():
    assert is_sm_at_least_89() or is_MI300(), (
        "Float8 dynamic activation quantization is only supported on GPUs with compute capability >= 8.9 (e.g., Ada GPUs) or AMD MI300+"
    )

Optionally, consider adding a check for the CUDA toolkit version to provide additional information in the error message.

Verification

To verify the fix, run the vLLM model with an FP8 TorchAO quantized model and check that the error message is updated correctly.

Extra Tips

When working with GPU-accelerated libraries, it's essential to understand the differences between CUDA toolkit versions, GPU compute capability, and GPU architecture.
Consider adding documentation or comments to clarify these distinctions and avoid similar issues in the future.
If you're experiencing issues with GPU compatibility, check the NVIDIA documentation for information on CUDA toolkit versions, GPU compute capability, and supported architectures.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #dependency conflict #environment setup #docker error #environment variable

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

vllm - ✅(Solved) Fix [Bug]: Misleading error message for FP8 quantization: refers to CUDA version instead of GPU compute capability [3 pull requests, 1 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

PR fix notes

PR #36854: [Bugfix] Clear error message for FP8 torchao quantization on unsupported GPUs

Description (problem / solution / changelog)

Purpose

Test Plan

Test Result

Changed files

PR #37033: fix: correct FP8 error message to reference compute capability

Description (problem / solution / changelog)

Summary

Details

Test plan

Changed files

PR #37036: fix: correct misleading FP8 error message about CUDA version vs compute capability

Description (problem / solution / changelog)

Summary

Test plan

Changed files

Code Example

Your current environment

Bug description

Root cause

Expected behaviour

🐛 Describe the bug

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING