transformers - 💡(How to fix) Fix [Windows] RTX 5070 Ti (Blackwell sm_120) - setup and deployment notes [1 comments, 2 participants]

TL;DR

Set TORCH_CUDA_ARCH_LIST=12.0 and apply the registry fix to utilize the NVIDIA GeForce RTX 5070 Ti Laptop GPU for deploying vLLM.

Guidance

• Verify that the TORCH_CUDA_ARCH_LIST environment variable is set correctly to 12.0 to support the Blackwell architecture.
• Apply the registry fix to set the GPU preference to the NVIDIA GPU, as the default setting routes Python to the Intel iGPU.
• Consider contributing a PR to update the supported architectures list in vLLM to include Blackwell (sm_120) for better support of RTX 5070 Ti/5080/5090 laptops.
• Ensure that the CUDA Toolkit and PyTorch cu130 PATH priorities are correctly configured to avoid DLL conflicts.

Example

export TORCH_CUDA_ARCH_LIST=12.0
export CUDA_VISIBLE_DEVICES=0
export CUDA_MODULE_LOADING=LAZY

And for Windows laptops:

import winreg
key = winreg.OpenKey(winreg.HKEY_CURRENT_USER,
    r'Software\Microsoft\DirectX\UserGpuPreferences',
    0, winreg.KEY_SET_VALUE)
winreg.SetValueEx(key, r'C:\path\to\python.exe', 0, winreg.REG_SZ, 'GpuPreference=2;')

Notes

The provided solution is specific to the NVIDIA GeForce RTX 5070 Ti Laptop GPU and may not apply to other GPU models or architectures.

Recommendation

Apply the workaround by setting TORCH_CUDA_ARCH_LIST=12.0 and applying the registry fix, as there is no indication of a fixed version that includes support for the Blackwell architecture.