vllm - 💡(How to fix) Fix [Installation]: not using prebuilt wheels + memory usage shoots up to 30+GB [1 participants]

TL;DR

The issue can be mitigated by increasing the swap space or using a machine with more memory to accommodate the CUDA building process, and ensuring that pre-built wheels are used from PyPI.

Guidance

• Verify that the pip version is up-to-date, as older versions may not properly handle pre-built wheels.
• Check the PyPI repository for available pre-built wheels for the vllm library to ensure they exist for the specific CUDA version (13.0) and Python version being used.
• Consider setting the TORCH_CUDA_ARCH_LIST environment variable to specify the exact CUDA architecture, which may help reduce the memory requirements during the build process.
• If possible, try installing vllm without the --torch-backend=auto flag to see if it makes a difference in using pre-built wheels.

Example

No code snippet is provided as it is not clearly supported by the issue.

Notes

The solution may not apply if the pre-built wheels are not available for the specific environment or if there are other underlying issues with the system configuration.

Recommendation

Apply workaround: increase swap space or use a machine with more memory, and investigate why pre-built wheels are not being used. This is because the root cause of the issue is likely related to the system's memory constraints and the build process, rather than a specific version of the library.