Your current environment

Collecting environment information... uv is set ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39

     PyTorch Info

============================== PyTorch version : 2.10.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A ============================== Python Environment

Python version : 3.12.3 (main, Mar 3 2026, 12:15:18) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA RTX PRO 5000 Blackwell Nvidia driver version : 595.79 cuDNN version : Could not collect ============================== CPU Info

Architecture: x86_64 Model name: Intel(R) Core(TM) Ultra 9 285K CPU(s): 24 Hypervisor vendor: Microsoft ============================== Versions of relevant libraries

[pip3] flashinfer-python==0.6.4 [pip3] numpy==2.2.6 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cuda-runtime-cu12==12.9.79 [pip3] torch==2.10.0+cu130 [pip3] torchvision==0.25.0+cu130 [pip3] transformers==4.57.6 [pip3] triton==3.6.0 ============================== vLLM Info

vLLM Version : 0.17.1 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled

How you are installing vllm

uv pip install vllm --torch-backend=auto

Summary

Setting up vLLM 0.17.1 from PyPI on a Blackwell GPU (RTX PRO 5000, SM120) with CUDA 13.0 on WSL2 requires solving 5 undocumented problems before any Qwen3.5 27B+ model will run. Each failure produces a cryptic error with no actionable guidance. This issue documents the full chain and proposes fixes.

Problem 1: libcudart.so.12 not found

Error:

ImportError: libcudart.so.12: cannot open shared object file: No such file or directory

Root cause: vLLM's C extensions (_C.abi3.so, _moe_C.abi3.so, _vllm_fa2_C.abi3.so, _vllm_fa3_C.abi3.so, _flashmla_C.abi3.so, _flashmla_extension_C.abi3.so) are compiled against CUDA 12 and link libcudart.so.12. CUDA 13 systems only ship libcudart.so.13.

Commonly suggested workaround: LD_LIBRARY_PATH. This is fragile — it doesn't survive process forks (vLLM's multiprocess EngineCore architecture), systemd services, background tasks, or Docker entrypoints.

What actually worked: Installing nvidia-cuda-runtime-cu12 (provides libcudart.so.12) and then running patchelf --set-rpath on all 8 affected .so files to bake in the path. No env vars needed.

Proposed fix: Either:

• (a) Add nvidia-cuda-runtime-cu12 as a dependency and set RPATH in vLLM's build system (CMakeLists.txt)
• (b) Publish versioned wheels (cu12, cu130) on PyPI like PyTorch does, with clear install instructions
• (c) At minimum, detect the mismatch at import time and print an actionable error message

Related: #30435, #31018, #28669, #35432

Problem 2: FlashInfer JIT requires undocumented system dependencies

FlashInfer JIT-compiles CUDA kernels at runtime for GDN/Mamba attention patterns (used by Qwen3.5 27B+). This requires four system packages that aren't documented anywhere:

Each failure is discovered sequentially — you fix one, hit the next. The errors come from deep inside FlashInfer's JIT layer, not vLLM, making them hard to trace.

Proposed fix:

• Document these as system requirements in the installation guide, at least for Blackwell/GDN models
• Detect missing dependencies at startup and print a single actionable message: "FlashInfer JIT compilation requires: gcc, python3-dev, nvcc (cuda-toolkit), ninja-build"
• Consider pre-compiling FlashInfer kernels for common architectures (SM120, SM100) and shipping them in the wheel

Related: #21960, #32826

Problem 3: Qwen3.5 27B+ GDN models silently require --max-num-batched-tokens 2096

Error:

RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Actual root cause (buried in EngineCore subprocess log):

AssertionError: In Mamba cache align mode, block_size (X) must be <= max_num_batched_tokens

All Qwen3.5 models with GDN (Gated DeltaNet) layers — including the 27B dense model (not just MoE variants) — require --max-num-batched-tokens 2096 due to mamba/GDN cache alignment constraints. vLLM's default of 8192 violates this.

The 27B has 64 layers: 16 groups × (3 GDN + 1 Attention). vLLM treats GDN like Mamba for cache alignment, setting attention block size to 1568 tokens to ensure that attention page size is >= mamba page size.

Proposed fix:

• Auto-detect GDN models and cap max_num_batched_tokens at the mamba page size
• At minimum, surface the EngineCore subprocess error message to the user (see Problem 5)
• Document this in the Qwen3.5 recipe

Related: #36010, #35502

Problem 4: --attention-backend FLASH_ATTN + --kv-cache-dtype fp8 = silent crash

Error (again, buried in EngineCore subprocess):

ValueError: Selected backend AttentionBackendEnum.FLASH_ATTN is not valid for this configuration.
Reason: ['kv_cache_dtype not supported']

Users may set --attention-backend FLASH_ATTN as a workaround for FlashInfer crashes (#36828), not knowing it's incompatible with --kv-cache-dtype fp8.

Proposed fix:

• Validate flag combinations before spawning EngineCore subprocesses
• Raise a clear error: "FLASH_ATTN does not support kv-cache-dtype=fp8. Use the default FlashInfer backend or remove --kv-cache-dtype fp8."

Related: #12543, #35577, PR #14221

Problem 5: "Engine core initialization failed" provides no useful information

RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

This error appears for every problem above (and many others — see #17618, #18730, #19002, #21882, #23176, #26898, #33245). The actual root cause is always in the EngineCore subprocess log, which users must manually find at /tmp/vllm-*.log or by adding --log-level DEBUG.

I understand a logging redesign is tracked at #31683. In the meantime, a minimal fix would be:

• Capture the EngineCore subprocess's last exception and include it in the parent's error message
• Change Failed core proc(s): {} to actually list the subprocess PIDs and their exit codes/signals

Reproduction

# On any Blackwell GPU with CUDA 13.0 driver, WSL2 or native Linux
pip install vllm  # or: uv pip install vllm

# Problem 1: immediate crash
python -c "import vllm"
# ImportError: libcudart.so.12: cannot open shared object file

# After fixing libcudart (patchelf or LD_LIBRARY_PATH):
# Problem 2: crash on GDN model load (gcc/nvcc/ninja missing)
vllm serve Qwen/Qwen3.5-27B-FP8

# After installing system deps:
# Problem 3: crash with default --max-num-batched-tokens
vllm serve Qwen/Qwen3.5-27B-FP8 --kv-cache-dtype fp8
# Engine core initialization failed

# Problem 4: crash with explicit FLASH_ATTN + fp8
vllm serve Qwen/Qwen3.5-27B-FP8 --attention-backend FLASH_ATTN --kv-cache-dtype fp8
# Engine core initialization failed

# What actually works (after all fixes):
vllm serve Qwen/Qwen3.5-27B-FP8 --max-num-batched-tokens 2096 --kv-cache-dtype fp8 --dtype bfloat16

Suggested priority

Problems 1 and 5 are the highest impact — they affect every Blackwell user installing from PyPI, and the error messages give no path to resolution. Problems 2-4 compound the frustration but are solvable once you know what to look for.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.