Issue Station
Sign In
Tools hereToolbox

vllm - 💡(How to fix) Fix [Bug]: Regression can no longer load Qwen 3.5 397B nvfp4 model - CUBLAS_STATUS_NOT_INITIALIZED [1 comments, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#38666•Fetched 2026-04-08 01:58:39
View on GitHub
Comments
1
Participants
1
Timeline
3
Reactions
0
Author
bitbottrapbitbottrap
Participants
bitbottrapbitbottrap
Timeline (top)
closed ×1commented ×1labeled ×1

Error Message

(Worker_TP3 pid=27021) ERROR 03-31 18:15:56 [multiproc_executor.py:949] RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasLtMatmulAlgoGetHeuristic( ltHandle, computeDesc.descriptor(), Adesc.descriptor(), Bdesc.descriptor(), Cdesc.descriptor(), Cdesc.descriptor(), preference.descriptor(), 1, &heuristicResult, &returnedResult) And still get the error.

Code Example

host: 0.0.0.0        # Bind to all interfaces (required for remote access / containers)
port: 8081           # HTTP API port
model: /mnt/models/Qwen3.5-397B-A17B-NVFP4-sehyo   # Active model (absolute path, no trailing slash)
served-model-name: qwen3.5-397b-a17b                     # Name returned by /v1/models and used by clients
tensor-parallel-size: 4          # Split model across 4 GPUs (tensor parallelism)
enable-auto-tool-choice: true       # Allow automatic tool selection
tool-call-parser: qwen3_coder       # Tool-call schema parser
reasoning-parser: qwen3             # Structured reasoning output parser
gpu-memory-utilization: 0.90        # Use up to 90% of available GPU memory
override-generation-config:
  temperature: 0.6
  top_p: 0.95
  top_k: 20
  min_p: 0.0
RAW_BUFFERClick to expand / collapse

Your current environment

env.txt

See attached environment and log.

Running CI builds and in the last day or so something broke. vllm crashes when attempting to start when loading Qwen 3.5 397B A17B - Seyho. I had been using tensor and expert parallel and other features. I simplified the configuration and it still is failing. This is across 4x RTX 6000 GPUs.

(Worker_TP3 pid=27021) ERROR 03-31 18:15:56 [multiproc_executor.py:949] RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasLtMatmulAlgoGetHeuristic( ltHandle, computeDesc.descriptor(), Adesc.descriptor(), Bdesc.descriptor(), Cdesc.descriptor(), Cdesc.descriptor(), preference.descriptor(), 1, &heuristicResult, &returnedResult)

log.txt

I've simplified my configuration down to:

host: 0.0.0.0        # Bind to all interfaces (required for remote access / containers)
port: 8081           # HTTP API port
model: /mnt/models/Qwen3.5-397B-A17B-NVFP4-sehyo   # Active model (absolute path, no trailing slash)
served-model-name: qwen3.5-397b-a17b                     # Name returned by /v1/models and used by clients
tensor-parallel-size: 4          # Split model across 4 GPUs (tensor parallelism)
enable-auto-tool-choice: true       # Allow automatic tool selection
tool-call-parser: qwen3_coder       # Tool-call schema parser
reasoning-parser: qwen3             # Structured reasoning output parser
gpu-memory-utilization: 0.90        # Use up to 90% of available GPU memory
override-generation-config:
  temperature: 0.6
  top_p: 0.95
  top_k: 20
  min_p: 0.0

And still get the error.

🐛 Describe the bug

vllm server crashes while starting on recent CI builds.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The most likely fix is to investigate and resolve the CUDA initialization issue, potentially by checking CUDA versions, updating CUDA drivers, or adjusting GPU memory utilization.

Guidance

  • Verify that the CUDA version is compatible with the version of cuBLAS being used, as the error CUBLAS_STATUS_NOT_INITIALIZED suggests a potential version mismatch or initialization issue.
  • Check the GPU memory utilization configuration (gpu-memory-utilization: 0.90) to ensure it's not causing memory allocation issues that could prevent CUDA from initializing properly.
  • Review the system's CUDA installation and driver versions to ensure they are up-to-date and compatible with the application's requirements.
  • Consider temporarily reducing the tensor-parallel-size to see if the issue persists, which could help isolate if the problem is related to multi-GPU configuration.

Example

No specific code example is provided as the issue seems related to environment and configuration rather than code syntax.

Notes

The solution may depend on the specifics of the CI environment, CUDA versions, and GPU drivers in use, which are not fully detailed in the provided information.

Recommendation

Apply a workaround by adjusting the gpu-memory-utilization setting to a lower value (e.g., 0.80) to reduce memory pressure and see if the server can start successfully, then investigate further based on the outcome.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#prompt formatting#chain error#conversation history#tool integration

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

vllm - 💡(How to fix) Fix [Bug]: Regression can no longer load Qwen 3.5 397B nvfp4 model - CUBLAS_STATUS_NOT_INITIALIZED [1 comments, 1 participants]