vllm - 💡(How to fix) Fix [Bug]: Running DeepSeek-V4 fails with  CUDA error: unspecified launch failure  in  synchronize_input_prep [1 comments, 1 participants]

Your current environment

Model: `DeepSeek v4 flash

• CUDA version: 13.0
• GPU model: H100*8
• OS: Ubuntu 20.04 LTS
• Container runtime: Docker deepseekv4 cu130

🐛 Describe the bug

vLLM fails to run DeepSeek-V4 with CUDA launch failure

When trying to serve DeepSeek-V4 using vLLM, the worker process crashes immediately with a CUDA error: unspecified launch failure error. The stack trace points to synchronize_input_prep in gpu_model_runner.py, and the process hangs before any requests are processed.

Steps to reproduce the behavior:

1. Run the following vLLM command to serve DeepSeek-V4:

   docker run --runtime=nvidia --gpus all \
   --name deepseek-v4 \
   -v /mnt/data/modelscope/deepseek-ai/DeepSeek-V4-Flash:/model \
   --env CUDA_VISIBLE_DEVICES=0,1,2,3 \
   --env VLLM_USE_DEEP_GEMM=1 \
   -p 8025:8000 \
   --ipc=host \
   --shm-size=32g \
   vllm/vllm-openai:deepseekv4-cu130 \
   --model /model \
   --served-model-name deepseek-v4 \
   --gpu-memory-utilization 0.85 \
   --max-num-seqs 8 \
   --max-model-len auto \
   --block-size 256 \
   --tensor-parallel-size 4 \
   --enable-chunked-prefill \
   --enable-expert-parallel \
   --enable-prefix-caching \
   --enable-auto-tool-choice \
   --tokenizer-mode deepseek_v4 \
   --tool-call-parser deepseek_v4 \
   --reasoning-parser deepseek_v4 \
   --trust-remote-code \
   --kv-cache-dtype fp8 \
   --no-disable-hybrid-kv-cache-manager

2. The server starts but immediately throws the following error and hangs:

   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3492, in synchronize_input_prep
       self.prepare_inputs_event.synchronize()
   torch.AcceleratorError: CUDA error: unspecified launch failure
   Search for 'cudaErrorLaunchFailure' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
   CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
   For debugging consider passing CUDA_LAUNCH_BLOCKING=1
   Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
   
   prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cach...
   broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work
   ``

Additional context

• The error occurs even with minimal inference requests; GPU utilization and memory usage are low at the time of the crash.
• I have tried setting CUDA_LAUNCH_BLOCKING=1 and TORCH_USE_CUDA_DSA, but the issue persists.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.