vllm - 💡(How to fix) Fix [Bug]: Unable to start Nemotron NVFP4 on 5090 after upgrade to vLLM 0.22

vllm2026-05-29 20:04:06

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

Error Message

After updating to vLLM 0.22.0, and using the same serve command, this failed, generally with the "RuntimeError: CUDA driver error: device not ready" . (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] EngineCore failed to start. (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] Traceback (most recent call last): (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1139, in run_engine_core (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return func(*args, **kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 905, in init (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] super().init( (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 121, in init (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self.model_executor = executor_class(vllm_config) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return func(*args, **kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in init (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self._init_executor() (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 65, in _init_executor (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self.driver_worker.load_model() (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 342, in load_model (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self.model_runner.load_model(load_dummy_weights=load_dummy_weights) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return func(*args, **kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5053, in load_model (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self.model = model_loader.load_model( (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper

Root Cause

WARNING 05-29 22:55:06 [interface.py:729] Using 'pin_memory=False' as WSL is detected. This may slow down the performance. (APIServer pid=402) INFO 05-29 22:55:06 [utils.py:344] (APIServer pid=402) INFO 05-29 22:55:06 [utils.py:344] █ █ █▄ ▄█ (APIServer pid=402) INFO 05-29 22:55:06 [utils.py:344] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.22.0 (APIServer pid=402) INFO 05-29 22:55:06 [utils.py:344] █▄█▀ █ █ █ █ model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 (APIServer pid=402) INFO 05-29 22:55:06 [utils.py:344] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=402) INFO 05-29 22:55:06 [utils.py:344] (APIServer pid=402) INFO 05-29 22:55:06 [utils.py:278] non-default args: {'model_tag': 'nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': 'nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4', 'trust_remote_code': True, 'max_model_len': 8192, 'served_model_name': ['model'], 'reasoning_parser': 'nano_v3', 'reasoning_parser_plugin': 'nano_v3_reasoning_parser.py', 'gpu_memory_utilization': 0.65, 'kv_cache_dtype': 'fp8', 'max_num_seqs': 1} (APIServer pid=402) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. (APIServer pid=402) INFO 05-29 22:55:07 [model.py:617] Resolved architecture: NemotronHForCausalLM (APIServer pid=402) INFO 05-29 22:55:07 [model.py:1752] Using max model len 8192 (APIServer pid=402) INFO 05-29 22:55:08 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor (APIServer pid=402) INFO 05-29 22:55:08 [config.py:413] Updating mamba_ssm_cache_dtype to 'float32' for NemotronH model (APIServer pid=402) WARNING 05-29 22:55:08 [modelopt.py:1022] Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4). Please note that the format is experimental and could change in future. (APIServer pid=402) INFO 05-29 22:55:08 [vllm.py:977] Asynchronous scheduling is enabled. (APIServer pid=402) INFO 05-29 22:55:08 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']) (APIServer pid=402) INFO 05-29 22:55:10 [compilation.py:312] Enabled custom fusions: act_quant (EngineCore pid=552) INFO 05-29 22:55:15 [core.py:112] Initializing a V1 LLM engine (v0.22.0) with config: model='nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4', speculative_config=None, tokenizer='nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=modelopt_fp4, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='nano_v3', reasoning_parser_plugin='nano_v3_reasoning_parser.py', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=model, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 2, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') (EngineCore pid=552) WARNING 05-29 22:55:15 [interface.py:729] Using 'pin_memory=False' as WSL is detected. This may slow down the performance. (EngineCore pid=552) INFO 05-29 22:55:15 [parallel_state.py:1422] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.22.250.214:45809 backend=nccl (EngineCore pid=552) INFO 05-29 22:55:15 [parallel_state.py:1735] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A (EngineCore pid=552) INFO 05-29 22:55:16 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling. (EngineCore pid=552) INFO 05-29 22:55:16 [gpu_model_runner.py:5037] Starting to load model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4... (EngineCore pid=552) INFO 05-29 22:55:16 [init.py:940] Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM (EngineCore pid=552) INFO 05-29 22:55:16 [nvfp4.py:326] Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION']. (EngineCore pid=552) INFO 05-29 22:55:16 [cuda.py:378] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN']. (EngineCore pid=552) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. (EngineCore pid=552) INFO 05-29 22:55:18 [weight_utils.py:922] Filesystem type for checkpoints: EXT4. Checkpoint size: 18.01 GiB. Available RAM: 60.07 GiB. (EngineCore pid=552) INFO 05-29 22:55:18 [weight_utils.py:945] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:04<00:16, 4.21s/it] Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:08<00:13, 4.51s/it] Loading safetensors checkpoint shards: 60% Completed | 3/5 [00:13<00:09, 4.51s/it] Loading safetensors checkpoint shards: 80% Completed | 4/5 [00:20<00:05, 5.32s/it] Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:23<00:00, 4.79s/it] Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:23<00:00, 4.77s/it] (EngineCore pid=552) (EngineCore pid=552) INFO 05-29 22:55:42 [default_loader.py:397] Loading weights took 22.10 seconds (EngineCore pid=552) INFO 05-29 22:55:42 [nvfp4.py:537] Using MoEPrepareAndFinalizeNoDPEPModular (EngineCore pid=552) WARNING 05-29 22:55:42 [kv_cache.py:109] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer). (EngineCore pid=552) WARNING 05-29 22:55:42 [kv_cache.py:123] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint. (EngineCore pid=552) WARNING 05-29 22:55:42 [kv_cache.py:162] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint. (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] EngineCore failed to start. (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] Traceback (most recent call last): (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1139, in run_engine_core (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return func(*args, **kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 905, in init (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] super().init( (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 121, in init (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self.model_executor = executor_class(vllm_config) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return func(*args, **kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in init (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self._init_executor() (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 65, in _init_executor (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self.driver_worker.load_model() (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 342, in load_model (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self.model_runner.load_model(load_dummy_weights=load_dummy_weights) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return func(*args, **kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5053, in load_model (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] self.model = model_loader.load_model( (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return func(*args, **kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 80, in load_model (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] process_weights_after_loading(model, model_config, target_device) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py", line 111, in process_weights_after_loading (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] quant_method.process_weights_after_loading(module) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/modelopt.py", line 1567, in process_weights_after_loading (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ) = convert_to_nvfp4_moe_kernel_format( (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py", line 392, in convert_to_nvfp4_moe_kernel_format (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ) = prepare_nvfp4_moe_layer_for_fi_or_cutlass( (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/utils/flashinfer_fp4_moe.py", line 387, in prepare_nvfp4_moe_layer_for_fi_or_cutlass (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] w2 = torch.nn.functional.pad(w2, (0, pad_size // 2, 0, 0)) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/torch/nn/functional.py", line 5416, in pad (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return handle_torch_function( (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/torch/overrides.py", line 1764, in handle_torch_function (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] result = torch_func_method(public_api, types, args, kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/parameter.py", line 126, in torch_function (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return super().torch_function(func, types, args, kwargs) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/torch/nn/functional.py", line 5430, in pad (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] return torch._C._nn.pad(input, pad, mode, value) (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) ERROR 05-29 22:55:43 [core.py:1165] RuntimeError: CUDA driver error: device not ready (EngineCore pid=552) Process EngineCore: (EngineCore pid=552) Traceback (most recent call last): (EngineCore pid=552) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap (EngineCore pid=552) self.run() (EngineCore pid=552) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run (EngineCore pid=552) self._target(*self._args, **self._kwargs) (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1169, in run_engine_core (EngineCore pid=552) raise e (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1139, in run_engine_core (EngineCore pid=552) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) return func(*args, **kwargs) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 905, in init (EngineCore pid=552) super().init( (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 121, in init (EngineCore pid=552) self.model_executor = executor_class(vllm_config) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) return func(*args, **kwargs) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in init (EngineCore pid=552) self._init_executor() (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 65, in _init_executor (EngineCore pid=552) self.driver_worker.load_model() (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 342, in load_model (EngineCore pid=552) self.model_runner.load_model(load_dummy_weights=load_dummy_weights) (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) return func(*args, **kwargs) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5053, in load_model (EngineCore pid=552) self.model = model_loader.load_model( (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=552) return func(*args, **kwargs) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 80, in load_model (EngineCore pid=552) process_weights_after_loading(model, model_config, target_device) (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py", line 111, in process_weights_after_loading (EngineCore pid=552) quant_method.process_weights_after_loading(module) (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/modelopt.py", line 1567, in process_weights_after_loading (EngineCore pid=552) ) = convert_to_nvfp4_moe_kernel_format( (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py", line 392, in convert_to_nvfp4_moe_kernel_format (EngineCore pid=552) ) = prepare_nvfp4_moe_layer_for_fi_or_cutlass( (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/utils/flashinfer_fp4_moe.py", line 387, in prepare_nvfp4_moe_layer_for_fi_or_cutlass (EngineCore pid=552) w2 = torch.nn.functional.pad(w2, (0, pad_size // 2, 0, 0)) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/torch/nn/functional.py", line 5416, in pad (EngineCore pid=552) return handle_torch_function( (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/torch/overrides.py", line 1764, in handle_torch_function (EngineCore pid=552) result = torch_func_method(public_api, types, args, kwargs) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/model_executor/parameter.py", line 126, in torch_function (EngineCore pid=552) return super().torch_function(func, types, args, kwargs) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/torch/nn/functional.py", line 5430, in pad (EngineCore pid=552) return torch._C._nn.pad(input, pad, mode, value) (EngineCore pid=552) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=552) RuntimeError: CUDA driver error: device not ready [rank0]:[W529 22:55:44.550684197 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) (APIServer pid=402) Traceback (most recent call last): (APIServer pid=402) File "/home/vrosu/LLM/vllm312/bin/vllm", line 10, in <module> (APIServer pid=402) sys.exit(main()) (APIServer pid=402) ^^^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 92, in main (APIServer pid=402) args.dispatch_function(args) (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 148, in cmd (APIServer pid=402) uvloop.run(run_server(args)) (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/uvloop/init.py", line 96, in run (APIServer pid=402) return __asyncio.run( (APIServer pid=402) ^^^^^^^^^^^^^^ (APIServer pid=402) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run (APIServer pid=402) return runner.run(main) (APIServer pid=402) ^^^^^^^^^^^^^^^^ (APIServer pid=402) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run (APIServer pid=402) return self._loop.run_until_complete(task) (APIServer pid=402) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=402) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/uvloop/init.py", line 48, in wrapper (APIServer pid=402) return await main (APIServer pid=402) ^^^^^^^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 678, in run_server (APIServer pid=402) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server_worker (APIServer pid=402) async with build_async_engine_client( (APIServer pid=402) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter (APIServer pid=402) return await anext(self.gen) (APIServer pid=402) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client (APIServer pid=402) async with build_async_engine_client_from_engine_args( (APIServer pid=402) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter (APIServer pid=402) return await anext(self.gen) (APIServer pid=402) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args (APIServer pid=402) async_llm = AsyncLLM.from_vllm_config( (APIServer pid=402) ^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config (APIServer pid=402) return cls( (APIServer pid=402) ^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in init (APIServer pid=402) self.engine_core = EngineCoreClient.make_async_mp_client( (APIServer pid=402) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=402) return func(*args, **kwargs) (APIServer pid=402) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 131, in make_async_mp_client (APIServer pid=402) return AsyncMPClient(*client_args) (APIServer pid=402) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=402) return func(*args, **kwargs) (APIServer pid=402) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 932, in init (APIServer pid=402) super().init( (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 567, in init (APIServer pid=402) with launch_core_engines( (APIServer pid=402) File "/usr/lib/python3.12/contextlib.py", line 144, in exit (APIServer pid=402) next(self.gen) (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1150, in launch_core_engines (APIServer pid=402) wait_for_engine_startup( (APIServer pid=402) File "/home/vrosu/LLM/vllm312/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1209, in wait_for_engine_startup (APIServer pid=402) raise RuntimeError( (APIServer pid=402) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Vendor ID: GenuineIntel Model name: 13th Gen Intel(R) Core(TM) i5-13600KF CPU family: 6 Model: 183 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 1 Stepping: 1 BogoMIPS: 6988.79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni vnmi umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities Virtualization: VT-x Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 480 KiB (10 instances) L1i cache: 320 KiB (10 instances) L2 cache: 20 MiB (10 instances) L3 cache: 24 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-19 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Mitigation; Clear Register File Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Code Example

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   :
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 5090
Nvidia driver version        : Could not collect
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        39 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               20
On-line CPU(s) list:                  0-19
Vendor ID:                            GenuineIntel
Model name:                           13th Gen Intel(R) Core(TM) i5-13600KF
CPU family:                           6
Model:                                183
Thread(s) per core:                   2
Core(s) per socket:                   10
Socket(s):                            1
Stepping:                             1
BogoMIPS:                             6988.79
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni vnmi umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities
Virtualization:                       VT-x
Hypervisor vendor:                    Microsoft
Virtualization type:                  full
L1d cache:                            480 KiB (10 instances)
L1i cache:                            320 KiB (10 instances)
L2 cache:                             20 MiB (10 instances)
L3 cache:                             24 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-19
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Mitigation; Clear Register File
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.11.post2
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cccl==13.3.3.3.1
[pip3] nvidia-cuda-crt==13.3.33
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvcc==13.3.33
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.2
[pip3] nvidia-cutlass-dsl-libs-base==4.5.2
[pip3] nvidia-cutlass-dsl-libs-cu13==4.5.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] nvidia-nvvm==13.3.33
[pip3] pyzmq==27.1.0
[pip3] tokenspeed-triton==3.7.10.post20260505
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.9.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.22.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X                              N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_vrosu

RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   :
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 5090
Nvidia driver version        : Could not collect
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        39 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               20
On-line CPU(s) list:                  0-19
Vendor ID:                            GenuineIntel
Model name:                           13th Gen Intel(R) Core(TM) i5-13600KF
CPU family:                           6
Model:                                183
Thread(s) per core:                   2
Core(s) per socket:                   10
Socket(s):                            1
Stepping:                             1
BogoMIPS:                             6988.79
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni vnmi umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities
Virtualization:                       VT-x
Hypervisor vendor:                    Microsoft
Virtualization type:                  full
L1d cache:                            480 KiB (10 instances)
L1i cache:                            320 KiB (10 instances)
L2 cache:                             20 MiB (10 instances)
L3 cache:                             24 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-19
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Mitigation; Clear Register File
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.11.post2
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cccl==13.3.3.3.1
[pip3] nvidia-cuda-crt==13.3.33
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvcc==13.3.33
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.2
[pip3] nvidia-cutlass-dsl-libs-base==4.5.2
[pip3] nvidia-cutlass-dsl-libs-cu13==4.5.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] nvidia-nvvm==13.3.33
[pip3] pyzmq==27.1.0
[pip3] tokenspeed-triton==3.7.10.post20260505
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.9.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.22.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X                              N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_vrosu

</details>

🐛 Describe the bug

I was running vLLM 0.18 successfully with Nemotron-3-Nano-30B-A3b-NVFP4. After updating to vLLM 0.22.0, and using the same serve command, this failed, generally with the "RuntimeError: CUDA driver error: device not ready" .

Below is my serve command (a script I start with): #!/usr/bin/env bash source /home/vrosu/LLM/vllm312/bin/activate export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True #export VLLM_SLEEP_WHEN_IDLE=1 export CUDA_LAUNCH_BLOCKING=1

vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
--served-model-name model
--max-num-seqs 1
--tensor-parallel-size 1
--max-model-len 8192
--port 8000
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser-plugin nano_v3_reasoning_parser.py
--reasoning-parser nano_v3
--kv-cache-dtype fp8
--gpu-memory-utilization 0.65
--moe-backend auto

This is the output:

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering