vllm - ✅(Solved) Fix [Bug]: vLLM 0.17.0 failed to serve Qwen3-30B-A3B-Instruct-2507 after adding `--enable_lora` [1 pull requests, 1 comments, 2 participants]

vllm2026-03-11 11:24:58

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

vllm-project/vllm#36781•Fetched 2026-04-08 00:34:44

View on GitHub

Comments

Participants

Timeline

Reactions

Author

Participants

Assignees

Timeline (top)

assigned ×1commented ×1cross-referenced ×1labeled ×1

Error Message

(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] WorkerProc hit an exception. (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] Traceback (most recent call last): (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/v1/executor/multiproc_executor.py", line 875, in worker_busy_loop (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] output = func(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/utils/contextlib.py", line 124, in decorate_context (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return func(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_worker.py", line 390, in determine_available_memory (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] self.model_runner.profile_run() (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5282, in profile_run (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] hidden_states, last_hidden_states = self.dummy_run( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/utils/contextlib.py", line 124, in decorate_context (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return func(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4976, in dummy_run (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] outputs = self.model( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/cuda_graph.py", line 223, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.runnable(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in wrapped_call_impl (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.call_impl(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1787, in call_impl (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return forward_call(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 783, in forward (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] hidden_states = self.model( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/decorators.py", line 563, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] output = self.aot_compiled_fn(self, *args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/dynamo/aot_compile.py", line 124, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.fn(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 470, in forward (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] def forward( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/caching.py", line 198, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.optimized_call(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 936, in call_wrapped (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.wrapped_call(self, *args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 455, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] raise e (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 442, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc] (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in wrapped_call_impl (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.call_impl(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1787, in call_impl (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return forward_call(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "<eval_with_key>.99", line 843, in forward (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] submod_2 = self.submod_2(getitem_3, s72, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_modules_base_layer_parameters_weight, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_token_lora_mapping, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_token_indices_sorted_by_lora_ids, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_a_stacked_0, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_num_tokens_per_lora, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_lora_token_start_loc, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_active_lora_ids, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_no_lora_flag_cpu, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_default_num_active_loras_cpu, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_b_stacked_0, l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight, getitem_4, l_self_modules_layers_modules_0_modules_mlp_modules_gate_modules_base_layer_parameters_weight, l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_a_stacked_0, l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_b_stacked_0, l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_modules_base_layer_parameters_weight, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_0, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_1, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_2, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_0_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_1_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_2_, l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_); getitem_3 = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_modules_base_layer_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_a_stacked_0_ = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_b_stacked_0_ = l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_ = getitem_4 = l_self_modules_layers_modules_0_modules_mlp_modules_gate_modules_base_layer_parameters_weight_ = l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_a_stacked_0_ = l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_b_stacked_0_ = l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_modules_base_layer_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_0_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_1_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_2_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_0_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_1_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_2_ = l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight_ = None (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/cuda_graph.py", line 223, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.runnable(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/piecewise_backend.py", line 343, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return range_entry.runnable(*args) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_inductor/standalone_compile.py", line 122, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self._compiled_fn(*args) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/eval_frame.py", line 1181, in _fn (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return fn(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/aot_autograd.py", line 1148, in forward (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return compiled_fn(full_args) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1962, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.compiled_fn(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 357, in runtime_wrapper (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] all_outs = call_func_at_runtime_with_args( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 134, in call_func_at_runtime_with_args (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] out = normalize_as_list(f(args)) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 531, in wrapper (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return compiled_fn(runtime_args) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_inductor/output_code.py", line 638, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.current_callable(inputs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_inductor/utils.py", line 3220, in run (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] out = model(new_inputs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/tmp/torchinductor_tiger/fd/cfdmqeuu22zyshstmjjpuszlamqp4lyxl75wi3lr55hvb4cjyqaa.py", line 1146, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] buf16 = torch.ops.vllm.moe_forward.default(buf9, buf13, None, 'from_forward_context') (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_ops.py", line 819, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self._op(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 85, in _moe_forward (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return layer.runner.forward_impl( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 689, in forward_impl (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] final_hidden_states = self.quant_method.apply( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/fused_moe_modular_method.py", line 94, in apply (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.moe_kernel.apply( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/lora/layers/fused_moe.py", line 172, in wrapper (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] result = func(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1753, in apply (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.impl.apply( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1537, in apply (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] fused_out = self._fused_experts( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1375, in _fused_experts (Worker pid=263605) (Worker_TP0 pid=263605) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] WorkerProc hit an exception. (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] Traceback (most recent call last): (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/v1/executor/multiproc_executor.py", line 875, in worker_busy_loop (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] output = func(*args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return func(*args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_worker.py", line 390, in determine_available_memory (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] self.model_runner.profile_run() (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5282, in profile_run (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] hidden_states, last_hidden_states = self._dummy_run( (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return func(*args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4976, in _dummy_run (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] outputs = self.model( (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/cuda_graph.py", line 223, in call (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.runnable(*args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self._call_impl(*args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return forward_call(*args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 783, in forward (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] hidden_states = self.model( (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/decorators.py", line 563, in call (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] output = self.aot_compiled_fn(self, *args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/aot_compile.py", line 124, in call (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.fn(*args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 470, in forward (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] def forward( (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/caching.py", line 198, in call (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self.optimized_call(*args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 936, in call_wrapped (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self._wrapped_call(self, *args, **kwargs) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 455, in call (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] raise e (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 442, in call (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc] (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self._call_impl(*args, **kwargs) ERROR 03-11 19:12:05 [multiproc_executor.py:880] self.fused_experts.apply( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/fused_moe.py", line 2121, in apply (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] self.activation( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/lora/layers/fused_moe.py", line 248, in wrapper (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] self.punica_wrapper.add_lora_fused_moe( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/lora/punica_wrapper/punica_gpu.py", line 427, in add_lora_fused_moe (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] fused_moe_lora( (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/_ops.py", line 1209, in call (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return self._op(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] return func(*args, **kwargs) (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] File "/usr/local/lib/python3.11/dist-packages/vllm/lora/ops/triton_ops/fused_moe_lora_op.py", line 725, in _fused_moe_lora (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] assert output.shape[0] == topk_weights.shape[0] (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] AssertionError

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 180 On-line CPU(s) list: 0-179 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8457C CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 45 Socket(s): 2 Stepping: 8 BogoMIPS: 5200.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities Hypervisor vendor: KVM Virtualization type: full L1d cache: 4.2 MiB (90 instances) L1i cache: 2.8 MiB (90 instances) L2 cache: 180 MiB (90 instances) L3 cache: 195 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-89 NUMA node1 CPU(s): 90-179 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; TSX disabled

PR fix notes

PR #37193: [LoRA] Add LoRA support for Qwen3OmniMoeThinkerForConditionalGeneration

Repository: vllm-project/vllm
Author: pratapyash
State: open | merged: False
Link: https://github.com/vllm-project/vllm/pull/37193

Description (problem / solution / changelog)

Purpose

Add LoRA support for Qwen3OmniMoeThinkerForConditionalGeneration.

The vLLM supported_models.md documentation marks this model with a LoRA checkmark under both "Multimodal Language Models" and "Speech-to-Text Language Models" tables, but the model class never implemented the SupportsLoRA protocol. This means --enable-lora fails at runtime despite the documentation claiming support.

This PR resolves that gap by adding the minimal required LoRA attributes to the model class in a single file.

FIX #31205 Related: #30461, PR #34097

Changes

Import and inherit SupportsLoRA -- enables --enable-lora for this model
Define packed_modules_mapping -- maps qkv_proj to [q_proj, k_proj, v_proj]. gate_up_proj is intentionally excluded because Qwen3-Omni uses MoE (FusedMoE) for FFN layers, not packed linear projections
Define embedding_modules = {} -- required by the SupportsLoRA protocol; empty because no embedding-layer LoRA is needed
Define lora_skip_prefixes = ["audio_tower.", "visual."] -- gracefully skips audio/vision tower modules during LoRA loading. Without this, adapters trained with broad target_modules (e.g., regex matching all Linear layers) crash with ValueError from check_unexpected_modules even though the thinker modules are valid. Follows the same pattern as NemotronH (lora_skip_prefixes = ["mtp."])

Why `gate_up_proj` is removed from `packed_modules_mapping`

Qwen3-Omni is MoE -- FFN uses FusedMoE with per-expert gate_proj/up_proj/down_proj, not a packed gate_up_proj. The inherited mapping came from Qwen2_5OmniThinkerForConditionalGeneration which is a dense model. Qwen3MoeForCausalLM (the authoritative MoE reference) deliberately excludes gate_up_proj and only adds it conditionally when mlp_only_layers is non-empty. Qwen3-Omni has mlp_only_layers: []. Keeping it is harmless (never matched) but misleading -- MoE expert LoRA is handled by FusedMoEWithLoRA, not packed_modules_mapping.

Test Plan

Public test adapter (random weights): yashpratap/Qwen3-Omni-30B-A3B-LoRA-test-r32

Server launch:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --tensor-parallel-size 4 \
  --enable-lora \
  --max-lora-rank 32 \
  --max-loras 2 \
  --lora-modules test=yashpratap/Qwen3-Omni-30B-A3B-LoRA-test-r32 \
  --enforce-eager \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

Inference:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "test", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}'

Test Result

Tested on 4x L40S (48GB each).

Test 1: Enforce-eager mode -- PASS

Base model and LoRA adapter both listed in /v1/models
Base model response: standard Qwen-Omni intro
LoRA adapter response: different output confirming adapter is applied
No errors or warnings

Test 2: CUDA graph compilation -- PASS

VLLM_COMPILE + PIECEWISE cudagraph mode
Both base model and LoRA inference work correctly

Test 3: Mixed adapter (thinker + tower modules) -- PASS

Adapter containing both thinker attention and audio/vision tower LoRA weights
Tower modules gracefully skipped via lora_skip_prefixes
Thinker modules loaded and applied correctly
Server starts, inference works

Test 4: With gate_up_proj in mapping -- PASS (but removed anyway)

Tested keeping gate_up_proj in packed_modules_mapping
No errors or crashes -- the mapping entry is never matched since no module is named gate_up_proj in the MoE architecture
Removed to stay consistent with Qwen3MoeForCausalLM which deliberately excludes it for MoE models

Purpose is clearly described
Test plan provided
Test results included
No documentation update needed (LoRA checkmark already present in supported_models.md)

AI assistance was used in developing and testing this PR, per AGENTS.md.

Changed files

vllm/model_executor/models/qwen3_omni_moe_thinker.py (modified, +8/-4)

Code Example

==============================
        System Info
==============================
OS                           : Debian GNU/Linux 12 (bookworm) (x86_64)
GCC version                  : (Debian 12.2.0-14+deb12u1) 12.2.0
Clang version                : Could not collect
CMake version                : version 3.31.6
Libc version                 : glibc-2.36

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] (64-bit runtime)
Python platform              : Linux-5.10.135.bsk.6-amd64-x86_64-with-glibc2.36

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H20
GPU 1: NVIDIA H20
GPU 2: NVIDIA H20
GPU 3: NVIDIA H20
GPU 4: NVIDIA H20
GPU 5: NVIDIA H20
GPU 6: NVIDIA H20
GPU 7: NVIDIA H20

Nvidia driver version        : 535.261.03
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.11.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   52 bits physical, 57 bits virtual
Byte Order:                      Little Endian
CPU(s):                          180
On-line CPU(s) list:             0-179
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) Platinum 8457C
CPU family:                      6
Model:                           143
Thread(s) per core:              2
Core(s) per socket:              45
Socket(s):                       2
Stepping:                        8
BogoMIPS:                        5200.00
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       4.2 MiB (90 instances)
L1i cache:                       2.8 MiB (90 instances)
L2 cache:                        180 MiB (90 instances)
L3 cache:                        195 MiB (2 instances)
NUMA node(s):                    2
NUMA node0 CPU(s):               0-89
NUMA node1 CPU(s):               90-179
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; TSX disabled

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cudnn-cu12==9.16.0.29
[pip3] nvidia-cudnn-frontend==1.19.0
[pip3] nvidia-cutlass-dsl==4.4.1
[pip3] nvidia-cutlass-dsl-libs-base==4.4.1
[pip3] nvidia-mathdx==25.6.0
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-modelopt==0.42.0
[pip3] nvidia-nccl-cu12==2.29.7
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-resiliency-ext==0.4.1
[pip3] onnx==1.20.1
[pip3] onnx-ir==0.2.0
[pip3] onnxscript==0.6.2
[pip3] pynvml==13.0.1
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu129
[pip3] torchdata==0.11.0
[pip3] torchvision==0.25.0+cu129
[pip3] transformer_engine_torch==2.10.0
[pip3] transformers==5.3.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.17.0
vLLM Build Flags:
  CUDA Archs: 9.0; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    CPU Affinity    NUMA Affinity      GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-89    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     0-89    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     0-89    0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     0-89    0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    90-179  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    90-179  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    90-179  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     90-179  1               N/A
NIC0    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC1    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC2    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    SYS     SYS     SYS     SYS
NIC3    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     SYS     NODE    NODE     X      NODE    SYS     SYS     SYS     SYS
NIC4    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      SYS     SYS     SYS     SYS
NIC5    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC6    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC7    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     SYS     NODE    NODE     X      NODE
NIC8    SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=GPU-d405f5e4-2255-6979-21c3-d3dfad7e1602,GPU-b87464ae-49ec-a8d6-fc09-2946ff688e92,GPU-8a324685-816c-a62d-052c-7f1b3c0a966d,GPU-332ce080-fa9e-4d46-896d-5c26cc156085,GPU-c720634a-4de4-9ad3-7bdb-72ffb3826408,GPU-7607694a-0c7d-9bcd-03ec-f7ac3c8f8a5f,GPU-a19c6142-4556-2f04-29ae-5d9cdacbb44a,GPU-5b702ffb-34e9-236b-7dd3-542f6321355e
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NCCL_NET_PLUGIN=none
NCCL_RAS_ENABLE=0
TORCH_CUDA_ARCH_LIST=9.0
NCCL_SOCKET_IFNAME==eth0
NCCL_NVLS_ENABLE=0
NCCL_DEBUG_SUBSYS=INIT,ENV,GRAPH
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NCCL_DEBUG=INFO
VLLM_WORKER_MULTIPROC_METHOD=spawn
NCCL_IB_HCA=^=mlx5_0
NCCL_IB_GID_INDEX=3
VLLM_ALLREDUCE_USE_SYMM_MEM=0
TORCH_USE_CUDA_DSA=1
NCCL_SOCKET_FAMILY=AF_INET6
NCCL_IB_TIMEOUT=23
MAX_JOBS=32
LD_LIBRARY_PATH=/opt/tiger/native_libhdfs/lib/native:/opt/tiger/jdk/jdk8u265-b01/jre/lib/amd64/server:/opt/tiger/yarn_deploy/hadoop_current/lib/native:/opt/tiger/yarn_deploy/hadoop_current/lib/native/ufs:/opt/tiger/yarn_deploy/hadoop/lib/native:/opt/tiger/yarn_deploy/hadoop_current/lib/native:/opt/tiger/yarn_deploy/hadoop_current/lzo/lib:/usr/local/lib/python3.11/dist-packages/nvidia/nccl/lib:/usr/local/cuda/lib64
NCCL_IB_DISABLE=0
OMP_NUM_THREADS=20
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
NCCL_IB_RETRY_CNT=7
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_tiger

---

vllm serve ./Qwen3-30B-A3B-Instruct-2507/ \
    --dtype bfloat16 \
    --load_format safetensors \
    --distributed_executor_backend mp \
    --worker_extension_cls 'verl.workers.rollout.vllm_rollout.utils.vLLMColocateWorkerExtension' \
    --max-model-len 262144 \
    --max_num_seqs 1024 \
    --enable_chunked_prefill \
    --max_num_batched_tokens 32768 \
    --enable_prefix_caching \
    --enable_sleep_mode \
    --logprobs_mode processed_logprobs \
    --gpu-memory-utilization 0.6 \
    --tensor-parallel-size 4 \
    --seed 0 \
    --override_generation_config '{"temperature": 1.0, "top_k": -1, "top_p": 1, "repetition_penalty": 1.0, "max_new_tokens": 2048}' \
    --hf_overrides '{}' \
    --scheduling_policy fcfs \
    --compilation_config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
    --enable_lora \
    --max_loras 1 \
    --max_lora_rank 64

---

(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] WorkerProc hit an exception.
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] Traceback (most recent call last):
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/executor/multiproc_executor.py", line 875, in worker_busy_loop
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     output = func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_worker.py", line 390, in determine_available_memory
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.model_runner.profile_run()
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5282, in profile_run
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     hidden_states, last_hidden_states = self._dummy_run(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                                         ^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4976, in _dummy_run
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     outputs = self.model(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]               ^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/cuda_graph.py", line 223, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.runnable(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._call_impl(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return forward_call(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 783, in forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     hidden_states = self.model(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                     ^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/decorators.py", line 563, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     output = self.aot_compiled_fn(self, *args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.fn(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 470, in forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     def forward(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/caching.py", line 198, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.optimized_call(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._wrapped_call(self, *args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     raise e
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._call_impl(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return forward_call(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "<eval_with_key>.99", line 843, in forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     submod_2 = self.submod_2(getitem_3, s72, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_modules_base_layer_parameters_weight_, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_token_lora_mapping, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_token_indices_sorted_by_lora_ids, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_a_stacked_0_, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_num_tokens_per_lora, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_lora_token_start_loc, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_active_lora_ids, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_no_lora_flag_cpu, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_default_num_active_loras_cpu, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_b_stacked_0_, l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_, getitem_4, l_self_modules_layers_modules_0_modules_mlp_modules_gate_modules_base_layer_parameters_weight_, l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_a_stacked_0_, l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_b_stacked_0_, l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_modules_base_layer_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_0_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_1_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_2_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_0_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_1_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_2_, l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_3 = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_modules_base_layer_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_a_stacked_0_ = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_b_stacked_0_ = l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_ = getitem_4 = l_self_modules_layers_modules_0_modules_mlp_modules_gate_modules_base_layer_parameters_weight_ = l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_a_stacked_0_ = l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_b_stacked_0_ = l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_modules_base_layer_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_0_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_1_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_2_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_0_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_1_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_2_ = l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight_ = None
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/cuda_graph.py", line 223, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.runnable(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/piecewise_backend.py", line 343, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return range_entry.runnable(*args)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_inductor/standalone_compile.py", line 122, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._compiled_fn(*args)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/eval_frame.py", line 1181, in _fn
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return fn(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/aot_autograd.py", line 1148, in forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return compiled_fn(full_args)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1962, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.compiled_fn(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 357, in runtime_wrapper
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     all_outs = call_func_at_runtime_with_args(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 134, in call_func_at_runtime_with_args
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     out = normalize_as_list(f(args))
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                             ^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 531, in wrapper
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return compiled_fn(runtime_args)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_inductor/output_code.py", line 638, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.current_callable(inputs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_inductor/utils.py", line 3220, in run
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     out = model(new_inputs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]           ^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/tmp/torchinductor_tiger/fd/cfdmqeuu22zyshstmjjpuszlamqp4lyxl75wi3lr55hvb4cjyqaa.py", line 1146, in call
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     buf16 = torch.ops.vllm.moe_forward.default(buf9, buf13, None, 'from_forward_context')
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_ops.py", line 819, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._op(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 85, in _moe_forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return layer.runner.forward_impl(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 689, in forward_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     final_hidden_states = self.quant_method.apply(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                           ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/fused_moe_modular_method.py", line 94, in apply
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.moe_kernel.apply(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/lora/layers/fused_moe.py", line 172, in wrapper
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     result = func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1753, in apply
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.impl.apply(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1537, in apply
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     fused_out = self._fused_experts(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                 ^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1375, in _fused_experts
(Worker pid=263605) (Worker_TP0 pid=263605) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] WorkerProc hit an exception.
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] Traceback (most recent call last):
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/executor/multiproc_executor.py", line 875, in worker_busy_loop
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     output = func(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_worker.py", line 390, in determine_available_memory
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.model_runner.profile_run()
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5282, in profile_run
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     hidden_states, last_hidden_states = self._dummy_run(
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                                         ^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4976, in _dummy_run
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     outputs = self.model(
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]               ^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/cuda_graph.py", line 223, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.runnable(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._call_impl(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return forward_call(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 783, in forward
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     hidden_states = self.model(
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                     ^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/decorators.py", line 563, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     output = self.aot_compiled_fn(self, *args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.fn(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 470, in forward
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     def forward(
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/caching.py", line 198, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.optimized_call(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._wrapped_call(self, *args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     raise e
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._call_impl(*args, **kwargs)
ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.fused_experts.apply(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/fused_moe.py", line 2121, in apply
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.activation(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/lora/layers/fused_moe.py", line 248, in wrapper
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.punica_wrapper.add_lora_fused_moe(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/lora/punica_wrapper/punica_gpu.py", line 427, in add_lora_fused_moe
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     fused_moe_lora(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_ops.py", line 1209, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._op(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/lora/ops/triton_ops/fused_moe_lora_op.py", line 725, in _fused_moe_lora
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     assert output.shape[0] == topk_weights.shape[0]
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] AssertionError

RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

==============================
        System Info
==============================
OS                           : Debian GNU/Linux 12 (bookworm) (x86_64)
GCC version                  : (Debian 12.2.0-14+deb12u1) 12.2.0
Clang version                : Could not collect
CMake version                : version 3.31.6
Libc version                 : glibc-2.36

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] (64-bit runtime)
Python platform              : Linux-5.10.135.bsk.6-amd64-x86_64-with-glibc2.36

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H20
GPU 1: NVIDIA H20
GPU 2: NVIDIA H20
GPU 3: NVIDIA H20
GPU 4: NVIDIA H20
GPU 5: NVIDIA H20
GPU 6: NVIDIA H20
GPU 7: NVIDIA H20

Nvidia driver version        : 535.261.03
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.11.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.11.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   52 bits physical, 57 bits virtual
Byte Order:                      Little Endian
CPU(s):                          180
On-line CPU(s) list:             0-179
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) Platinum 8457C
CPU family:                      6
Model:                           143
Thread(s) per core:              2
Core(s) per socket:              45
Socket(s):                       2
Stepping:                        8
BogoMIPS:                        5200.00
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       4.2 MiB (90 instances)
L1i cache:                       2.8 MiB (90 instances)
L2 cache:                        180 MiB (90 instances)
L3 cache:                        195 MiB (2 instances)
NUMA node(s):                    2
NUMA node0 CPU(s):               0-89
NUMA node1 CPU(s):               90-179
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; TSX disabled

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cudnn-cu12==9.16.0.29
[pip3] nvidia-cudnn-frontend==1.19.0
[pip3] nvidia-cutlass-dsl==4.4.1
[pip3] nvidia-cutlass-dsl-libs-base==4.4.1
[pip3] nvidia-mathdx==25.6.0
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-modelopt==0.42.0
[pip3] nvidia-nccl-cu12==2.29.7
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-resiliency-ext==0.4.1
[pip3] onnx==1.20.1
[pip3] onnx-ir==0.2.0
[pip3] onnxscript==0.6.2
[pip3] pynvml==13.0.1
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu129
[pip3] torchdata==0.11.0
[pip3] torchvision==0.25.0+cu129
[pip3] transformer_engine_torch==2.10.0
[pip3] transformers==5.3.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.17.0
vLLM Build Flags:
  CUDA Archs: 9.0; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    CPU Affinity    NUMA Affinity      GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-89    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     0-89    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     0-89    0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     0-89    0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    90-179  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    90-179  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    90-179  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     90-179  1               N/A
NIC0    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC1    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC2    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    SYS     SYS     SYS     SYS
NIC3    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     SYS     NODE    NODE     X      NODE    SYS     SYS     SYS     SYS
NIC4    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      SYS     SYS     SYS     SYS
NIC5    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC6    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC7    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     SYS     NODE    NODE     X      NODE
NIC8    SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=GPU-d405f5e4-2255-6979-21c3-d3dfad7e1602,GPU-b87464ae-49ec-a8d6-fc09-2946ff688e92,GPU-8a324685-816c-a62d-052c-7f1b3c0a966d,GPU-332ce080-fa9e-4d46-896d-5c26cc156085,GPU-c720634a-4de4-9ad3-7bdb-72ffb3826408,GPU-7607694a-0c7d-9bcd-03ec-f7ac3c8f8a5f,GPU-a19c6142-4556-2f04-29ae-5d9cdacbb44a,GPU-5b702ffb-34e9-236b-7dd3-542f6321355e
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NCCL_NET_PLUGIN=none
NCCL_RAS_ENABLE=0
TORCH_CUDA_ARCH_LIST=9.0
NCCL_SOCKET_IFNAME==eth0
NCCL_NVLS_ENABLE=0
NCCL_DEBUG_SUBSYS=INIT,ENV,GRAPH
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NCCL_DEBUG=INFO
VLLM_WORKER_MULTIPROC_METHOD=spawn
NCCL_IB_HCA=^=mlx5_0
NCCL_IB_GID_INDEX=3
VLLM_ALLREDUCE_USE_SYMM_MEM=0
TORCH_USE_CUDA_DSA=1
NCCL_SOCKET_FAMILY=AF_INET6
NCCL_IB_TIMEOUT=23
MAX_JOBS=32
LD_LIBRARY_PATH=/opt/tiger/native_libhdfs/lib/native:/opt/tiger/jdk/jdk8u265-b01/jre/lib/amd64/server:/opt/tiger/yarn_deploy/hadoop_current/lib/native:/opt/tiger/yarn_deploy/hadoop_current/lib/native/ufs:/opt/tiger/yarn_deploy/hadoop/lib/native:/opt/tiger/yarn_deploy/hadoop_current/lib/native:/opt/tiger/yarn_deploy/hadoop_current/lzo/lib:/usr/local/lib/python3.11/dist-packages/nvidia/nccl/lib:/usr/local/cuda/lib64
NCCL_IB_DISABLE=0
OMP_NUM_THREADS=20
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
NCCL_IB_RETRY_CNT=7
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_tiger

</details>

🐛 Describe the bug

This serving command

vllm serve ./Qwen3-30B-A3B-Instruct-2507/ \
    --dtype bfloat16 \
    --load_format safetensors \
    --distributed_executor_backend mp \
    --worker_extension_cls 'verl.workers.rollout.vllm_rollout.utils.vLLMColocateWorkerExtension' \
    --max-model-len 262144 \
    --max_num_seqs 1024 \
    --enable_chunked_prefill \
    --max_num_batched_tokens 32768 \
    --enable_prefix_caching \
    --enable_sleep_mode \
    --logprobs_mode processed_logprobs \
    --gpu-memory-utilization 0.6 \
    --tensor-parallel-size 4 \
    --seed 0 \
    --override_generation_config '{"temperature": 1.0, "top_k": -1, "top_p": 1, "repetition_penalty": 1.0, "max_new_tokens": 2048}' \
    --hf_overrides '{}' \
    --scheduling_policy fcfs \
    --compilation_config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
    --enable_lora \
    --max_loras 1 \
    --max_lora_rank 64

raises the the following error:

(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] WorkerProc hit an exception.
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] Traceback (most recent call last):
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/executor/multiproc_executor.py", line 875, in worker_busy_loop
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     output = func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_worker.py", line 390, in determine_available_memory
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.model_runner.profile_run()
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5282, in profile_run
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     hidden_states, last_hidden_states = self._dummy_run(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                                         ^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4976, in _dummy_run
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     outputs = self.model(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]               ^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/cuda_graph.py", line 223, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.runnable(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._call_impl(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return forward_call(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 783, in forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     hidden_states = self.model(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                     ^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/decorators.py", line 563, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     output = self.aot_compiled_fn(self, *args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.fn(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 470, in forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     def forward(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/caching.py", line 198, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.optimized_call(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._wrapped_call(self, *args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     raise e
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._call_impl(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return forward_call(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "<eval_with_key>.99", line 843, in forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     submod_2 = self.submod_2(getitem_3, s72, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_modules_base_layer_parameters_weight_, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_token_lora_mapping, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_token_indices_sorted_by_lora_ids, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_a_stacked_0_, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_num_tokens_per_lora, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_lora_token_start_loc, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_active_lora_ids, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_no_lora_flag_cpu, l_self_modules_embed_tokens_punica_wrapper_token_mapping_meta_default_num_active_loras_cpu, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_b_stacked_0_, l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_, getitem_4, l_self_modules_layers_modules_0_modules_mlp_modules_gate_modules_base_layer_parameters_weight_, l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_a_stacked_0_, l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_b_stacked_0_, l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_modules_base_layer_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_0_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_1_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_2_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_0_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_1_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_2_, l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_3 = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_modules_base_layer_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_a_stacked_0_ = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_lora_b_stacked_0_ = l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_ = getitem_4 = l_self_modules_layers_modules_0_modules_mlp_modules_gate_modules_base_layer_parameters_weight_ = l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_a_stacked_0_ = l_self_modules_layers_modules_0_modules_mlp_modules_gate_lora_b_stacked_0_ = l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_modules_base_layer_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_0_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_1_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_a_stacked_2_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_0_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_1_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_lora_b_stacked_2_ = l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight_ = None
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/cuda_graph.py", line 223, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.runnable(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/piecewise_backend.py", line 343, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return range_entry.runnable(*args)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_inductor/standalone_compile.py", line 122, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._compiled_fn(*args)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/eval_frame.py", line 1181, in _fn
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return fn(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/aot_autograd.py", line 1148, in forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return compiled_fn(full_args)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1962, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.compiled_fn(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 357, in runtime_wrapper
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     all_outs = call_func_at_runtime_with_args(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 134, in call_func_at_runtime_with_args
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     out = normalize_as_list(f(args))
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                             ^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 531, in wrapper
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return compiled_fn(runtime_args)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_inductor/output_code.py", line 638, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.current_callable(inputs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_inductor/utils.py", line 3220, in run
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     out = model(new_inputs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]           ^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/tmp/torchinductor_tiger/fd/cfdmqeuu22zyshstmjjpuszlamqp4lyxl75wi3lr55hvb4cjyqaa.py", line 1146, in call
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     buf16 = torch.ops.vllm.moe_forward.default(buf9, buf13, None, 'from_forward_context')
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_ops.py", line 819, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._op(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 85, in _moe_forward
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return layer.runner.forward_impl(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 689, in forward_impl
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     final_hidden_states = self.quant_method.apply(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                           ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/fused_moe_modular_method.py", line 94, in apply
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.moe_kernel.apply(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/lora/layers/fused_moe.py", line 172, in wrapper
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     result = func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1753, in apply
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.impl.apply(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1537, in apply
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     fused_out = self._fused_experts(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                 ^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1375, in _fused_experts
(Worker pid=263605) (Worker_TP0 pid=263605) (Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] WorkerProc hit an exception.
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880] Traceback (most recent call last):
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/executor/multiproc_executor.py", line 875, in worker_busy_loop
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     output = func(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_worker.py", line 390, in determine_available_memory
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.model_runner.profile_run()
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5282, in profile_run
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     hidden_states, last_hidden_states = self._dummy_run(
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                                         ^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4976, in _dummy_run
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     outputs = self.model(
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]               ^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/cuda_graph.py", line 223, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.runnable(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._call_impl(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return forward_call(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 783, in forward
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     hidden_states = self.model(
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]                     ^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/decorators.py", line 563, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     output = self.aot_compiled_fn(self, *args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.fn(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 470, in forward
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     def forward(
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/compilation/caching.py", line 198, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self.optimized_call(*args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._wrapped_call(self, *args, **kwargs)
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     raise e
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=263606) (Worker_TP1 pid=263606) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._call_impl(*args, **kwargs)
ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.fused_experts.apply(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/fused_moe.py", line 2121, in apply
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.activation(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/lora/layers/fused_moe.py", line 248, in wrapper
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     self.punica_wrapper.add_lora_fused_moe(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/lora/punica_wrapper/punica_gpu.py", line 427, in add_lora_fused_moe
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     fused_moe_lora(
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/_ops.py", line 1209, in __call__
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return self._op(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     return func(*args, **kwargs)
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]   File "/usr/local/lib/python3.11/dist-packages/vllm/lora/ops/triton_ops/fused_moe_lora_op.py", line 725, in _fused_moe_lora
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]     assert output.shape[0] == topk_weights.shape[0]
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=263605) (Worker_TP0 pid=263605) ERROR 03-11 19:12:05 [multiproc_executor.py:880] AssertionError

But after removing --enable_lora --max_loras 1 --max_lora_rank 64, it works.

--enable_lora is required by veRL framework, which would sync lora weights later during the training process.

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

To resolve the issue, you can try the following steps:

Update the fused_moe_lora_op.py file: Modify the fused_moe_lora_op.py file to handle the case where output.shape[0] is not equal to topk_weights.shape[0]. You can add a check to ensure that the shapes match before asserting their equality.
Modify the add_lora_fused_moe method: Update the add_lora_fused_moe method in the punica_gpu.py file to handle the case where the shapes of output and topk_weights do not match.
Check the input shapes: Verify that the input shapes to the fused_moe_lora function are correct and match the expected shapes.

Here's an example of how you can modify the fused_moe_lora_op.py file:

def _fused_moe_lora(...):
    ...
    if output.shape[0] != topk_weights.shape[0]:
        # Handle the case where shapes do not match
        # You can either pad the output or topk_weights to match the shapes
        # or raise a custom error with a meaningful message
        raise ValueError("Output shape does not match topk_weights shape")
    ...

Alternatively, you can also try to update the veRL framework to handle the case where --enable_lora is required but causes issues with the fused_moe_lora function.

Verification

To verify that the fix worked, you can run the serving command with the --enable_lora flag and check if the error is resolved. You can also add additional logging or debugging statements to ensure that the shapes of output and topk_weights are correct.

Extra Tips

Make sure to test the modified code thoroughly to ensure that it works as expected.
If you are using a specific version of the veRL framework, check if there are any known issues or updates that may resolve the problem.
Consider reaching out to the veRL framework community or support team for further assistance if the issue persists.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #model loading #dependency error #configuration error #environment variable

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - ✅(Solved) Fix [Bug]: vLLM 0.17.0 failed to serve Qwen3-30B-A3B-Instruct-2507 after adding `--enable_lora` [1 pull requests, 1 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Fix Action

Fix / Workaround

============================== CPU Info

PR fix notes

PR #37193: [LoRA] Add LoRA support for Qwen3OmniMoeThinkerForConditionalGeneration

Description (problem / solution / changelog)

Purpose

Changes

Why `gate_up_proj` is removed from `packed_modules_mapping`

Test Plan

Test Result

Changed files

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

TRENDING

vllm - ✅(Solved) Fix [Bug]: vLLM 0.17.0 failed to serve Qwen3-30B-A3B-Instruct-2507 after adding `--enable_lora` [1 pull requests, 1 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Fix Action

Fix / Workaround

============================== CPU Info

PR fix notes

PR #37193: [LoRA] Add LoRA support for Qwen3OmniMoeThinkerForConditionalGeneration

Description (problem / solution / changelog)

Purpose

Changes

Why gate_up_proj is removed from packed_modules_mapping

Test Plan

Test Result

Changed files

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING

Why `gate_up_proj` is removed from `packed_modules_mapping`