Issue Station
Sign In
Tools hereToolbox

vllm - 💡(How to fix) Fix [Bug]: VLLM 0.17.1 initial mtp with FLASH_ATTN randomly crash [4 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#36843•Fetched 2026-04-08 00:34:18
View on GitHub
Comments
4
Participants
2
Timeline
29
Reactions
0
Author
flutistflutist
Participants
flutistflutistgithub-actions[bot]github-actions[bot]
Timeline (top)
subscribed ×7mentioned ×6commented ×4labeled ×3

Error Message

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 vllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":5}' -tp 2 (APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] (APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] █ █ █▄ ▄█ (APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.17.1rc1.dev83+g8647c6cf5 (APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] █▄█▀ █ █ █ █ model Qwen/Qwen3.5-27B-GPTQ-Int4 (APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] (APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:233] non-default args: {'model_tag': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'model': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'tensor_parallel_size': 2, 'speculative_config': {'method': 'qwen3_next_mtp', 'num_speculative_tokens': 5}} (APIServer pid=1375776) INFO 03-12 11:36:51 [model.py:533] Resolved architecture: Qwen3_5ForConditionalGeneration (APIServer pid=1375776) INFO 03-12 11:36:51 [model.py:1580] Using max model len 262144 (APIServer pid=1375776) INFO 03-12 11:36:51 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel. (APIServer pid=1375776) WARNING 03-12 11:36:51 [speculative.py:358] method qwen3_next_mtp is deprecated and replaced with mtp. (APIServer pid=1375776) INFO 03-12 11:36:52 [model.py:533] Resolved architecture: Qwen3_5MTP (APIServer pid=1375776) INFO 03-12 11:36:52 [model.py:1580] Using max model len 262144 (APIServer pid=1375776) INFO 03-12 11:36:52 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel. (APIServer pid=1375776) WARNING 03-12 11:36:52 [speculative.py:502] Enabling num_speculative_tokens > 1 will run multiple times of forward on same MTP layer,which may result in lower acceptance rate (APIServer pid=1375776) INFO 03-12 11:36:52 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048. (APIServer pid=1375776) INFO 03-12 11:36:52 [config.py:224] Setting attention block size to 816 tokens to ensure that attention page size is >= mamba page size. (APIServer pid=1375776) INFO 03-12 11:36:52 [config.py:255] Padding mamba page size by 0.99% to ensure that mamba page size and attention page size are exactly equal. Parse safetensors files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 20.72it/s] (APIServer pid=1375776) INFO 03-12 11:36:54 [vllm.py:748] Asynchronous scheduling is enabled. (EngineCore_DP0 pid=1376795) INFO 03-12 11:37:18 [core.py:101] Initializing a V1 LLM engine (v0.17.1rc1.dev83+g8647c6cf5) with config: model='Qwen/Qwen3.5-27B-GPTQ-Int4', speculative_config=SpeculativeConfig(method='mtp', model='Qwen/Qwen3.5-27B-GPTQ-Int4', num_spec_tokens=5), tokenizer='Qwen/Qwen3.5-27B-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.5-27B-GPTQ-Int4, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []} (EngineCore_DP0 pid=1376795) WARNING 03-12 11:37:18 [multiproc_executor.py:997] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. (EngineCore_DP0 pid=1376795) INFO 03-12 11:37:18 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=33.1.35.33 (local), world_size=2, local_world_size=2 (Worker pid=1377089) INFO 03-12 11:37:29 [parallel_state.py:1395] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:49651 backend=nccl (Worker pid=1377333) INFO 03-12 11:37:37 [parallel_state.py:1395] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:49651 backend=nccl (Worker pid=1377333) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. (Worker pid=1377089) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. (Worker pid=1377333) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. (Worker pid=1377089) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. (Worker pid=1377089) INFO 03-12 11:37:38 [pynccl.py:111] vLLM is using nccl==2.27.5 (Worker pid=1377089) WARNING 03-12 11:37:39 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available. (Worker pid=1377333) WARNING 03-12 11:37:39 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available. (Worker pid=1377089) INFO 03-12 11:37:39 [parallel_state.py:1717] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A (Worker pid=1377333) INFO 03-12 11:37:39 [parallel_state.py:1717] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A, EPLB rank N/A (Worker pid=1377089) WARNING 03-12 11:37:39 [init.py:204] min_p and logit_bias parameters won't work with speculative decoding. (Worker pid=1377333) WARNING 03-12 11:37:39 [init.py:204] min_p and logit_bias parameters won't work with speculative decoding. (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:52 [gpu_model_runner.py:4501] Starting to load model Qwen/Qwen3.5-27B-GPTQ-Int4... (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [mm_encoder_attention.py:215] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [gptq_marlin.py:376] Using MarlinLinearKernel for GPTQMarlinLinearMethod (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [cuda.py:317] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [flash_attn.py:593] Using FlashAttention version 2 (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:37:53 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:37:53 [mm_encoder_attention.py:215] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:37:53 [gptq_marlin.py:376] Using MarlinLinearKernel for GPTQMarlinLinearMethod Loading safetensors checkpoint shards: 0% Completed | 0/11 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 9% Completed | 1/11 [00:00<00:04, 2.14it/s] Loading safetensors checkpoint shards: 18% Completed | 2/11 [00:00<00:03, 2.65it/s] Loading safetensors checkpoint shards: 27% Completed | 3/11 [00:01<00:02, 3.10it/s] Loading safetensors checkpoint shards: 36% Completed | 4/11 [00:01<00:02, 3.30it/s] Loading safetensors checkpoint shards: 45% Completed | 5/11 [00:01<00:01, 3.54it/s] Loading safetensors checkpoint shards: 55% Completed | 6/11 [00:01<00:01, 3.68it/s] Loading safetensors checkpoint shards: 64% Completed | 7/11 [00:02<00:01, 3.76it/s] Loading safetensors checkpoint shards: 73% Completed | 8/11 [00:02<00:00, 3.27it/s] Loading safetensors checkpoint shards: 82% Completed | 9/11 [00:03<00:00, 2.52it/s] Loading safetensors checkpoint shards: 91% Completed | 10/11 [00:03<00:00, 1.85it/s] Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:04<00:00, 1.87it/s] Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:04<00:00, 2.48it/s] (Worker pid=1377089) (Worker_TP0 pid=1377089) (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:59 [default_loader.py:293] Loading weights took 4.49 seconds (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:59 [gpu_model_runner.py:4525] Loading drafter model... (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:00 [gpu_model_runner.py:4525] Loading drafter model... Loading safetensors checkpoint shards: 0% Completed | 0/11 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 9% Completed | 1/11 [00:00<00:04, 2.13it/s] Loading safetensors checkpoint shards: 18% Completed | 2/11 [00:00<00:02, 3.33it/s] Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:00<00:00, 14.86it/s] (Worker pid=1377089) (Worker_TP0 pid=1377089) (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:00 [default_loader.py:293] Loading weights took 0.74 seconds (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:03 [eagle.py:1365] Detected MTP model. Sharing target model embedding weights with the draft model. (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:03 [eagle.py:1419] Detected MTP model. Sharing target model lm_head weights with the draft model. (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:03 [eagle.py:1365] Detected MTP model. Sharing target model embedding weights with the draft model. (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:03 [eagle.py:1419] Detected MTP model. Sharing target model lm_head weights with the draft model. (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:04 [gpu_model_runner.py:4584] Model loading took 14.24 GiB memory and 10.609512 seconds (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:04 [gpu_model_runner.py:5506] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:04 [gpu_model_runner.py:5506] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:20 [backends.py:988] Using cache directory: /home/admin/.cache/vllm/torch_compile_cache/defb396318/rank_0_0/backbone for vLLM's torch.compile (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:20 [backends.py:1048] Dynamo bytecode transform time: 10.25 s (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:20 [backends.py:371] Cache the graph of compile range (1, 2048) for later use (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:21 [backends.py:371] Cache the graph of compile range (1, 2048) for later use (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:23 [backends.py:387] Compiling a graph for compile range (1, 2048) takes 2.00 s (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:26 [decorators.py:627] saved AOT compiled function to /home/admin/.cache/vllm/torch_compile_cache/torch_aot_compile/479c124d924359a4af57dcd6c794ae06948c14b8fcf1103c42f2642ee31556f3/rank_0_0/model (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:26 [monitor.py:48] torch.compile took 15.87 s in total (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:26 [monitor.py:76] Initial profiling/warmup run took 0.69 s (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [backends.py:988] Using cache directory: /home/admin/.cache/vllm/torch_compile_cache/defb396318/rank_0_0/eagle_head for vLLM's torch.compile (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [backends.py:1048] Dynamo bytecode transform time: 0.53 s (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [backends.py:387] Compiling a graph for compile range (1, 2048) takes 0.13 s (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [decorators.py:627] saved AOT compiled function to /home/admin/.cache/vllm/torch_compile_cache/torch_aot_compile/3bbb0da6849f142b59b7c396fd019fc0a8041b740d1a0da01f47218bd618289e/rank_0_0/model (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [monitor.py:48] torch.compile took 0.78 s in total (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [monitor.py:76] Initial profiling/warmup run took 0.03 s (Worker pid=1377333) (Worker_TP1 pid=1377333) WARNING 03-12 11:38:28 [kv_cache_utils.py:1054] Add 3 padding layers, may waste at most 6.25% KV cache memory (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:28 [kv_cache_utils.py:826] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512 (Worker pid=1377089) (Worker_TP0 pid=1377089) WARNING 03-12 11:38:28 [kv_cache_utils.py:1054] Add 3 padding layers, may waste at most 6.25% KV cache memory (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:28 [kv_cache_utils.py:826] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512 (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:28 [gpu_model_runner.py:5625] Profiling CUDA graph memory: PIECEWISE=48 (largest=498), FULL=48 (largest=498) (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:28 [gpu_model_runner.py:5625] Profiling CUDA graph memory: PIECEWISE=48 (largest=498), FULL=48 (largest=498) (Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:40 [custom_all_reduce.py:216] Registering 260 cuda graph addresses (Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:40 [custom_all_reduce.py:216] Registering 260 cuda graph addresses (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] WorkerProc hit an exception. (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Traceback (most recent call last): (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 301, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] output = self.runnable(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return self._call_impl(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return forward_call(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 765, in forward (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] hidden_states = self.language_model.model( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 452, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return self.aot_compiled_fn(self, *args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/aot_compile.py", line 124, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return self.fn(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1156, in forward (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] def forward( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/caching.py", line 206, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return self.optimized_call(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return self._wrapped_call(self, *args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 455, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] raise e (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 442, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc] (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return self._call_impl(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return forward_call(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "<eval_with_key>.134", line 1168, in forward (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] submod_1 = self.submod_1(getitem, s59, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return self._wrapped_call(self, *args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 455, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] raise e (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 442, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc] (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return self._call_impl(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return forward_call(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "<eval_with_key>.136", line 5, in forward (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] gdn_attention_core = torch.ops.vllm.gdn_attention_core(mixed_qkv, b_1, a_1, core_attn_out, 'language_model.model.layers.0.linear_attn'); mixed_qkv = b_1 = a_1 = core_attn_out = gdn_attention_core = None (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/_ops.py", line 1209, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return self._op(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1485, in gdn_attention_core (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] self._forward_core( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 704, in _forward_core (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] mixed_qkv_spec = causal_conv1d_update( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/mamba/ops/causal_conv1d.py", line 1196, in causal_conv1d_update (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] _causal_conv1d_update_kernel[grid]( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 370, in <lambda> (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 743, in run (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] launch_metadata = kernel.launch_metadata(grid, stream, *bound_args.values()) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/compiler/compiler.py", line 482, in launch_metadata (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] self._init_handles() (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/compiler/compiler.py", line 465, in _init_handles (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] self.module, self.function, self.n_regs, self.n_spills, self.n_max_threads = driver.active.utils.load_binary( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] RuntimeError: Triton Error [CUDA]: operation not permitted when stream is capturing (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] During handling of the above exception, another exception occurred: (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Traceback (most recent call last): (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 927, in worker_busy_loop (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] output = func(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 397, in determine_available_memory (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5654, in profile_cudagraph_memory (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] self._warmup_and_capture( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5794, in _warmup_and_capture (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] self._dummy_run( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5228, in _dummy_run (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] outputs = self.model( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 295, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] with torch.cuda.graph( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/cuda/graphs.py", line 268, in exit (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] self.cuda_graph.capture_end() (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/cuda/graphs.py", line 130, in capture_end (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] super().capture_end() (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] torch.AcceleratorError: CUDA error: operation failed due to a previous error during capture (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Search for cudaErrorStreamCaptureInvalidated' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] For debugging consider passing CUDA_LAUNCH_BLOCKING=1 (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Compile with TORCH_USE_CUDA_DSA` to enable device-side assertions. (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Traceback (most recent call last): (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 301, in call (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] output = self.runnable(*args, **kwargs) (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]

Code Example

python collect_env.py
Collecting environment information...
==============================
        System Info
==============================
OS                           : Alibaba Cloud Linux 3 (Soaring Falcon) (x86_64)
GCC version                  : (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3.8 2.32)
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.32

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  3 2026, 14:59:34) [Clang 21.1.4 ] (64-bit runtime)
Python platform              : Linux-5.10.134-010.ali5000.al8.x86_64-x86_64-with-glibc2.32

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA L20
GPU 1: NVIDIA L20

Nvidia driver version        : 535.161.08
cuDNN version                : Probably one of the following:
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_adv.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_cnn.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_engines_precompiled.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_engines_runtime_compiled.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_graph.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_heuristic.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_ops.so.9.10.2
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:         x86_64
CPU op-mode(s):       32-bit, 64-bit
Byte Order:           Little Endian
CPU(s):               192
On-line CPU(s) list:  0,1,3-5,8,15-27,66,67,96,97,99-101,104,110-123,162,163,170-190
Off-line CPU(s) list: 2,6,7,9-14,28-65,68-95,98,102,103,105-109,124-161,164-169,191
Thread(s) per core:   0
Core(s) per socket:   48
Socket(s):            2
NUMA node(s):         2
Vendor ID:            GenuineIntel
CPU family:           6
Model:                143
Model name:           Intel(R) Xeon(R) Platinum 8475B
Stepping:             8
CPU MHz:              3200.000
CPU max MHz:          3800.0000
CPU min MHz:          800.0000
BogoMIPS:             5400.00
Virtualization:       VT-x
L1d cache:            48K
L1i cache:            32K
L2 cache:             2048K
L3 cache:             99840K
NUMA node0 CPU(s):    0-47,96-143
NUMA node1 CPU(s):    48-95,144-191
Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm uintr md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.4
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.1
[pip3] nvidia-cutlass-dsl-libs-base==4.4.1
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu129
[pip3] torchvision==0.25.0+cu129
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] numpy                       2.2.6                 pypi_0              pypi
[conda] nvidia-cublas-cu12          12.9.1.4              pypi_0              pypi
[conda] nvidia-cuda-cupti-cu12      12.9.79               pypi_0              pypi
[conda] nvidia-cuda-nvrtc-cu12      12.9.86               pypi_0              pypi
[conda] nvidia-cuda-runtime-cu12    12.9.79               pypi_0              pypi
[conda] nvidia-cudnn-cu12           9.10.2.21             pypi_0              pypi
[conda] nvidia-cufft-cu12           11.4.1.4              pypi_0              pypi
[conda] nvidia-cufile-cu12          1.14.1.1              pypi_0              pypi
[conda] nvidia-curand-cu12          10.3.10.19            pypi_0              pypi
[conda] nvidia-cusolver-cu12        11.7.5.82             pypi_0              pypi
[conda] nvidia-cusparse-cu12        12.5.10.65            pypi_0              pypi
[conda] nvidia-cusparselt-cu12      0.7.1                 pypi_0              pypi
[conda] nvidia-ml-py                13.590.48             pypi_0              pypi
[conda] nvidia-nccl-cu12            2.27.5                pypi_0              pypi
[conda] nvidia-nvjitlink-cu12       12.9.86               pypi_0              pypi
[conda] nvidia-nvshmem-cu12         3.4.5                 pypi_0              pypi
[conda] nvidia-nvtx-cu12            12.9.79               pypi_0              pypi
[conda] pyzmq                       27.1.0                pypi_0              pypi
[conda] torch                       2.10.0+cu129          pypi_0              pypi
[conda] torchaudio                  2.10.0+cu129          pypi_0              pypi
[conda] torchvision                 0.25.0+cu129          pypi_0              pypi
[conda] triton                      3.6.0                 pypi_0              pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.17.1rc1.dev83+g8647c6cf5 (git sha: 8647c6cf5)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     0-1,3-5,8,15-27 0               N/A
GPU1    SYS      X      66-67,162-163   1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64/::/usr/local/cuda/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/usr/local/lib64:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/usr/local/lib:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/usr/lib64:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/usr/lib:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/lib64:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/lib:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64/::/usr/local/cuda/lib64
NVIDIA_VISIBLE_DEVICES=0,1
NVIDIA_GDRCOPY=enabled
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
NVIDIA_VOL_MNT_PATH=/usr/local/nvidia/
NVIDIA_DRIVER_CAPABILITIES=all
TORCH_HOME=/home/admin/workspace/aop_lab/app_data/.cache
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_admin

---

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 vllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":5}'  -tp 2
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] 
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297]        █     █     █▄   ▄█
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.17.1rc1.dev83+g8647c6cf5
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297]   █▄█▀ █     █     █     █  model   Qwen/Qwen3.5-27B-GPTQ-Int4
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] 
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:233] non-default args: {'model_tag': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'model': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'tensor_parallel_size': 2, 'speculative_config': {'method': 'qwen3_next_mtp', 'num_speculative_tokens': 5}}
(APIServer pid=1375776) INFO 03-12 11:36:51 [model.py:533] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=1375776) INFO 03-12 11:36:51 [model.py:1580] Using max model len 262144
(APIServer pid=1375776) INFO 03-12 11:36:51 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
(APIServer pid=1375776) WARNING 03-12 11:36:51 [speculative.py:358] method `qwen3_next_mtp` is deprecated and replaced with mtp.
(APIServer pid=1375776) INFO 03-12 11:36:52 [model.py:533] Resolved architecture: Qwen3_5MTP
(APIServer pid=1375776) INFO 03-12 11:36:52 [model.py:1580] Using max model len 262144
(APIServer pid=1375776) INFO 03-12 11:36:52 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
(APIServer pid=1375776) WARNING 03-12 11:36:52 [speculative.py:502] Enabling num_speculative_tokens > 1 will run multiple times of forward on same MTP layer,which may result in lower acceptance rate
(APIServer pid=1375776) INFO 03-12 11:36:52 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=1375776) INFO 03-12 11:36:52 [config.py:224] Setting attention block size to 816 tokens to ensure that attention page size is >= mamba page size.
(APIServer pid=1375776) INFO 03-12 11:36:52 [config.py:255] Padding mamba page size by 0.99% to ensure that mamba page size and attention page size are exactly equal.
Parse safetensors files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 20.72it/s]
(APIServer pid=1375776) INFO 03-12 11:36:54 [vllm.py:748] Asynchronous scheduling is enabled.
(EngineCore_DP0 pid=1376795) INFO 03-12 11:37:18 [core.py:101] Initializing a V1 LLM engine (v0.17.1rc1.dev83+g8647c6cf5) with config: model='Qwen/Qwen3.5-27B-GPTQ-Int4', speculative_config=SpeculativeConfig(method='mtp', model='Qwen/Qwen3.5-27B-GPTQ-Int4', num_spec_tokens=5), tokenizer='Qwen/Qwen3.5-27B-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.5-27B-GPTQ-Int4, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=1376795) WARNING 03-12 11:37:18 [multiproc_executor.py:997] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore_DP0 pid=1376795) INFO 03-12 11:37:18 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=33.1.35.33 (local), world_size=2, local_world_size=2
(Worker pid=1377089) INFO 03-12 11:37:29 [parallel_state.py:1395] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:49651 backend=nccl
(Worker pid=1377333) INFO 03-12 11:37:37 [parallel_state.py:1395] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:49651 backend=nccl
(Worker pid=1377333) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=1377089) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=1377333) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=1377089) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=1377089) INFO 03-12 11:37:38 [pynccl.py:111] vLLM is using nccl==2.27.5
(Worker pid=1377089) WARNING 03-12 11:37:39 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available.
(Worker pid=1377333) WARNING 03-12 11:37:39 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available.
(Worker pid=1377089) INFO 03-12 11:37:39 [parallel_state.py:1717] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(Worker pid=1377333) INFO 03-12 11:37:39 [parallel_state.py:1717] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A, EPLB rank N/A
(Worker pid=1377089) WARNING 03-12 11:37:39 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=1377333) WARNING 03-12 11:37:39 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:52 [gpu_model_runner.py:4501] Starting to load model Qwen/Qwen3.5-27B-GPTQ-Int4...
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [mm_encoder_attention.py:215] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [gptq_marlin.py:376] Using MarlinLinearKernel for GPTQMarlinLinearMethod
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [cuda.py:317] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [flash_attn.py:593] Using FlashAttention version 2
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:37:53 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:37:53 [mm_encoder_attention.py:215] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:37:53 [gptq_marlin.py:376] Using MarlinLinearKernel for GPTQMarlinLinearMethod
Loading safetensors checkpoint shards:   0% Completed | 0/11 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   9% Completed | 1/11 [00:00<00:04,  2.14it/s]
Loading safetensors checkpoint shards:  18% Completed | 2/11 [00:00<00:03,  2.65it/s]
Loading safetensors checkpoint shards:  27% Completed | 3/11 [00:01<00:02,  3.10it/s]
Loading safetensors checkpoint shards:  36% Completed | 4/11 [00:01<00:02,  3.30it/s]
Loading safetensors checkpoint shards:  45% Completed | 5/11 [00:01<00:01,  3.54it/s]
Loading safetensors checkpoint shards:  55% Completed | 6/11 [00:01<00:01,  3.68it/s]
Loading safetensors checkpoint shards:  64% Completed | 7/11 [00:02<00:01,  3.76it/s]
Loading safetensors checkpoint shards:  73% Completed | 8/11 [00:02<00:00,  3.27it/s]
Loading safetensors checkpoint shards:  82% Completed | 9/11 [00:03<00:00,  2.52it/s]
Loading safetensors checkpoint shards:  91% Completed | 10/11 [00:03<00:00,  1.85it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:04<00:00,  1.87it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:04<00:00,  2.48it/s]
(Worker pid=1377089) (Worker_TP0 pid=1377089) 
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:59 [default_loader.py:293] Loading weights took 4.49 seconds
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:59 [gpu_model_runner.py:4525] Loading drafter model...
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:00 [gpu_model_runner.py:4525] Loading drafter model...
Loading safetensors checkpoint shards:   0% Completed | 0/11 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   9% Completed | 1/11 [00:00<00:04,  2.13it/s]
Loading safetensors checkpoint shards:  18% Completed | 2/11 [00:00<00:02,  3.33it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:00<00:00, 14.86it/s]
(Worker pid=1377089) (Worker_TP0 pid=1377089) 
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:00 [default_loader.py:293] Loading weights took 0.74 seconds
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:03 [eagle.py:1365] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:03 [eagle.py:1419] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:03 [eagle.py:1365] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:03 [eagle.py:1419] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:04 [gpu_model_runner.py:4584] Model loading took 14.24 GiB memory and 10.609512 seconds
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:04 [gpu_model_runner.py:5506] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:04 [gpu_model_runner.py:5506] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:20 [backends.py:988] Using cache directory: /home/admin/.cache/vllm/torch_compile_cache/defb396318/rank_0_0/backbone for vLLM's torch.compile
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:20 [backends.py:1048] Dynamo bytecode transform time: 10.25 s
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:20 [backends.py:371] Cache the graph of compile range (1, 2048) for later use
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:21 [backends.py:371] Cache the graph of compile range (1, 2048) for later use
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:23 [backends.py:387] Compiling a graph for compile range (1, 2048) takes 2.00 s
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:26 [decorators.py:627] saved AOT compiled function to /home/admin/.cache/vllm/torch_compile_cache/torch_aot_compile/479c124d924359a4af57dcd6c794ae06948c14b8fcf1103c42f2642ee31556f3/rank_0_0/model
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:26 [monitor.py:48] torch.compile took 15.87 s in total
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:26 [monitor.py:76] Initial profiling/warmup run took 0.69 s
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [backends.py:988] Using cache directory: /home/admin/.cache/vllm/torch_compile_cache/defb396318/rank_0_0/eagle_head for vLLM's torch.compile
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [backends.py:1048] Dynamo bytecode transform time: 0.53 s
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [backends.py:387] Compiling a graph for compile range (1, 2048) takes 0.13 s
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [decorators.py:627] saved AOT compiled function to /home/admin/.cache/vllm/torch_compile_cache/torch_aot_compile/3bbb0da6849f142b59b7c396fd019fc0a8041b740d1a0da01f47218bd618289e/rank_0_0/model
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [monitor.py:48] torch.compile took 0.78 s in total
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [monitor.py:76] Initial profiling/warmup run took 0.03 s
(Worker pid=1377333) (Worker_TP1 pid=1377333) WARNING 03-12 11:38:28 [kv_cache_utils.py:1054] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:28 [kv_cache_utils.py:826] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker pid=1377089) (Worker_TP0 pid=1377089) WARNING 03-12 11:38:28 [kv_cache_utils.py:1054] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:28 [kv_cache_utils.py:826] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:28 [gpu_model_runner.py:5625] Profiling CUDA graph memory: PIECEWISE=48 (largest=498), FULL=48 (largest=498)
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:28 [gpu_model_runner.py:5625] Profiling CUDA graph memory: PIECEWISE=48 (largest=498), FULL=48 (largest=498)
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:40 [custom_all_reduce.py:216] Registering 260 cuda graph addresses
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:40 [custom_all_reduce.py:216] Registering 260 cuda graph addresses
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] WorkerProc hit an exception.
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Traceback (most recent call last):
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 301, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     output = self.runnable(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._call_impl(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return forward_call(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 765, in forward
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     hidden_states = self.language_model.model(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 452, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self.aot_compiled_fn(self, *args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self.fn(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1156, in forward
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     def forward(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/caching.py", line 206, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self.optimized_call(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._wrapped_call(self, *args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     raise e
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._call_impl(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return forward_call(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "<eval_with_key>.134", line 1168, in forward
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     submod_1 = self.submod_1(getitem, s59, getitem_1, getitem_2, getitem_3);  getitem = getitem_1 = getitem_2 = submod_1 = None
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._wrapped_call(self, *args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     raise e
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._call_impl(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return forward_call(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "<eval_with_key>.136", line 5, in forward
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     gdn_attention_core = torch.ops.vllm.gdn_attention_core(mixed_qkv, b_1, a_1, core_attn_out, 'language_model.model.layers.0.linear_attn');  mixed_qkv = b_1 = a_1 = core_attn_out = gdn_attention_core = None
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/_ops.py", line 1209, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._op(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1485, in gdn_attention_core
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self._forward_core(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 704, in _forward_core
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     mixed_qkv_spec = causal_conv1d_update(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                      ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/mamba/ops/causal_conv1d.py", line 1196, in causal_conv1d_update
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     _causal_conv1d_update_kernel[grid](
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 370, in <lambda>
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 743, in run
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     launch_metadata = kernel.launch_metadata(grid, stream, *bound_args.values())
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/compiler/compiler.py", line 482, in launch_metadata
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self._init_handles()
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/compiler/compiler.py", line 465, in _init_handles
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self.module, self.function, self.n_regs, self.n_spills, self.n_max_threads = driver.active.utils.load_binary(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] RuntimeError: Triton Error [CUDA]: operation not permitted when stream is capturing
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] 
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] During handling of the above exception, another exception occurred:
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] 
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Traceback (most recent call last):
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 927, in worker_busy_loop
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     output = func(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]              ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return func(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 397, in determine_available_memory
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return func(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5654, in profile_cudagraph_memory
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self._warmup_and_capture(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5794, in _warmup_and_capture
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self._dummy_run(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return func(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5228, in _dummy_run
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     outputs = self.model(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]               ^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 295, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     with torch.cuda.graph(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]          ^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/cuda/graphs.py", line 268, in __exit__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self.cuda_graph.capture_end()
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/cuda/graphs.py", line 130, in capture_end
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     super().capture_end()
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] torch.AcceleratorError: CUDA error: operation failed due to a previous error during capture
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Search for `cudaErrorStreamCaptureInvalidated' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] 
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Traceback (most recent call last):
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 301, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     output = self.runnable(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]
RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary> 
python collect_env.py
Collecting environment information...
==============================
        System Info
==============================
OS                           : Alibaba Cloud Linux 3 (Soaring Falcon) (x86_64)
GCC version                  : (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3.8 2.32)
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.32

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  3 2026, 14:59:34) [Clang 21.1.4 ] (64-bit runtime)
Python platform              : Linux-5.10.134-010.ali5000.al8.x86_64-x86_64-with-glibc2.32

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA L20
GPU 1: NVIDIA L20

Nvidia driver version        : 535.161.08
cuDNN version                : Probably one of the following:
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_adv.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_cnn.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_engines_precompiled.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_engines_runtime_compiled.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_graph.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_heuristic.so.9.10.2
/usr/local/cuda-12.9/targets/x86_64-linux/lib/libcudnn_ops.so.9.10.2
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:         x86_64
CPU op-mode(s):       32-bit, 64-bit
Byte Order:           Little Endian
CPU(s):               192
On-line CPU(s) list:  0,1,3-5,8,15-27,66,67,96,97,99-101,104,110-123,162,163,170-190
Off-line CPU(s) list: 2,6,7,9-14,28-65,68-95,98,102,103,105-109,124-161,164-169,191
Thread(s) per core:   0
Core(s) per socket:   48
Socket(s):            2
NUMA node(s):         2
Vendor ID:            GenuineIntel
CPU family:           6
Model:                143
Model name:           Intel(R) Xeon(R) Platinum 8475B
Stepping:             8
CPU MHz:              3200.000
CPU max MHz:          3800.0000
CPU min MHz:          800.0000
BogoMIPS:             5400.00
Virtualization:       VT-x
L1d cache:            48K
L1i cache:            32K
L2 cache:             2048K
L3 cache:             99840K
NUMA node0 CPU(s):    0-47,96-143
NUMA node1 CPU(s):    48-95,144-191
Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm uintr md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.4
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.1
[pip3] nvidia-cutlass-dsl-libs-base==4.4.1
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu129
[pip3] torchvision==0.25.0+cu129
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] numpy                       2.2.6                 pypi_0              pypi
[conda] nvidia-cublas-cu12          12.9.1.4              pypi_0              pypi
[conda] nvidia-cuda-cupti-cu12      12.9.79               pypi_0              pypi
[conda] nvidia-cuda-nvrtc-cu12      12.9.86               pypi_0              pypi
[conda] nvidia-cuda-runtime-cu12    12.9.79               pypi_0              pypi
[conda] nvidia-cudnn-cu12           9.10.2.21             pypi_0              pypi
[conda] nvidia-cufft-cu12           11.4.1.4              pypi_0              pypi
[conda] nvidia-cufile-cu12          1.14.1.1              pypi_0              pypi
[conda] nvidia-curand-cu12          10.3.10.19            pypi_0              pypi
[conda] nvidia-cusolver-cu12        11.7.5.82             pypi_0              pypi
[conda] nvidia-cusparse-cu12        12.5.10.65            pypi_0              pypi
[conda] nvidia-cusparselt-cu12      0.7.1                 pypi_0              pypi
[conda] nvidia-ml-py                13.590.48             pypi_0              pypi
[conda] nvidia-nccl-cu12            2.27.5                pypi_0              pypi
[conda] nvidia-nvjitlink-cu12       12.9.86               pypi_0              pypi
[conda] nvidia-nvshmem-cu12         3.4.5                 pypi_0              pypi
[conda] nvidia-nvtx-cu12            12.9.79               pypi_0              pypi
[conda] pyzmq                       27.1.0                pypi_0              pypi
[conda] torch                       2.10.0+cu129          pypi_0              pypi
[conda] torchaudio                  2.10.0+cu129          pypi_0              pypi
[conda] torchvision                 0.25.0+cu129          pypi_0              pypi
[conda] triton                      3.6.0                 pypi_0              pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.17.1rc1.dev83+g8647c6cf5 (git sha: 8647c6cf5)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     0-1,3-5,8,15-27 0               N/A
GPU1    SYS      X      66-67,162-163   1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64/::/usr/local/cuda/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/usr/local/lib64:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/usr/local/lib:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/usr/lib64:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/usr/lib:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/lib64:/home/admin/hippo/worker/slave/adc_c2_app_online_hippo_os30_7u_inst_35257_adc_c2group_online_hippo_os30_7u_inst_35257.default_S521172_42_51/binary/lib:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64/::/usr/local/cuda/lib64
NVIDIA_VISIBLE_DEVICES=0,1
NVIDIA_GDRCOPY=enabled
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
NVIDIA_VOL_MNT_PATH=/usr/local/nvidia/
NVIDIA_DRIVER_CAPABILITIES=all
TORCH_HOME=/home/admin/workspace/aop_lab/app_data/.cache
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_admin
</details>

🐛 Describe the bug

when execute VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 vllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":5}' -tp 2 , it randomly crash with error msg:(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] self.module, self.function, self.n_regs, self.n_spills, self.n_max_threads = driver.active.utils.load_binary( (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] RuntimeError: Triton Error [CUDA]: operation not permitted when stream is capturing full error msg:

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 vllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":5}'  -tp 2
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] 
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297]        █     █     █▄   ▄█
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.17.1rc1.dev83+g8647c6cf5
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297]   █▄█▀ █     █     █     █  model   Qwen/Qwen3.5-27B-GPTQ-Int4
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:297] 
(APIServer pid=1375776) INFO 03-12 11:36:49 [utils.py:233] non-default args: {'model_tag': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'model': 'Qwen/Qwen3.5-27B-GPTQ-Int4', 'tensor_parallel_size': 2, 'speculative_config': {'method': 'qwen3_next_mtp', 'num_speculative_tokens': 5}}
(APIServer pid=1375776) INFO 03-12 11:36:51 [model.py:533] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=1375776) INFO 03-12 11:36:51 [model.py:1580] Using max model len 262144
(APIServer pid=1375776) INFO 03-12 11:36:51 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
(APIServer pid=1375776) WARNING 03-12 11:36:51 [speculative.py:358] method `qwen3_next_mtp` is deprecated and replaced with mtp.
(APIServer pid=1375776) INFO 03-12 11:36:52 [model.py:533] Resolved architecture: Qwen3_5MTP
(APIServer pid=1375776) INFO 03-12 11:36:52 [model.py:1580] Using max model len 262144
(APIServer pid=1375776) INFO 03-12 11:36:52 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
(APIServer pid=1375776) WARNING 03-12 11:36:52 [speculative.py:502] Enabling num_speculative_tokens > 1 will run multiple times of forward on same MTP layer,which may result in lower acceptance rate
(APIServer pid=1375776) INFO 03-12 11:36:52 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=1375776) INFO 03-12 11:36:52 [config.py:224] Setting attention block size to 816 tokens to ensure that attention page size is >= mamba page size.
(APIServer pid=1375776) INFO 03-12 11:36:52 [config.py:255] Padding mamba page size by 0.99% to ensure that mamba page size and attention page size are exactly equal.
Parse safetensors files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 20.72it/s]
(APIServer pid=1375776) INFO 03-12 11:36:54 [vllm.py:748] Asynchronous scheduling is enabled.
(EngineCore_DP0 pid=1376795) INFO 03-12 11:37:18 [core.py:101] Initializing a V1 LLM engine (v0.17.1rc1.dev83+g8647c6cf5) with config: model='Qwen/Qwen3.5-27B-GPTQ-Int4', speculative_config=SpeculativeConfig(method='mtp', model='Qwen/Qwen3.5-27B-GPTQ-Int4', num_spec_tokens=5), tokenizer='Qwen/Qwen3.5-27B-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.5-27B-GPTQ-Int4, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}
(EngineCore_DP0 pid=1376795) WARNING 03-12 11:37:18 [multiproc_executor.py:997] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore_DP0 pid=1376795) INFO 03-12 11:37:18 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=33.1.35.33 (local), world_size=2, local_world_size=2
(Worker pid=1377089) INFO 03-12 11:37:29 [parallel_state.py:1395] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:49651 backend=nccl
(Worker pid=1377333) INFO 03-12 11:37:37 [parallel_state.py:1395] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:49651 backend=nccl
(Worker pid=1377333) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=1377089) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(Worker pid=1377333) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=1377089) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=1377089) INFO 03-12 11:37:38 [pynccl.py:111] vLLM is using nccl==2.27.5
(Worker pid=1377089) WARNING 03-12 11:37:39 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available.
(Worker pid=1377333) WARNING 03-12 11:37:39 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available.
(Worker pid=1377089) INFO 03-12 11:37:39 [parallel_state.py:1717] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(Worker pid=1377333) INFO 03-12 11:37:39 [parallel_state.py:1717] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A, EPLB rank N/A
(Worker pid=1377089) WARNING 03-12 11:37:39 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=1377333) WARNING 03-12 11:37:39 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:52 [gpu_model_runner.py:4501] Starting to load model Qwen/Qwen3.5-27B-GPTQ-Int4...
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [mm_encoder_attention.py:215] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [gptq_marlin.py:376] Using MarlinLinearKernel for GPTQMarlinLinearMethod
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [cuda.py:317] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:53 [flash_attn.py:593] Using FlashAttention version 2
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:37:53 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:37:53 [mm_encoder_attention.py:215] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:37:53 [gptq_marlin.py:376] Using MarlinLinearKernel for GPTQMarlinLinearMethod
Loading safetensors checkpoint shards:   0% Completed | 0/11 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   9% Completed | 1/11 [00:00<00:04,  2.14it/s]
Loading safetensors checkpoint shards:  18% Completed | 2/11 [00:00<00:03,  2.65it/s]
Loading safetensors checkpoint shards:  27% Completed | 3/11 [00:01<00:02,  3.10it/s]
Loading safetensors checkpoint shards:  36% Completed | 4/11 [00:01<00:02,  3.30it/s]
Loading safetensors checkpoint shards:  45% Completed | 5/11 [00:01<00:01,  3.54it/s]
Loading safetensors checkpoint shards:  55% Completed | 6/11 [00:01<00:01,  3.68it/s]
Loading safetensors checkpoint shards:  64% Completed | 7/11 [00:02<00:01,  3.76it/s]
Loading safetensors checkpoint shards:  73% Completed | 8/11 [00:02<00:00,  3.27it/s]
Loading safetensors checkpoint shards:  82% Completed | 9/11 [00:03<00:00,  2.52it/s]
Loading safetensors checkpoint shards:  91% Completed | 10/11 [00:03<00:00,  1.85it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:04<00:00,  1.87it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:04<00:00,  2.48it/s]
(Worker pid=1377089) (Worker_TP0 pid=1377089) 
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:59 [default_loader.py:293] Loading weights took 4.49 seconds
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:37:59 [gpu_model_runner.py:4525] Loading drafter model...
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:00 [gpu_model_runner.py:4525] Loading drafter model...
Loading safetensors checkpoint shards:   0% Completed | 0/11 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   9% Completed | 1/11 [00:00<00:04,  2.13it/s]
Loading safetensors checkpoint shards:  18% Completed | 2/11 [00:00<00:02,  3.33it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:00<00:00, 14.86it/s]
(Worker pid=1377089) (Worker_TP0 pid=1377089) 
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:00 [default_loader.py:293] Loading weights took 0.74 seconds
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:03 [eagle.py:1365] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:03 [eagle.py:1419] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:03 [eagle.py:1365] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:03 [eagle.py:1419] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:04 [gpu_model_runner.py:4584] Model loading took 14.24 GiB memory and 10.609512 seconds
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:04 [gpu_model_runner.py:5506] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:04 [gpu_model_runner.py:5506] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:20 [backends.py:988] Using cache directory: /home/admin/.cache/vllm/torch_compile_cache/defb396318/rank_0_0/backbone for vLLM's torch.compile
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:20 [backends.py:1048] Dynamo bytecode transform time: 10.25 s
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:20 [backends.py:371] Cache the graph of compile range (1, 2048) for later use
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:21 [backends.py:371] Cache the graph of compile range (1, 2048) for later use
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:23 [backends.py:387] Compiling a graph for compile range (1, 2048) takes 2.00 s
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:26 [decorators.py:627] saved AOT compiled function to /home/admin/.cache/vllm/torch_compile_cache/torch_aot_compile/479c124d924359a4af57dcd6c794ae06948c14b8fcf1103c42f2642ee31556f3/rank_0_0/model
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:26 [monitor.py:48] torch.compile took 15.87 s in total
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:26 [monitor.py:76] Initial profiling/warmup run took 0.69 s
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [backends.py:988] Using cache directory: /home/admin/.cache/vllm/torch_compile_cache/defb396318/rank_0_0/eagle_head for vLLM's torch.compile
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [backends.py:1048] Dynamo bytecode transform time: 0.53 s
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [backends.py:387] Compiling a graph for compile range (1, 2048) takes 0.13 s
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [decorators.py:627] saved AOT compiled function to /home/admin/.cache/vllm/torch_compile_cache/torch_aot_compile/3bbb0da6849f142b59b7c396fd019fc0a8041b740d1a0da01f47218bd618289e/rank_0_0/model
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [monitor.py:48] torch.compile took 0.78 s in total
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:27 [monitor.py:76] Initial profiling/warmup run took 0.03 s
(Worker pid=1377333) (Worker_TP1 pid=1377333) WARNING 03-12 11:38:28 [kv_cache_utils.py:1054] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:28 [kv_cache_utils.py:826] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker pid=1377089) (Worker_TP0 pid=1377089) WARNING 03-12 11:38:28 [kv_cache_utils.py:1054] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:28 [kv_cache_utils.py:826] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:28 [gpu_model_runner.py:5625] Profiling CUDA graph memory: PIECEWISE=48 (largest=498), FULL=48 (largest=498)
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:28 [gpu_model_runner.py:5625] Profiling CUDA graph memory: PIECEWISE=48 (largest=498), FULL=48 (largest=498)
(Worker pid=1377089) (Worker_TP0 pid=1377089) INFO 03-12 11:38:40 [custom_all_reduce.py:216] Registering 260 cuda graph addresses
(Worker pid=1377333) (Worker_TP1 pid=1377333) INFO 03-12 11:38:40 [custom_all_reduce.py:216] Registering 260 cuda graph addresses
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] WorkerProc hit an exception.
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Traceback (most recent call last):
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 301, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     output = self.runnable(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._call_impl(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return forward_call(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 765, in forward
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     hidden_states = self.language_model.model(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 452, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self.aot_compiled_fn(self, *args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self.fn(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1156, in forward
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     def forward(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/caching.py", line 206, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self.optimized_call(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._wrapped_call(self, *args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     raise e
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._call_impl(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return forward_call(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "<eval_with_key>.134", line 1168, in forward
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     submod_1 = self.submod_1(getitem, s59, getitem_1, getitem_2, getitem_3);  getitem = getitem_1 = getitem_2 = submod_1 = None
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._wrapped_call(self, *args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     raise e
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._call_impl(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return forward_call(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "<eval_with_key>.136", line 5, in forward
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     gdn_attention_core = torch.ops.vllm.gdn_attention_core(mixed_qkv, b_1, a_1, core_attn_out, 'language_model.model.layers.0.linear_attn');  mixed_qkv = b_1 = a_1 = core_attn_out = gdn_attention_core = None
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/_ops.py", line 1209, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return self._op(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1485, in gdn_attention_core
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self._forward_core(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 704, in _forward_core
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     mixed_qkv_spec = causal_conv1d_update(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                      ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/mamba/ops/causal_conv1d.py", line 1196, in causal_conv1d_update
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     _causal_conv1d_update_kernel[grid](
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 370, in <lambda>
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 743, in run
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     launch_metadata = kernel.launch_metadata(grid, stream, *bound_args.values())
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/compiler/compiler.py", line 482, in launch_metadata
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self._init_handles()
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/triton/compiler/compiler.py", line 465, in _init_handles
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self.module, self.function, self.n_regs, self.n_spills, self.n_max_threads = driver.active.utils.load_binary(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] RuntimeError: Triton Error [CUDA]: operation not permitted when stream is capturing
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] 
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] During handling of the above exception, another exception occurred:
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] 
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Traceback (most recent call last):
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 927, in worker_busy_loop
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     output = func(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]              ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return func(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 397, in determine_available_memory
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return func(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5654, in profile_cudagraph_memory
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self._warmup_and_capture(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5794, in _warmup_and_capture
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self._dummy_run(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     return func(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5228, in _dummy_run
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     outputs = self.model(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]               ^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 295, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     with torch.cuda.graph(
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]          ^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/cuda/graphs.py", line 268, in __exit__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     self.cuda_graph.capture_end()
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/cuda/graphs.py", line 130, in capture_end
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     super().capture_end()
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] torch.AcceleratorError: CUDA error: operation failed due to a previous error during capture
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Search for `cudaErrorStreamCaptureInvalidated' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] 
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932] Traceback (most recent call last):
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 301, in __call__
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]     output = self.runnable(*args, **kwargs)
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]   File "/home/admin/workspace/aop_lab/app_source/test_vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker pid=1377089) (Worker_TP0 pid=1377089) ERROR 03-12 11:38:40 [multiproc_executor.py:932]

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

The error message indicates a CUDA error due to a previous error during capture. This issue seems related to the use of CUDA graphs and the torch.compile functionality in PyTorch.

To fix this issue, you can try the following steps:

  1. Disable CUDA Graphs: Try setting the environment variable VLLM_DISABLE_CUDAGRAPH=1 before running your command. This will disable the use of CUDA graphs, which might help resolve the issue.
  2. Update PyTorch and Triton: Ensure that you are using the latest versions of PyTorch and Triton. You can update them using pip: pip install --upgrade torch triton.
  3. Set CUDA_LAUNCH_BLOCKING=1: As suggested in the error message, set the environment variable CUDA_LAUNCH_BLOCKING=1 before running your command. This will help with debugging by making CUDA kernel launches blocking.
  4. Compile with TORCH_USE_CUDA_DSA: If you are compiling PyTorch from source, try enabling TORCH_USE_CUDA_DSA to enable device-side assertions, which can help with debugging.

Example code to disable CUDA graphs:

import os

# Disable CUDA graphs
os.environ['VLLM_DISABLE_CUDAGRAPH'] = '1'

# Your code here

Example command to set environment variables:

VLLM_DISABLE_CUDAGRAPH=1 CUDA_LAUNCH_BLOCKING=1 vllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":5}'  -tp 2

Verification

After applying these fixes, verify that the issue is resolved by running your command again. If the issue persists, try to isolate the problem by running a minimal example that reproduces the error.

Extra Tips

  • Make sure to check the documentation for any specific requirements or recommendations for your setup.
  • If you are using a custom PyTorch build, try switching to a pre-built version to rule out any compilation issues.
  • Consider seeking help from the PyTorch or Triton communities if the issue persists after trying these fixes.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#response parsing#generation error#model loading#environment variable

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING