vllm - ✅(Solved) Fix [Bug]: Buffer overflow when allocating memory error on Qwen3.5-122B-A10B-GPTQ-Int4 and NVFP4 [1 pull requests]

vllm2026-04-20 16:22:47

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

Error Message

sync && echo 3 | tee /proc/sys/vm/drop_caches &&
vllm bench throughput
--model=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4
--trust-remote-code --load-format=dummy
--num-prompts=32 --output-len=256 --input-len=256
--quantization=gptq_marlin --kv-cache-dtype=auto
--gpu-memory-utilization=0.85 --max-model-len=2048
--max-num-batched-tokens=2048 --max-num-seqs=512
--attention-backend=flashinfer
--tensor-parallel-size=1 2>&1 | tee /tmp/qwen35-122B-gptq-flashinfer-spark.log 3 /usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py:848: UserWarning: Both --input-len and --random-input-len are specified. The random version (--random-input-len) will be preferred in this run. validate_args(args) /usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py:848: UserWarning: Both --output-len and --random-output-len are specified. The random version (--random-output-len) will be preferred in this run. validate_args(args) /usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py:848: UserWarning: Both --prefix-len and --random-prefix-len are specified. The random version (--random-prefix-len) will be preferred in this run. validate_args(args) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. When dataset path is not set, it will default to random dataset INFO 04-20 16:08:32 [datasets.py:700] Sampling input_len from [1024, 1024] and output_len from [128, 128] INFO 04-20 16:08:32 [utils.py:233] non-default args: {'tokenizer': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', 'trust_remote_code': True, 'load_format': 'dummy', 'max_model_len': 2048, 'gpu_memory_utilization': 0.85, 'max_num_batched_tokens': 2048, 'max_num_seqs': 512, 'quantization': 'gptq_marlin', 'enable_lora': None, 'reasoning_parser_plugin': '', 'attention_backend': 'flashinfer', 'model': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4'} INFO 04-20 16:08:40 [model.py:549] Resolved architecture: Qwen3_5MoeForConditionalGeneration INFO 04-20 16:08:40 [model.py:1678] Using max model len 2048 INFO 04-20 16:08:40 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel. /usr/local/lib/python3.12/dist-packages/torch/cuda/init.py:435: UserWarning: Found GPU0 NVIDIA GB10 which is of cuda capability 12.1. Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)

queued_call() Qwen2VLImageProcessorFast is deprecated. The Fast suffix for image processors has been removed; use Qwen2VLImageProcessor instead. INFO 04-20 16:08:40 [config.py:281] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size. INFO 04-20 16:08:40 [config.py:312] Padding mamba page size by 0.58% to ensure that mamba page size and attention page size are exactly equal. Parse safetensors files: 100%|██████████| 39/39 [00:02<00:00, 13.60it/s] INFO 04-20 16:08:45 [vllm.py:790] Asynchronous scheduling is enabled. The use_fast parameter is deprecated and will be removed in a future version. Use backend="torchvision" instead of use_fast=True, or backend="pil" instead of use_fast=False. (EngineCore pid=269) INFO 04-20 16:09:02 [core.py:105] Initializing a V1 LLM engine (v0.19.1) with config: model='Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', speculative_config=None, tokenizer='Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/torch/cuda/init.py:435: UserWarning: (EngineCore pid=269) Found GPU0 NVIDIA GB10 which is of cuda capability 12.1. (EngineCore pid=269) Minimum and Maximum cuda capability supported by this version of PyTorch is (EngineCore pid=269) (8.0) - (12.0) (EngineCore pid=269) (EngineCore pid=269) queued_call() (EngineCore pid=269) Qwen2VLImageProcessorFast is deprecated. The Fast suffix for image processors has been removed; use Qwen2VLImageProcessor instead. (EngineCore pid=269) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. (EngineCore pid=269) INFO 04-20 16:09:05 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.57.195.222:55441 backend=nccl (EngineCore pid=269) INFO 04-20 16:09:05 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A (EngineCore pid=269) The use_fast parameter is deprecated and will be removed in a future version. Use backend="torchvision" instead of use_fast=True, or backend="pil" instead of use_fast=False. (EngineCore pid=269) INFO 04-20 16:09:16 [gpu_model_runner.py:4735] Starting to load model Qwen/Qwen3.5-122B-A10B-GPTQ-Int4... (EngineCore pid=269) INFO 04-20 16:09:16 [cuda.py:390] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention (EngineCore pid=269) INFO 04-20 16:09:16 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (EngineCore pid=269) INFO 04-20 16:09:17 [gdn_linear_attn.py:147] Using Triton/FLA GDN prefill kernel (EngineCore pid=269) INFO 04-20 16:09:18 [cuda.py:274] Using AttentionBackendEnum.FLASHINFER backend. (EngineCore pid=269) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. (EngineCore pid=269) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. (EngineCore pid=269) INFO 04-20 16:09:25 [gpu_model_runner.py:4820] Model loading took 68.36 GiB memory and 8.165419 seconds (EngineCore pid=269) INFO 04-20 16:09:25 [gpu_model_runner.py:5753] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. (EngineCore pid=269) INFO 04-20 16:09:36 [backends.py:1051] Using cache directory: /root/.cache/vllm/torch_compile_cache/14431a8247/rank_0_0/backbone for vLLM's torch.compile (EngineCore pid=269) INFO 04-20 16:09:36 [backends.py:1111] Dynamo bytecode transform time: 5.26 s (EngineCore pid=269) [rank0]:W0420 16:09:41.870000 269 torch/_inductor/utils.py:1679] Not enough SMs to use max_autotune_gemm mode (EngineCore pid=269) INFO 04-20 16:09:43 [backends.py:372] Cache the graph of compile range (1, 2048) for later use (EngineCore pid=269) INFO 04-20 16:10:04 [backends.py:390] Compiling a graph for compile range (1, 2048) takes 27.57 s (EngineCore pid=269) INFO 04-20 16:10:05 [decorators.py:655] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/a3057a8f26052af55dccb86f4e1004a7008bf8d77be2a6a5d83ce0a4f64010ac/rank_0_0/model (EngineCore pid=269) INFO 04-20 16:10:05 [monitor.py:48] torch.compile took 33.86 s in total (EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...]. (EngineCore pid=269) return fn(*contiguous_args, **contiguous_kwargs) (EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (32) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...]. (EngineCore pid=269) return fn(*contiguous_args, **contiguous_kwargs) (EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...]. (EngineCore pid=269) return fn(*contiguous_args, **contiguous_kwargs) (EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (32) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...]. (EngineCore pid=269) return fn(*contiguous_args, **contiguous_kwargs) (EngineCore pid=269) INFO 04-20 16:11:14 [monitor.py:76] Initial profiling/warmup run took 69.19 s (EngineCore pid=269) INFO 04-20 16:11:19 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512 (EngineCore pid=269) INFO 04-20 16:11:20 [gpu_model_runner.py:5876] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] EngineCore failed to start. (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] Traceback (most recent call last): (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in init (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] super().init( (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 124, in init (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] kv_cache_config = self._initialize_kv_caches(vllm_config) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 247, in _initialize_kv_caches (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] available_gpu_memory = self.model_executor.determine_available_memory() (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 136, in determine_available_memory (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] return self.collective_rpc("determine_available_memory") (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] result = run_method(self.driver_worker, method, args, kwargs) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 510, in run_method (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5905, in profile_cudagraph_memory (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] self._warmup_and_capture( (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6066, in _warmup_and_capture (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] self._dummy_run( (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5394, in _dummy_run (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] attn_metadata, _ = self._build_attention_metadata( (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2319, in _build_attention_metadata (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] _build_attn_group_metadata(kv_cache_gid, attn_gid, cm) (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2270, in _build_attn_group_metadata (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] attn_metadata_i = builder.build( (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1158, in build (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] fast_plan_decode( (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1719, in fast_plan_decode (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] self.plan( (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/flashinfer/decode.py", line 1093, in plan (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] self._plan_info = self._cached_module.plan( (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] File "python/tvm_ffi/cython/function.pxi", line 929, in tvm_ffi.core.Function.call (EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size. (EngineCore pid=269) Process EngineCore: (EngineCore pid=269) Traceback (most recent call last): (EngineCore pid=269) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap (EngineCore pid=269) self.run() (EngineCore pid=269) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run (EngineCore pid=269) self._target(*self._args, **self._kwargs) (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1112, in run_engine_core (EngineCore pid=269) raise e (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core (EngineCore pid=269) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=269) return func(*args, **kwargs) (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in init (EngineCore pid=269) super().init( (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 124, in init (EngineCore pid=269) kv_cache_config = self._initialize_kv_caches(vllm_config) (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=269) return func(*args, **kwargs) (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 247, in _initialize_kv_caches (EngineCore pid=269) available_gpu_memory = self.model_executor.determine_available_memory() (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 136, in determine_available_memory (EngineCore pid=269) return self.collective_rpc("determine_available_memory") (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc (EngineCore pid=269) result = run_method(self.driver_worker, method, args, kwargs) (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 510, in run_method (EngineCore pid=269) return func(*args, **kwargs) (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=269) return func(*args, **kwargs) (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory (EngineCore pid=269) cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=269) return func(*args, **kwargs) (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5905, in profile_cudagraph_memory (EngineCore pid=269) self._warmup_and_capture( (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6066, in _warmup_and_capture (EngineCore pid=269) self._dummy_run( (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=269) return func(*args, **kwargs) (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5394, in _dummy_run (EngineCore pid=269) attn_metadata, _ = self._build_attention_metadata( (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2319, in _build_attention_metadata (EngineCore pid=269) _build_attn_group_metadata(kv_cache_gid, attn_gid, cm) (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2270, in _build_attn_group_metadata (EngineCore pid=269) attn_metadata_i = builder.build( (EngineCore pid=269) ^^^^^^^^^^^^^^ (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1158, in build (EngineCore pid=269) fast_plan_decode( (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1719, in fast_plan_decode (EngineCore pid=269) self.plan( (EngineCore pid=269) File "/usr/local/lib/python3.12/dist-packages/flashinfer/decode.py", line 1093, in plan (EngineCore pid=269) self._plan_info = self._cached_module.plan( (EngineCore pid=269) ^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=269) File "python/tvm_ffi/cython/function.pxi", line 929, in tvm_ffi.core.Function.call (EngineCore pid=269) RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size. [rank0]:[W420 16:11:22.693089162 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) Traceback (most recent call last): File "/usr/local/bin/vllm", line 10, in <module> sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main args.dispatch_function(args) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/benchmark/throughput.py", line 21, in cmd main(args) File "/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py", line 879, in main elapsed_time, request_outputs = run_vllm( ^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py", line 55, in run_vllm llm = LLM.from_engine_args(engine_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py", line 415, in from_engine_args return cls(**vars(engine_args)) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py", line 382, in init self.llm_engine = LLMEngine.from_engine_args( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 177, in from_engine_args return cls( ^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 111, in init self.engine_core = EngineCoreClient.make_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 101, in make_client return SyncMPClient(vllm_config, executor_class, log_stats) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 710, in init super().init( File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 535, in init with launch_core_engines( ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in exit next(self.gen) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 998, in launch_core_engines wait_for_engine_startup( File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1057, in wait_for_engine_startup raise RuntimeError( RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Vendor ID: ARM BIOS Vendor ID: NVIDIA BIOS Model name: GB10 Model: 1 Thread(s) per core: 1 Core(s) per socket: 5 Socket(s): 1 Stepping: r0p1 CPU max MHz: 2808.0000 CPU min MHz: 338.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt BIOS Model name: GB10 Model: 1 Thread(s) per core: 1 Core(s) per socket: 5 Socket(s): 1 Stepping: r0p1 CPU max MHz: 3900.0000 CPU min MHz: 1378.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt BIOS Model name: GB10 Model: 1 Thread(s) per core: 1 Core(s) per socket: 5 Socket(s): 1 Stepping: r0p1 CPU max MHz: 2860.0000 CPU min MHz: 338.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt BIOS Model name: GB10 Model: 1 Thread(s) per core: 1 Core(s) per socket: 5 Socket(s): 1 Stepping: r0p1 CPU max MHz: 4004.0000 CPU min MHz: 1378.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt L1d cache: 1.3 MiB (20 instances) L1i cache: 1.3 MiB (20 instances) L2 cache: 25 MiB (20 instances) L3 cache: 24 MiB (2 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-19 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Not affected Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; CSV2, but not BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected

PR fix notes

PR #40383: [BugFix]Auto-size FlashInfer workspace buffer to prevent buffer overflows on model

Repository: vllm-project/vllm
Author: ECMGit
State: open | merged: False
Link: https://github.com/vllm-project/vllm/pull/40383

Description (problem / solution / changelog)

Purpose

Fixes https://github.com/vllm-project/vllm/issues/40381

When running large MoE models (e.g. Qwen3.5-122B-A10B) with FlashInfer native attention at TP=1, the default 394 MiB is hardcoded in env.py .VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE, and it causing a RuntimeError: Buffer overflow when allocating memory for batch_prefill_tmp_v during CUDA graph capture.

This PR dynamically estimates the required FlashInfer workspace buffer size based on the GPU's SM count, head dimensions, and KV head count — matching FlashInfer's internal allocation formula (batch_prefill_tmp_v + batch_prefill_tmp_s). The buffer is auto-sized only when the estimate exceeds the default; otherwise the default is used.

Additionally:

Logs an INFO message when the buffer is auto-sized, aiding memory debugging on large models.
If the user explicitly sets VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE, the user-set value is honored (not overridden), with a WARNING if the estimate exceeds it.

Test Plan

Reproduce the crash with any model that has many QO heads (e.g. 64) at TP=1 using FlashInfer native attention (not TRTLLM):

# Crashes without this fix (buffer overflow during CUDA graph capture)
vllm bench throughput \
  --model=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
  --trust-remote-code --load-format=dummy \
  --num-prompts=32 --output-len=256 --input-len=256 \
  --quantization=gptq_marlin --kv-cache-dtype=auto \
  --gpu-memory-utilization=0.85 --max-model-len=2048 \
  --max-num-batched-tokens=2048 --max-num-seqs=512 \
  --attention-backend=flashinfer \
  --attention-config '{"use_trtllm_attention": false}' \
  --tensor-parallel-size=1

On SM 100+ (B200/GB200), --attention-config '{"use_trtllm_attention": false}' is needed to disable TRTLLM auto-detection and force FlashInfer native path. On SM 90 (H100/H200), TRTLLM is not available so the crash happens without the flag.

Test Result

Before (v0.19.1, default 394 MiB buffer):

RuntimeError: Error in function 'aligned_alloc' at
/flashinfer/include/flashinfer/allocator.h:49:
Buffer overflow when allocating memory for batch_prefill_tmp_v
with size 536346624 and alignment 16, but only 413138944 bytes
available in AlignedAllocator.

Reproduced on: H100 (TP=1), DGX Spark (SM 120), AGX Thor (SM 120), B200 (TRTLLM disabled).

After (auto-sized buffer):

INFO: Auto-sized FlashInfer workspace buffer from 394 MiB to 512 MiB
(num_qo_heads=64, num_kv_heads=4, head_dim=128, num_sm=132).

Benchmark completes successfully.

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Changed files

vllm/v1/attention/backends/flashinfer.py (modified, +51/-0)

Code Example

Collecting environment information...
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:435: UserWarning:
    Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (8.0) - (12.0)

  queued_call()
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (aarch64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.14.0-1014-nvidia-aarch64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   :
GPU models and configuration : GPU 0: NVIDIA GB10
Nvidia driver version        : 580.95.05
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  20
On-line CPU(s) list:                     0-19
Vendor ID:                               ARM
BIOS Vendor ID:                          NVIDIA
BIOS Model name:                         GB10
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      5
Socket(s):                               1
Stepping:                                r0p1
CPU max MHz:                             2808.0000
CPU min MHz:                             338.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
BIOS Model name:                         GB10
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      5
Socket(s):                               1
Stepping:                                r0p1
CPU max MHz:                             3900.0000
CPU min MHz:                             1378.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
BIOS Model name:                         GB10
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      5
Socket(s):                               1
Stepping:                                r0p1
CPU max MHz:                             2860.0000
CPU min MHz:                             338.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
BIOS Model name:                         GB10
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      5
Socket(s):                               1
Stepping:                                r0p1
CPU max MHz:                             4004.0000
CPU min MHz:                             1378.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
L1d cache:                               1.3 MiB (20 instances)
L1i cache:                               1.3 MiB (20 instances)
L2 cache:                                25 MiB (20 instances)
L3 cache:                                24 MiB (2 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-19
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Not affected
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, but not BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.6
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu129
[pip3] torchvision==0.25.0+cu129
[pip3] transformers==5.5.4
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.19.1
vLLM Build Flags:
  CUDA Archs: 8.7 8.9 9.0 10.0+PTX 12.0; ROCm: Disabled
GPU Topology:
        GPU0    NIC0    NIC1    NIC2    NIC3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    NODE    NODE    NODE    0-19    0               N/A
NIC0    NODE     X      PIX     NODE    NODE
NIC1    NODE    PIX      X      NODE    NODE
NIC2    NODE    NODE    NODE     X      PIX
NIC3    NODE    NODE    NODE    PIX      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: rocep1s0f0
  NIC1: rocep1s0f1
  NIC2: roceP2p1s0f0
  NIC3: roceP2p1s0f1

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=void
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
TORCH_CUDA_ARCH_LIST=8.7 8.9 9.0 10.0+PTX 12.0
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.9.1
VLLM_ENABLE_CUDA_COMPATIBILITY=0
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64
NVIDIA_CTK_LIBCUDA_DIR=/usr/lib/aarch64-linux-gnu
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

---

docker run \
      --gpus all --runtime=nvidia --privileged \
      -it --rm -u 0:0 \
      --shm-size=256g \
      --ulimit memlock=-1 --ulimit stack=67108864 \
      --ipc=host --network=host \
      -v /tmp/.cache/huggingface:/root/.cache/huggingface \
      --entrypoint /bin/bash \
      vllm/vllm-openai:latest

---

sync && echo 3 | tee /proc/sys/vm/drop_caches && \
vllm bench throughput \
  --model=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
  --trust-remote-code --load-format=dummy \
  --num-prompts=32 --output-len=256 --input-len=256 \
  --quantization=gptq_marlin --kv-cache-dtype=auto \
  --gpu-memory-utilization=0.85 --max-model-len=2048 \
  --max-num-batched-tokens=2048 --max-num-seqs=512 \
  --attention-backend=flashinfer \
  --tensor-parallel-size=1

---

sync && echo 3 | tee /proc/sys/vm/drop_caches && \
  vllm bench throughput \
    --model=RedHatAI/Qwen3.5-122B-A10B-NVFP4 \
    --tokenizer=Qwen/Qwen3.5-122B-A10B \
    --trust-remote-code --load-format=dummy \
    --num-prompts=32 --output-len=256 --input-len=256 \
    --kv-cache-dtype=auto \
    --gpu-memory-utilization=0.85 --max-model-len=2048 \
    --attention-backend=flashinfer \
    --tensor-parallel-size=1

---

sync && echo 3 | tee /proc/sys/vm/drop_caches && \
  vllm bench throughput \
    --model=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
    --trust-remote-code --load-format=dummy \
    --num-prompts=32 --output-len=256 --input-len=256 \
    --quantization=gptq_marlin --kv-cache-dtype=auto \
    --gpu-memory-utilization=0.85 --max-model-len=2048 \
    --max-num-batched-tokens=2048 --max-num-seqs=512 \
    --attention-backend=flashinfer \
    --tensor-parallel-size=1 2>&1 | tee /tmp/qwen35-122B-gptq-flashinfer-spark.log
3
/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py:848: UserWarning: Both --input-len and --random-input-len are specified. The random version (--random-input-len) will be preferred in this run.
  validate_args(args)
/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py:848: UserWarning: Both --output-len and --random-output-len are specified. The random version (--random-output-len) will be preferred in this run.
  validate_args(args)
/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py:848: UserWarning: Both --prefix-len and --random-prefix-len are specified. The random version (--random-prefix-len) will be preferred in this run.
  validate_args(args)
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
When dataset path is not set, it will default to random dataset
INFO 04-20 16:08:32 [datasets.py:700] Sampling input_len from [1024, 1024] and output_len from [128, 128]
INFO 04-20 16:08:32 [utils.py:233] non-default args: {'tokenizer': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', 'trust_remote_code': True, 'load_format': 'dummy', 'max_model_len': 2048, 'gpu_memory_utilization': 0.85, 'max_num_batched_tokens': 2048, 'max_num_seqs': 512, 'quantization': 'gptq_marlin', 'enable_lora': None, 'reasoning_parser_plugin': '', 'attention_backend': 'flashinfer', 'model': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4'}
INFO 04-20 16:08:40 [model.py:549] Resolved architecture: Qwen3_5MoeForConditionalGeneration
INFO 04-20 16:08:40 [model.py:1678] Using max model len 2048
INFO 04-20 16:08:40 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:435: UserWarning:
    Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (8.0) - (12.0)

  queued_call()
`Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
INFO 04-20 16:08:40 [config.py:281] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size.
INFO 04-20 16:08:40 [config.py:312] Padding mamba page size by 0.58% to ensure that mamba page size and attention page size are exactly equal.
Parse safetensors files: 100%|██████████| 39/39 [00:02<00:00, 13.60it/s]
INFO 04-20 16:08:45 [vllm.py:790] Asynchronous scheduling is enabled.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=269) INFO 04-20 16:09:02 [core.py:105] Initializing a V1 LLM engine (v0.19.1) with config: model='Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', speculative_config=None, tokenizer='Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:435: UserWarning:
(EngineCore pid=269)     Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
(EngineCore pid=269)     Minimum and Maximum cuda capability supported by this version of PyTorch is
(EngineCore pid=269)     (8.0) - (12.0)
(EngineCore pid=269)
(EngineCore pid=269)   queued_call()
(EngineCore pid=269) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=269) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(EngineCore pid=269) INFO 04-20 16:09:05 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.57.195.222:55441 backend=nccl
(EngineCore pid=269) INFO 04-20 16:09:05 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=269) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=269) INFO 04-20 16:09:16 [gpu_model_runner.py:4735] Starting to load model Qwen/Qwen3.5-122B-A10B-GPTQ-Int4...
(EngineCore pid=269) INFO 04-20 16:09:16 [cuda.py:390] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=269) INFO 04-20 16:09:16 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=269) INFO 04-20 16:09:17 [gdn_linear_attn.py:147] Using Triton/FLA GDN prefill kernel
(EngineCore pid=269) INFO 04-20 16:09:18 [cuda.py:274] Using AttentionBackendEnum.FLASHINFER backend.
(EngineCore pid=269) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore pid=269) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore pid=269) INFO 04-20 16:09:25 [gpu_model_runner.py:4820] Model loading took 68.36 GiB memory and 8.165419 seconds
(EngineCore pid=269) INFO 04-20 16:09:25 [gpu_model_runner.py:5753] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=269) INFO 04-20 16:09:36 [backends.py:1051] Using cache directory: /root/.cache/vllm/torch_compile_cache/14431a8247/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=269) INFO 04-20 16:09:36 [backends.py:1111] Dynamo bytecode transform time: 5.26 s
(EngineCore pid=269) [rank0]:W0420 16:09:41.870000 269 torch/_inductor/utils.py:1679] Not enough SMs to use max_autotune_gemm mode
(EngineCore pid=269) INFO 04-20 16:09:43 [backends.py:372] Cache the graph of compile range (1, 2048) for later use
(EngineCore pid=269) INFO 04-20 16:10:04 [backends.py:390] Compiling a graph for compile range (1, 2048) takes 27.57 s
(EngineCore pid=269) INFO 04-20 16:10:05 [decorators.py:655] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/a3057a8f26052af55dccb86f4e1004a7008bf8d77be2a6a5d83ce0a4f64010ac/rank_0_0/model
(EngineCore pid=269) INFO 04-20 16:10:05 [monitor.py:48] torch.compile took 33.86 s in total
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(EngineCore pid=269)   return fn(*contiguous_args, **contiguous_kwargs)
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (32) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(EngineCore pid=269)   return fn(*contiguous_args, **contiguous_kwargs)
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(EngineCore pid=269)   return fn(*contiguous_args, **contiguous_kwargs)
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (32) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(EngineCore pid=269)   return fn(*contiguous_args, **contiguous_kwargs)
(EngineCore pid=269) INFO 04-20 16:11:14 [monitor.py:76] Initial profiling/warmup run took 69.19 s
(EngineCore pid=269) INFO 04-20 16:11:19 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(EngineCore pid=269) INFO 04-20 16:11:20 [gpu_model_runner.py:5876] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] EngineCore failed to start.
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] Traceback (most recent call last):
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in __init__
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     super().__init__(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 124, in __init__
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 247, in _initialize_kv_caches
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5905, in profile_cudagraph_memory
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     self._warmup_and_capture(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6066, in _warmup_and_capture
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     self._dummy_run(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5394, in _dummy_run
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     attn_metadata, _ = self._build_attention_metadata(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2319, in _build_attention_metadata
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     _build_attn_group_metadata(kv_cache_gid, attn_gid, cm)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2270, in _build_attn_group_metadata
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     attn_metadata_i = builder.build(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                       ^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1158, in build
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     fast_plan_decode(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1719, in fast_plan_decode
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     self.plan(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/decode.py", line 1093, in plan
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     self._plan_info = self._cached_module.plan(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                       ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "python/tvm_ffi/cython/function.pxi", line 929, in tvm_ffi.core.Function.__call__
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.
(EngineCore pid=269) Process EngineCore:
(EngineCore pid=269) Traceback (most recent call last):
(EngineCore pid=269)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=269)     self.run()
(EngineCore pid=269)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=269)     self._target(*self._args, **self._kwargs)
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1112, in run_engine_core
(EngineCore pid=269)     raise e
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core
(EngineCore pid=269)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=269)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in __init__
(EngineCore pid=269)     super().__init__(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 124, in __init__
(EngineCore pid=269)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=269)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 247, in _initialize_kv_caches
(EngineCore pid=269)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=269)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=269)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=269)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=269)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory
(EngineCore pid=269)     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=269)                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5905, in profile_cudagraph_memory
(EngineCore pid=269)     self._warmup_and_capture(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6066, in _warmup_and_capture
(EngineCore pid=269)     self._dummy_run(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5394, in _dummy_run
(EngineCore pid=269)     attn_metadata, _ = self._build_attention_metadata(
(EngineCore pid=269)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2319, in _build_attention_metadata
(EngineCore pid=269)     _build_attn_group_metadata(kv_cache_gid, attn_gid, cm)
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2270, in _build_attn_group_metadata
(EngineCore pid=269)     attn_metadata_i = builder.build(
(EngineCore pid=269)                       ^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1158, in build
(EngineCore pid=269)     fast_plan_decode(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1719, in fast_plan_decode
(EngineCore pid=269)     self.plan(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/flashinfer/decode.py", line 1093, in plan
(EngineCore pid=269)     self._plan_info = self._cached_module.plan(
(EngineCore pid=269)                       ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "python/tvm_ffi/cython/function.pxi", line 929, in tvm_ffi.core.Function.__call__
(EngineCore pid=269) RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.
[rank0]:[W420 16:11:22.693089162 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/benchmark/throughput.py", line 21, in cmd
    main(args)
  File "/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py", line 879, in main
    elapsed_time, request_outputs = run_vllm(
                                    ^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py", line 55, in run_vllm
    llm = LLM.from_engine_args(engine_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py", line 415, in from_engine_args
    return cls(**vars(engine_args))
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py", line 382, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 177, in from_engine_args
    return cls(
           ^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 111, in __init__
    self.engine_core = EngineCoreClient.make_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 101, in make_client
    return SyncMPClient(vllm_config, executor_class, log_stats)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 710, in __init__
    super().__init__(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 535, in __init__
    with launch_core_engines(
         ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 998, in launch_core_engines
    wait_for_engine_startup(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1057, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

Collecting environment information...
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:435: UserWarning:
    Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (8.0) - (12.0)

  queued_call()
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (aarch64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.14.0-1014-nvidia-aarch64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   :
GPU models and configuration : GPU 0: NVIDIA GB10
Nvidia driver version        : 580.95.05
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  20
On-line CPU(s) list:                     0-19
Vendor ID:                               ARM
BIOS Vendor ID:                          NVIDIA
BIOS Model name:                         GB10
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      5
Socket(s):                               1
Stepping:                                r0p1
CPU max MHz:                             2808.0000
CPU min MHz:                             338.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
BIOS Model name:                         GB10
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      5
Socket(s):                               1
Stepping:                                r0p1
CPU max MHz:                             3900.0000
CPU min MHz:                             1378.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
BIOS Model name:                         GB10
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      5
Socket(s):                               1
Stepping:                                r0p1
CPU max MHz:                             2860.0000
CPU min MHz:                             338.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
BIOS Model name:                         GB10
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      5
Socket(s):                               1
Stepping:                                r0p1
CPU max MHz:                             4004.0000
CPU min MHz:                             1378.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
L1d cache:                               1.3 MiB (20 instances)
L1i cache:                               1.3 MiB (20 instances)
L2 cache:                                25 MiB (20 instances)
L3 cache:                                24 MiB (2 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-19
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Not affected
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, but not BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.6
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu129
[pip3] torchvision==0.25.0+cu129
[pip3] transformers==5.5.4
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.19.1
vLLM Build Flags:
  CUDA Archs: 8.7 8.9 9.0 10.0+PTX 12.0; ROCm: Disabled
GPU Topology:
        GPU0    NIC0    NIC1    NIC2    NIC3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    NODE    NODE    NODE    0-19    0               N/A
NIC0    NODE     X      PIX     NODE    NODE
NIC1    NODE    PIX      X      NODE    NODE
NIC2    NODE    NODE    NODE     X      PIX
NIC3    NODE    NODE    NODE    PIX      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: rocep1s0f0
  NIC1: rocep1s0f1
  NIC2: roceP2p1s0f0
  NIC3: roceP2p1s0f1

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=void
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
TORCH_CUDA_ARCH_LIST=8.7 8.9 9.0 10.0+PTX 12.0
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.9.1
VLLM_ENABLE_CUDA_COMPATIBILITY=0
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64
NVIDIA_CTK_LIBCUDA_DIR=/usr/lib/aarch64-linux-gnu
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

</details>

🐛 Describe the bug

The bug can be safely reproduce on Spark, B100 and B200(single GPU device) on latest build(v0.19.1)

pull the image:

docker run \
      --gpus all --runtime=nvidia --privileged \
      -it --rm -u 0:0 \
      --shm-size=256g \
      --ulimit memlock=-1 --ulimit stack=67108864 \
      --ipc=host --network=host \
      -v /tmp/.cache/huggingface:/root/.cache/huggingface \
      --entrypoint /bin/bash \
      vllm/vllm-openai:latest

reproduce the bug: on Spark: Qwen/Qwen3.5-122B-A10B-GPTQ-Int4

sync && echo 3 | tee /proc/sys/vm/drop_caches && \
vllm bench throughput \
  --model=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
  --trust-remote-code --load-format=dummy \
  --num-prompts=32 --output-len=256 --input-len=256 \
  --quantization=gptq_marlin --kv-cache-dtype=auto \
  --gpu-memory-utilization=0.85 --max-model-len=2048 \
  --max-num-batched-tokens=2048 --max-num-seqs=512 \
  --attention-backend=flashinfer \
  --tensor-parallel-size=1

RedHatAI/Qwen3.5-122B-A10B-NVFP4

sync && echo 3 | tee /proc/sys/vm/drop_caches && \
  vllm bench throughput \
    --model=RedHatAI/Qwen3.5-122B-A10B-NVFP4 \
    --tokenizer=Qwen/Qwen3.5-122B-A10B \
    --trust-remote-code --load-format=dummy \
    --num-prompts=32 --output-len=256 --input-len=256 \
    --kv-cache-dtype=auto \
    --gpu-memory-utilization=0.85 --max-model-len=2048 \
    --attention-backend=flashinfer \
    --tensor-parallel-size=1

on B100 and B200, you need to add --attention-config '{"use_trtllm_attention": false}' for avoiding attention backend fall back to TRTLLM

the error message:

sync && echo 3 | tee /proc/sys/vm/drop_caches && \
  vllm bench throughput \
    --model=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
    --trust-remote-code --load-format=dummy \
    --num-prompts=32 --output-len=256 --input-len=256 \
    --quantization=gptq_marlin --kv-cache-dtype=auto \
    --gpu-memory-utilization=0.85 --max-model-len=2048 \
    --max-num-batched-tokens=2048 --max-num-seqs=512 \
    --attention-backend=flashinfer \
    --tensor-parallel-size=1 2>&1 | tee /tmp/qwen35-122B-gptq-flashinfer-spark.log
3
/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py:848: UserWarning: Both --input-len and --random-input-len are specified. The random version (--random-input-len) will be preferred in this run.
  validate_args(args)
/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py:848: UserWarning: Both --output-len and --random-output-len are specified. The random version (--random-output-len) will be preferred in this run.
  validate_args(args)
/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py:848: UserWarning: Both --prefix-len and --random-prefix-len are specified. The random version (--random-prefix-len) will be preferred in this run.
  validate_args(args)
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
When dataset path is not set, it will default to random dataset
INFO 04-20 16:08:32 [datasets.py:700] Sampling input_len from [1024, 1024] and output_len from [128, 128]
INFO 04-20 16:08:32 [utils.py:233] non-default args: {'tokenizer': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', 'trust_remote_code': True, 'load_format': 'dummy', 'max_model_len': 2048, 'gpu_memory_utilization': 0.85, 'max_num_batched_tokens': 2048, 'max_num_seqs': 512, 'quantization': 'gptq_marlin', 'enable_lora': None, 'reasoning_parser_plugin': '', 'attention_backend': 'flashinfer', 'model': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4'}
INFO 04-20 16:08:40 [model.py:549] Resolved architecture: Qwen3_5MoeForConditionalGeneration
INFO 04-20 16:08:40 [model.py:1678] Using max model len 2048
INFO 04-20 16:08:40 [gptq_marlin.py:229] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:435: UserWarning:
    Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (8.0) - (12.0)

  queued_call()
`Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
INFO 04-20 16:08:40 [config.py:281] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size.
INFO 04-20 16:08:40 [config.py:312] Padding mamba page size by 0.58% to ensure that mamba page size and attention page size are exactly equal.
Parse safetensors files: 100%|██████████| 39/39 [00:02<00:00, 13.60it/s]
INFO 04-20 16:08:45 [vllm.py:790] Asynchronous scheduling is enabled.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=269) INFO 04-20 16:09:02 [core.py:105] Initializing a V1 LLM engine (v0.19.1) with config: model='Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', speculative_config=None, tokenizer='Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=dummy, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:435: UserWarning:
(EngineCore pid=269)     Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
(EngineCore pid=269)     Minimum and Maximum cuda capability supported by this version of PyTorch is
(EngineCore pid=269)     (8.0) - (12.0)
(EngineCore pid=269)
(EngineCore pid=269)   queued_call()
(EngineCore pid=269) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=269) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(EngineCore pid=269) INFO 04-20 16:09:05 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.57.195.222:55441 backend=nccl
(EngineCore pid=269) INFO 04-20 16:09:05 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=269) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=269) INFO 04-20 16:09:16 [gpu_model_runner.py:4735] Starting to load model Qwen/Qwen3.5-122B-A10B-GPTQ-Int4...
(EngineCore pid=269) INFO 04-20 16:09:16 [cuda.py:390] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=269) INFO 04-20 16:09:16 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=269) INFO 04-20 16:09:17 [gdn_linear_attn.py:147] Using Triton/FLA GDN prefill kernel
(EngineCore pid=269) INFO 04-20 16:09:18 [cuda.py:274] Using AttentionBackendEnum.FLASHINFER backend.
(EngineCore pid=269) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore pid=269) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore pid=269) INFO 04-20 16:09:25 [gpu_model_runner.py:4820] Model loading took 68.36 GiB memory and 8.165419 seconds
(EngineCore pid=269) INFO 04-20 16:09:25 [gpu_model_runner.py:5753] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=269) INFO 04-20 16:09:36 [backends.py:1051] Using cache directory: /root/.cache/vllm/torch_compile_cache/14431a8247/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=269) INFO 04-20 16:09:36 [backends.py:1111] Dynamo bytecode transform time: 5.26 s
(EngineCore pid=269) [rank0]:W0420 16:09:41.870000 269 torch/_inductor/utils.py:1679] Not enough SMs to use max_autotune_gemm mode
(EngineCore pid=269) INFO 04-20 16:09:43 [backends.py:372] Cache the graph of compile range (1, 2048) for later use
(EngineCore pid=269) INFO 04-20 16:10:04 [backends.py:390] Compiling a graph for compile range (1, 2048) takes 27.57 s
(EngineCore pid=269) INFO 04-20 16:10:05 [decorators.py:655] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/a3057a8f26052af55dccb86f4e1004a7008bf8d77be2a6a5d83ce0a4f64010ac/rank_0_0/model
(EngineCore pid=269) INFO 04-20 16:10:05 [monitor.py:48] torch.compile took 33.86 s in total
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(EngineCore pid=269)   return fn(*contiguous_args, **contiguous_kwargs)
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (32) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(EngineCore pid=269)   return fn(*contiguous_args, **contiguous_kwargs)
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (16) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(EngineCore pid=269)   return fn(*contiguous_args, **contiguous_kwargs)
(EngineCore pid=269) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (32) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].
(EngineCore pid=269)   return fn(*contiguous_args, **contiguous_kwargs)
(EngineCore pid=269) INFO 04-20 16:11:14 [monitor.py:76] Initial profiling/warmup run took 69.19 s
(EngineCore pid=269) INFO 04-20 16:11:19 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(EngineCore pid=269) INFO 04-20 16:11:20 [gpu_model_runner.py:5876] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] EngineCore failed to start.
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] Traceback (most recent call last):
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in __init__
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     super().__init__(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 124, in __init__
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 247, in _initialize_kv_caches
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5905, in profile_cudagraph_memory
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     self._warmup_and_capture(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6066, in _warmup_and_capture
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     self._dummy_run(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5394, in _dummy_run
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     attn_metadata, _ = self._build_attention_metadata(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2319, in _build_attention_metadata
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     _build_attn_group_metadata(kv_cache_gid, attn_gid, cm)
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2270, in _build_attn_group_metadata
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     attn_metadata_i = builder.build(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                       ^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1158, in build
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     fast_plan_decode(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1719, in fast_plan_decode
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     self.plan(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/decode.py", line 1093, in plan
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]     self._plan_info = self._cached_module.plan(
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]                       ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108]   File "python/tvm_ffi/cython/function.pxi", line 929, in tvm_ffi.core.Function.__call__
(EngineCore pid=269) ERROR 04-20 16:11:21 [core.py:1108] RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.
(EngineCore pid=269) Process EngineCore:
(EngineCore pid=269) Traceback (most recent call last):
(EngineCore pid=269)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=269)     self.run()
(EngineCore pid=269)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=269)     self._target(*self._args, **self._kwargs)
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1112, in run_engine_core
(EngineCore pid=269)     raise e
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core
(EngineCore pid=269)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=269)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in __init__
(EngineCore pid=269)     super().__init__(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 124, in __init__
(EngineCore pid=269)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=269)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 247, in _initialize_kv_caches
(EngineCore pid=269)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=269)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=269)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=269)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=269)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory
(EngineCore pid=269)     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=269)                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5905, in profile_cudagraph_memory
(EngineCore pid=269)     self._warmup_and_capture(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6066, in _warmup_and_capture
(EngineCore pid=269)     self._dummy_run(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=269)     return func(*args, **kwargs)
(EngineCore pid=269)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5394, in _dummy_run
(EngineCore pid=269)     attn_metadata, _ = self._build_attention_metadata(
(EngineCore pid=269)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2319, in _build_attention_metadata
(EngineCore pid=269)     _build_attn_group_metadata(kv_cache_gid, attn_gid, cm)
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2270, in _build_attn_group_metadata
(EngineCore pid=269)     attn_metadata_i = builder.build(
(EngineCore pid=269)                       ^^^^^^^^^^^^^^
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1158, in build
(EngineCore pid=269)     fast_plan_decode(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1719, in fast_plan_decode
(EngineCore pid=269)     self.plan(
(EngineCore pid=269)   File "/usr/local/lib/python3.12/dist-packages/flashinfer/decode.py", line 1093, in plan
(EngineCore pid=269)     self._plan_info = self._cached_module.plan(
(EngineCore pid=269)                       ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=269)   File "python/tvm_ffi/cython/function.pxi", line 929, in tvm_ffi.core.Function.__call__
(EngineCore pid=269) RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.
[rank0]:[W420 16:11:22.693089162 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/benchmark/throughput.py", line 21, in cmd
    main(args)
  File "/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py", line 879, in main
    elapsed_time, request_outputs = run_vllm(
                                    ^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/throughput.py", line 55, in run_vllm
    llm = LLM.from_engine_args(engine_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py", line 415, in from_engine_args
    return cls(**vars(engine_args))
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py", line 382, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 177, in from_engine_args
    return cls(
           ^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 111, in __init__
    self.engine_core = EngineCoreClient.make_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 101, in make_client
    return SyncMPClient(vllm_config, executor_class, log_stats)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 710, in __init__
    super().__init__(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 535, in __init__
    with launch_core_engines(
         ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 998, in launch_core_engines
    wait_for_engine_startup(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1057, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The error is caused by a buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes are available in AlignedAllocator, suggesting that increasing the workspace buffer size may resolve the issue.

Guidance

Increase the workspace buffer size: The error message explicitly suggests increasing the workspace buffer size to resolve the buffer overflow issue. This can be done by adjusting the configuration or environment variables related to memory allocation.
Verify GPU memory utilization: Ensure that the GPU memory utilization is not exceeding the available memory, as specified by the --gpu-memory-utilization flag. Adjusting this flag may help prevent memory overallocation.
Check for memory leaks: Although not directly indicated, it's essential to verify that there are no memory leaks in the application, as these can contribute to unexpected memory allocation issues.

Example

No specific code example is provided, as the issue seems to be related to configuration and memory allocation rather than a specific code snippet.

Notes

The issue is specific to the flashinfer attention backend and the vllm application.
Increasing the workspace buffer size may require adjustments to the system configuration or environment variables.
It's crucial to monitor GPU memory utilization and adjust the --gpu-memory-utilization flag accordingly to prevent similar issues.

Recommendation

Apply a workaround by increasing the workspace buffer size, as this directly addresses the error message's suggestion. If the issue persists, further investigation into memory leaks or other configuration adjustments may be necessary.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#tensor shape #model loading #environment variable #integration issue #index setup

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - ✅(Solved) Fix [Bug]: Buffer overflow when allocating memory error on Qwen3.5-122B-A10B-GPTQ-Int4 and NVFP4 [1 pull requests]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

PR fix notes

PR #40383: [BugFix]Auto-size FlashInfer workspace buffer to prevent buffer overflows on model

Description (problem / solution / changelog)

Purpose

Test Plan

Test Result

Changed files

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

TRENDING

vllm - ✅(Solved) Fix [Bug]: Buffer overflow when allocating memory error on Qwen3.5-122B-A10B-GPTQ-Int4 and NVFP4 [1 pull requests]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

PR fix notes

PR #40383: [BugFix]Auto-size FlashInfer workspace buffer to prevent buffer overflows on model

Description (problem / solution / changelog)

Purpose

Test Plan

Test Result

Changed files

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

RELATED_DISCOVERY

TRENDING