vllm - 💡(How to fix) Fix [Bug]: FlashInfer workspace buffer overflow during CUDA graph capture [1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#40023Fetched 2026-04-17 08:27:33
View on GitHub
Comments
0
Participants
1
Timeline
2
Reactions
0
Author
Participants
Timeline (top)
closed ×1labeled ×1

vLLM crashes during server startup with FlashInfer aligned_alloc buffer overflow. See error and stack trace below.

Related: #25342

Error Message

RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49:
  Buffer overflow when allocating memory for batch_prefill_tmp_v
  with size 536346624 and alignment 16,
  but only 413138944 bytes available in AlignedAllocator.
  Increase the workspace buffer size.

Root Cause

+ export VLLM_USE_FLASHINFER_MOE_FP4=1
+ VLLM_USE_FLASHINFER_MOE_FP4=1
+ '[' 1 -gt 1 ']'
+ VLLM_SERVER__MODEL_ARG='--model nvidia/Qwen3.5-397B-A17B-NVFP4'
+ VLLM_SERVER__TP_ARG=
+ VLLM_SERVER__PP_ARG=
+ VLLM_SERVER__DATA_PARALLEL_SIZE_ARG='--data-parallel-size 8'
+ VLLM_SERVER__DTYPE_ARG=
+ VLLM_SERVER__KV_CACHE_DTYPE_ARG='--kv-cache-dtype fp8_e4m3'
+ '[' 528 '!=' 512 ']'
+ VLLM_SERVER__MAX_NUM_SEQS_ARG='--max-num-seqs 528'
+ VLLM_SERVER__TRUST_REMOTE_CODE_ARG=--trust-remote-code
+ VLLM_SERVER__MAX_MODEL_LEN_ARG='--max-model-len 3072'
+ VLLM_SERVER__GPU_MEMORY_UTILIZATION_ARG='--gpu-memory-utilization 0.85'
vLLM runtime version: 0.19.1rc1.dev231+g9dd5ee011.cu130
Starting vLLM server...
+ python3 -m vllm.entrypoints.openai.api_server --model nvidia/Qwen3.5-397B-A17B-NVFP4 --data-parallel-size 8 --kv-cache-dtype fp8_e4m3 --max-num-seqs 528 --trust-remote-code --max-model-len 3072 --gpu-memory-utilization 0.85 --no-enable-prefix-caching --language-model-only --async-scheduling --attention-backend FLASHINFER --enable-expert-parallel --quantization modelopt --host 0.0.0.0 --port 60000
(APIServer pid=1936149) INFO 04-13 17:07:51 [utils.py:233] non-default args: {'host': '0.0.0.0', 'port': 60000, 'model': 'nvidia/Qwen3.5-397B-A17B-NVFP4', 'trust_remote_code': True, 'max_model_len': 3072, 'quantization': 'modelopt', 'attention_backend': 'FLASHINFER', 'data_parallel_size': 8, 'enable_expert_parallel': True, 'gpu_memory_utilization': 0.85, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': False, 'language_model_only': True, 'max_num_seqs': 528, 'async_scheduling': True}
(APIServer pid=1936149) INFO 04-13 17:08:09 [model.py:554] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1936149) INFO 04-13 17:08:09 [model.py:1684] Using max model len 3072
(APIServer pid=1936149) INFO 04-13 17:08:09 [scheduler.py:238] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1936149) WARNING 04-13 17:08:10 [modelopt.py:1011] Detected ModelOpt NVFP4 checkpoint. Please note that the format is experimental and could change in future.
(APIServer pid=1936149) INFO 04-13 17:08:10 [vllm.py:809] Asynchronous scheduling is enabled.
(Worker_DP0_EP0 pid=1936437) INFO 04-13 17:14:34 [monitor.py:76] Initial profiling/warmup run took 96.15 s
(Worker_DP6_EP6 pid=1936440) INFO 04-13 17:14:46 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_DP6_EP6 pid=1936440) INFO 04-13 17:14:46 [gpu_model_runner.py:5914] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971] WorkerProc hit an exception.
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5943, in profile_cudagraph_memory
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     self._warmup_and_capture(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6104, in _warmup_and_capture
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     self._dummy_run(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5421, in _dummy_run
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     attn_metadata, _ = self._build_attention_metadata(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1164, in build
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     fast_plan_decode(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1722, in fast_plan_decode
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     self.plan(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/decode.py", line 1099, in plan
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971] RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.
(EngineCore_DP5 pid=1936291) ERROR 04-13 17:14:52 [core.py:1110] EngineCore failed to start.
(EngineCore_DP5 pid=1936291) ERROR 04-13 17:14:52 [core.py:1110] RuntimeError: Worker failed with error 'Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.', please check the stack trace above for the root cause
(Worker_DP5_EP5 pid=1936420) WARNING 04-13 17:14:52 [multiproc_executor.py:893] WorkerProc was terminated

Code Example

python3 -m vllm.entrypoints.openai.api_server \
    --model nvidia/Qwen3.5-397B-A17B-NVFP4 \
    --data-parallel-size 8 \
    --kv-cache-dtype fp8_e4m3 \
    --max-num-seqs 528 \
    --trust-remote-code \
    --max-model-len 3072 \
    --gpu-memory-utilization 0.85 \
    --no-enable-prefix-caching \
    --language-model-only \
    --async-scheduling \
    --attention-backend FLASHINFER \
    --enable-expert-parallel \
    --quantization modelopt \
    --host 0.0.0.0 --port 60000

---

RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49:
  Buffer overflow when allocating memory for batch_prefill_tmp_v
  with size 536346624 and alignment 16,
  but only 413138944 bytes available in AlignedAllocator.
  Increase the workspace buffer size.

---

+ export VLLM_USE_FLASHINFER_MOE_FP4=1
+ VLLM_USE_FLASHINFER_MOE_FP4=1
+ '[' 1 -gt 1 ']'
+ VLLM_SERVER__MODEL_ARG='--model nvidia/Qwen3.5-397B-A17B-NVFP4'
+ VLLM_SERVER__TP_ARG=
+ VLLM_SERVER__PP_ARG=
+ VLLM_SERVER__DATA_PARALLEL_SIZE_ARG='--data-parallel-size 8'
+ VLLM_SERVER__DTYPE_ARG=
+ VLLM_SERVER__KV_CACHE_DTYPE_ARG='--kv-cache-dtype fp8_e4m3'
+ '[' 528 '!=' 512 ']'
+ VLLM_SERVER__MAX_NUM_SEQS_ARG='--max-num-seqs 528'
+ VLLM_SERVER__TRUST_REMOTE_CODE_ARG=--trust-remote-code
+ VLLM_SERVER__MAX_MODEL_LEN_ARG='--max-model-len 3072'
+ VLLM_SERVER__GPU_MEMORY_UTILIZATION_ARG='--gpu-memory-utilization 0.85'
vLLM runtime version: 0.19.1rc1.dev231+g9dd5ee011.cu130
Starting vLLM server...
+ python3 -m vllm.entrypoints.openai.api_server --model nvidia/Qwen3.5-397B-A17B-NVFP4 --data-parallel-size 8 --kv-cache-dtype fp8_e4m3 --max-num-seqs 528 --trust-remote-code --max-model-len 3072 --gpu-memory-utilization 0.85 --no-enable-prefix-caching --language-model-only --async-scheduling --attention-backend FLASHINFER --enable-expert-parallel --quantization modelopt --host 0.0.0.0 --port 60000
(APIServer pid=1936149) INFO 04-13 17:07:51 [utils.py:233] non-default args: {'host': '0.0.0.0', 'port': 60000, 'model': 'nvidia/Qwen3.5-397B-A17B-NVFP4', 'trust_remote_code': True, 'max_model_len': 3072, 'quantization': 'modelopt', 'attention_backend': 'FLASHINFER', 'data_parallel_size': 8, 'enable_expert_parallel': True, 'gpu_memory_utilization': 0.85, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': False, 'language_model_only': True, 'max_num_seqs': 528, 'async_scheduling': True}
(APIServer pid=1936149) INFO 04-13 17:08:09 [model.py:554] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1936149) INFO 04-13 17:08:09 [model.py:1684] Using max model len 3072
(APIServer pid=1936149) INFO 04-13 17:08:09 [scheduler.py:238] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1936149) WARNING 04-13 17:08:10 [modelopt.py:1011] Detected ModelOpt NVFP4 checkpoint. Please note that the format is experimental and could change in future.
(APIServer pid=1936149) INFO 04-13 17:08:10 [vllm.py:809] Asynchronous scheduling is enabled.
(Worker_DP0_EP0 pid=1936437) INFO 04-13 17:14:34 [monitor.py:76] Initial profiling/warmup run took 96.15 s
(Worker_DP6_EP6 pid=1936440) INFO 04-13 17:14:46 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_DP6_EP6 pid=1936440) INFO 04-13 17:14:46 [gpu_model_runner.py:5914] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971] WorkerProc hit an exception.
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5943, in profile_cudagraph_memory
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     self._warmup_and_capture(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6104, in _warmup_and_capture
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     self._dummy_run(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5421, in _dummy_run
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     attn_metadata, _ = self._build_attention_metadata(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1164, in build
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     fast_plan_decode(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1722, in fast_plan_decode
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     self.plan(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/decode.py", line 1099, in plan
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971] RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.
(EngineCore_DP5 pid=1936291) ERROR 04-13 17:14:52 [core.py:1110] EngineCore failed to start.
(EngineCore_DP5 pid=1936291) ERROR 04-13 17:14:52 [core.py:1110] RuntimeError: Worker failed with error 'Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.', please check the stack trace above for the root cause
(Worker_DP5_EP5 pid=1936420) WARNING 04-13 17:14:52 [multiproc_executor.py:893] WorkerProc was terminated
RAW_BUFFERClick to expand / collapse

Your current environment

  • vLLM: 0.19.1rc1.dev231+g9dd5ee011.cu130
  • Model: nvidia/Qwen3.5-397B-A17B-NVFP4
  • Hardware: B300

🐛 Describe the bug

Summary

vLLM crashes during server startup with FlashInfer aligned_alloc buffer overflow. See error and stack trace below.

Related: #25342

Reproduce

python3 -m vllm.entrypoints.openai.api_server \
    --model nvidia/Qwen3.5-397B-A17B-NVFP4 \
    --data-parallel-size 8 \
    --kv-cache-dtype fp8_e4m3 \
    --max-num-seqs 528 \
    --trust-remote-code \
    --max-model-len 3072 \
    --gpu-memory-utilization 0.85 \
    --no-enable-prefix-caching \
    --language-model-only \
    --async-scheduling \
    --attention-backend FLASHINFER \
    --enable-expert-parallel \
    --quantization modelopt \
    --host 0.0.0.0 --port 60000

Error

RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49:
  Buffer overflow when allocating memory for batch_prefill_tmp_v
  with size 536346624 and alignment 16,
  but only 413138944 bytes available in AlignedAllocator.
  Increase the workspace buffer size.

Affected configurations

Observed with nvidia/Qwen3.5-397B-A17B-NVFP4 on B300. Crashes at startup with --max-num-seqs 528 and above:

  • -tp 1 -dp 2 -pp 1 --enable-expert-parallel
  • -tp 1 -dp 4 -pp 1 --enable-expert-parallel
  • -tp 1 -dp 8 -pp 1 --enable-expert-parallel
  • -tp 2 -dp 1 -pp 1
  • -tp 2 -dp 1 -pp 1 --enable-expert-parallel

Not affected:

  • -tp 4 -dp 1 -pp 1
  • -tp 8 -dp 1 -pp 1

Server log

+ export VLLM_USE_FLASHINFER_MOE_FP4=1
+ VLLM_USE_FLASHINFER_MOE_FP4=1
+ '[' 1 -gt 1 ']'
+ VLLM_SERVER__MODEL_ARG='--model nvidia/Qwen3.5-397B-A17B-NVFP4'
+ VLLM_SERVER__TP_ARG=
+ VLLM_SERVER__PP_ARG=
+ VLLM_SERVER__DATA_PARALLEL_SIZE_ARG='--data-parallel-size 8'
+ VLLM_SERVER__DTYPE_ARG=
+ VLLM_SERVER__KV_CACHE_DTYPE_ARG='--kv-cache-dtype fp8_e4m3'
+ '[' 528 '!=' 512 ']'
+ VLLM_SERVER__MAX_NUM_SEQS_ARG='--max-num-seqs 528'
+ VLLM_SERVER__TRUST_REMOTE_CODE_ARG=--trust-remote-code
+ VLLM_SERVER__MAX_MODEL_LEN_ARG='--max-model-len 3072'
+ VLLM_SERVER__GPU_MEMORY_UTILIZATION_ARG='--gpu-memory-utilization 0.85'
vLLM runtime version: 0.19.1rc1.dev231+g9dd5ee011.cu130
Starting vLLM server...
+ python3 -m vllm.entrypoints.openai.api_server --model nvidia/Qwen3.5-397B-A17B-NVFP4 --data-parallel-size 8 --kv-cache-dtype fp8_e4m3 --max-num-seqs 528 --trust-remote-code --max-model-len 3072 --gpu-memory-utilization 0.85 --no-enable-prefix-caching --language-model-only --async-scheduling --attention-backend FLASHINFER --enable-expert-parallel --quantization modelopt --host 0.0.0.0 --port 60000
(APIServer pid=1936149) INFO 04-13 17:07:51 [utils.py:233] non-default args: {'host': '0.0.0.0', 'port': 60000, 'model': 'nvidia/Qwen3.5-397B-A17B-NVFP4', 'trust_remote_code': True, 'max_model_len': 3072, 'quantization': 'modelopt', 'attention_backend': 'FLASHINFER', 'data_parallel_size': 8, 'enable_expert_parallel': True, 'gpu_memory_utilization': 0.85, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': False, 'language_model_only': True, 'max_num_seqs': 528, 'async_scheduling': True}
(APIServer pid=1936149) INFO 04-13 17:08:09 [model.py:554] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1936149) INFO 04-13 17:08:09 [model.py:1684] Using max model len 3072
(APIServer pid=1936149) INFO 04-13 17:08:09 [scheduler.py:238] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1936149) WARNING 04-13 17:08:10 [modelopt.py:1011] Detected ModelOpt NVFP4 checkpoint. Please note that the format is experimental and could change in future.
(APIServer pid=1936149) INFO 04-13 17:08:10 [vllm.py:809] Asynchronous scheduling is enabled.
(Worker_DP0_EP0 pid=1936437) INFO 04-13 17:14:34 [monitor.py:76] Initial profiling/warmup run took 96.15 s
(Worker_DP6_EP6 pid=1936440) INFO 04-13 17:14:46 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_DP6_EP6 pid=1936440) INFO 04-13 17:14:46 [gpu_model_runner.py:5914] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=51 (largest=512)
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971] WorkerProc hit an exception.
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5943, in profile_cudagraph_memory
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     self._warmup_and_capture(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6104, in _warmup_and_capture
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     self._dummy_run(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5421, in _dummy_run
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     attn_metadata, _ = self._build_attention_metadata(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1164, in build
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     fast_plan_decode(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py", line 1722, in fast_plan_decode
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]     self.plan(
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/decode.py", line 1099, in plan
(Worker_DP5_EP5 pid=1936420) ERROR 04-13 17:14:52 [multiproc_executor.py:971] RuntimeError: Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.
(EngineCore_DP5 pid=1936291) ERROR 04-13 17:14:52 [core.py:1110] EngineCore failed to start.
(EngineCore_DP5 pid=1936291) ERROR 04-13 17:14:52 [core.py:1110] RuntimeError: Worker failed with error 'Error in function 'aligned_alloc' at /workspace/include/flashinfer/allocator.h:49: Buffer overflow when allocating memory for batch_prefill_tmp_v with size 536346624 and alignment 16, but only 413138944 bytes available in AlignedAllocator. Increase the workspace buffer size.', please check the stack trace above for the root cause
(Worker_DP5_EP5 pid=1936420) WARNING 04-13 17:14:52 [multiproc_executor.py:893] WorkerProc was terminated

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The most likely fix is to increase the workspace buffer size to prevent the buffer overflow error in the aligned_alloc function.

Guidance

  • The error message indicates a buffer overflow when allocating memory for batch_prefill_tmp_v with a size of 536346624 bytes and an alignment of 16, but only 413138944 bytes are available in the AlignedAllocator.
  • To fix this, you can try increasing the workspace buffer size by adjusting the relevant configuration parameters or environment variables.
  • Check the documentation for the flashinfer library and the vllm framework to see if there are any configuration options or environment variables that control the workspace buffer size.
  • You can also try reducing the --max-num-seqs parameter to a value below 528, as the issue seems to be triggered by this parameter.

Example

No code snippet is provided as the issue is related to configuration and memory allocation.

Notes

The exact steps to increase the workspace buffer size are not provided as they depend on the specific configuration and environment variables used in the project. It is recommended to consult the documentation for the flashinfer library and the vllm framework for more information.

Recommendation

Apply a workaround by reducing the --max-num-seqs parameter to a value below 528 or increasing the workspace buffer size, as the root cause of the issue is related to memory allocation and buffer overflow.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING