vllm - 💡(How to fix) Fix [Bug]: Qwen 3.6 awq can't load, always OOM error

vllm2026-05-09 08:27:06

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

Error Message

(z3) z@z3:~$ VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 vllm serve cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit --gpu-memory-utilization 0.6 --kv-cache-dtype fp8 --max-model-len 65536 --enable-prefix-caching --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --language-model-only --cpu-offload-gb 6 (APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] (APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] █ █ █▄ ▄█ (APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.20.1 (APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] █▄█▀ █ █ █ █ model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit (APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] (APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:233] non-default args: {'model_tag': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'max_model_len': 65536, 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.6, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'cpu_offload_gb': 6.0, 'language_model_only': True} (APIServer pid=72781) INFO 05-09 16:20:50 [model.py:555] Resolved architecture: Qwen3_5MoeForConditionalGeneration (APIServer pid=72781) INFO 05-09 16:20:50 [model.py:1680] Using max model len 65536 (APIServer pid=72781) INFO 05-09 16:20:51 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL. (APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:34] NIXL is not available (APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:44] NIXL agent config is not available (APIServer pid=72781) INFO 05-09 16:20:51 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor (APIServer pid=72781) WARNING 05-09 16:20:51 [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled (APIServer pid=72781) INFO 05-09 16:20:51 [config.py:387] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe. (APIServer pid=72781) INFO 05-09 16:20:51 [vllm.py:840] Asynchronous scheduling is enabled. (APIServer pid=72781) INFO 05-09 16:20:51 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native']) (APIServer pid=72781) [transformers] Qwen2VLImageProcessorFast is deprecated. The Fast suffix for image processors has been removed; use Qwen2VLImageProcessor instead. (APIServer pid=72781) INFO 05-09 16:20:56 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode. WARNING 05-09 16:21:04 [nixl_utils.py:34] NIXL is not available WARNING 05-09 16:21:04 [nixl_utils.py:44] NIXL agent config is not available (EngineCore pid=72838) INFO 05-09 16:21:04 [core.py:109] Initializing a V1 LLM engine (v0.20.1) with config: model='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', speculative_config=None, tokenizer='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto') (EngineCore pid=72838) [transformers] Qwen2VLImageProcessorFast is deprecated. The Fast suffix for image processors has been removed; use Qwen2VLImageProcessor instead. (EngineCore pid=72838) INFO 05-09 16:21:10 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode. (EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1402] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://139.24.23.9:48235 backend=nccl (EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1715] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A (EngineCore pid=72838) INFO 05-09 16:21:10 [base.py:123] Offloader set to UVAOffloader (EngineCore pid=72838) INFO 05-09 16:21:10 [gpu_model_runner.py:4777] Starting to load model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit... (EngineCore pid=72838) INFO 05-09 16:21:11 [cuda.py:423] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention (EngineCore pid=72838) INFO 05-09 16:21:11 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (EngineCore pid=72838) INFO 05-09 16:21:11 [gdn_linear_attn.py:153] Using Triton/FLA GDN prefill kernel (EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe.py:122] Using CompressedTensorsWNA16MarlinMoEMethod (EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe_wna16_marlin.py:87] Using Marlin backend for WNA16 MoE (group_size=32, num_bits=4) (EngineCore pid=72838) INFO 05-09 16:21:12 [cuda.py:368] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN']. (EngineCore pid=72838) INFO 05-09 16:21:17 [uva.py:58] Total CPU offloaded parameters: 6.24 (EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:904] Filesystem type for checkpoints: FUSE.MERGERFS. Checkpoint size: 22.78 GiB. Available RAM: 114.37 GiB. (EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:927] Auto-prefetch is disabled because the filesystem (FUSE.MERGERFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:11<00:45, 11.38s/it] Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:19<00:27, 9.31s/it] Loading safetensors checkpoint shards: 60% Completed | 3/5 [00:27<00:17, 8.72s/it] Loading safetensors checkpoint shards: 80% Completed | 4/5 [00:37<00:09, 9.39s/it] Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00, 7.64s/it] Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00, 8.44s/it] (EngineCore pid=72838) (EngineCore pid=72838) INFO 05-09 16:22:01 [default_loader.py:384] Loading weights took 42.24 seconds (EngineCore pid=72838) INFO 05-09 16:22:08 [gpu_model_runner.py:4879] Model loading took 15.38 GiB memory and 57.027123 seconds (EngineCore pid=72838) INFO 05-09 16:22:08 [interface.py:606] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size. (EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1069] Using cache directory: /home/z/.cache/vllm/torch_compile_cache/3f9e87c5e2/rank_0_0/backbone for vLLM's torch.compile (EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1128] Dynamo bytecode transform time: 8.01 s (EngineCore pid=72838) INFO 05-09 16:22:18 [backends.py:376] Cache the graph of compile range (1, 2048) for later use (EngineCore pid=72838) INFO 05-09 16:22:48 [backends.py:391] Compiling a graph for compile range (1, 2048) takes 31.27 s (EngineCore pid=72838) INFO 05-09 16:22:51 [decorators.py:668] saved AOT compiled function to /home/z/.cache/vllm/torch_compile_cache/torch_aot_compile/90a747e06d0e8c397ccb2c580c362fed44da87f2a8c85e0b79d323e7dd81626c/rank_0_0/model (EngineCore pid=72838) INFO 05-09 16:22:51 [monitor.py:53] torch.compile took 43.17 s in total (EngineCore pid=72838) INFO 05-09 16:23:57 [monitor.py:81] Initial profiling/warmup run took 65.63 s (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] EngineCore failed to start. (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] Traceback (most recent call last): (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] return func(*args, **kwargs) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in init (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] super().init( (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in init (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] kv_cache_config = self._initialize_kv_caches(vllm_config) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] return func(*args, **kwargs) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] available_gpu_memory = self.model_executor.determine_available_memory() (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] return self.collective_rpc("determine_available_memory") (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] result = run_method(self.driver_worker, method, args, kwargs) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] return func(*args, **kwargs) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] return func(*args, **kwargs) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] return func(*args, **kwargs) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] self._init_minimal_kv_cache_for_profiling() (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] self.initialize_kv_cache(minimal_config, is_profiling=True) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] kv_caches = self.initialize_kv_cache_tensors( (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config) (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] tensor = torch.zeros( (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] ^^^^^^^^^^^^ (EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf) (EngineCore pid=72838) Process EngineCore: (EngineCore pid=72838) Traceback (most recent call last): (EngineCore pid=72838) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap (EngineCore pid=72838) self.run() (EngineCore pid=72838) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run (EngineCore pid=72838) self._target(*self._args, **self._kwargs) (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1140, in run_engine_core (EngineCore pid=72838) raise e (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core (EngineCore pid=72838) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=72838) return func(*args, **kwargs) (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in init (EngineCore pid=72838) super().init( (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in init (EngineCore pid=72838) kv_cache_config = self._initialize_kv_caches(vllm_config) (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=72838) return func(*args, **kwargs) (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches (EngineCore pid=72838) available_gpu_memory = self.model_executor.determine_available_memory() (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory (EngineCore pid=72838) return self.collective_rpc("determine_available_memory") (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc (EngineCore pid=72838) result = run_method(self.driver_worker, method, args, kwargs) (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method (EngineCore pid=72838) return func(*args, **kwargs) (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=72838) return func(*args, **kwargs) (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory (EngineCore pid=72838) cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=72838) return func(*args, **kwargs) (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory (EngineCore pid=72838) self._init_minimal_kv_cache_for_profiling() (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling (EngineCore pid=72838) self.initialize_kv_cache(minimal_config, is_profiling=True) (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache (EngineCore pid=72838) kv_caches = self.initialize_kv_cache_tensors( (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors (EngineCore pid=72838) kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config) (EngineCore pid=72838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=72838) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors (EngineCore pid=72838) tensor = torch.zeros( (EngineCore pid=72838) ^^^^^^^^^^^^ (EngineCore pid=72838) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf) [rank0]:[W509 16:24:06.962826627 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) (APIServer pid=72781) Traceback (most recent call last): (APIServer pid=72781) File "/home/z/vl/.venv/bin/vllm", line 10, in <module> (APIServer pid=72781) sys.exit(main()) (APIServer pid=72781) ^^^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 92, in main (APIServer pid=72781) args.dispatch_function(args) (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd (APIServer pid=72781) uvloop.run(run_server(args)) (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/init.py", line 96, in run (APIServer pid=72781) return __asyncio.run( (APIServer pid=72781) ^^^^^^^^^^^^^^ (APIServer pid=72781) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run (APIServer pid=72781) return runner.run(main) (APIServer pid=72781) ^^^^^^^^^^^^^^^^ (APIServer pid=72781) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run (APIServer pid=72781) return self._loop.run_until_complete(task) (APIServer pid=72781) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=72781) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/init.py", line 48, in wrapper (APIServer pid=72781) return await main (APIServer pid=72781) ^^^^^^^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 678, in run_server (APIServer pid=72781) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server_worker (APIServer pid=72781) async with build_async_engine_client( (APIServer pid=72781) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter (APIServer pid=72781) return await anext(self.gen) (APIServer pid=72781) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client (APIServer pid=72781) async with build_async_engine_client_from_engine_args( (APIServer pid=72781) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter (APIServer pid=72781) return await anext(self.gen) (APIServer pid=72781) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args (APIServer pid=72781) async_llm = AsyncLLM.from_vllm_config( (APIServer pid=72781) ^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config (APIServer pid=72781) return cls( (APIServer pid=72781) ^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in init (APIServer pid=72781) self.engine_core = EngineCoreClient.make_async_mp_client( (APIServer pid=72781) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=72781) return func(*args, **kwargs) (APIServer pid=72781) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client (APIServer pid=72781) return AsyncMPClient(*client_args) (APIServer pid=72781) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=72781) return func(*args, **kwargs) (APIServer pid=72781) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 900, in init (APIServer pid=72781) super().init( (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 535, in init (APIServer pid=72781) with launch_core_engines( (APIServer pid=72781) File "/usr/lib/python3.12/contextlib.py", line 144, in exit (APIServer pid=72781) next(self.gen) (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1119, in launch_core_engines (APIServer pid=72781) wait_for_engine_startup( (APIServer pid=72781) File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1178, in wait_for_engine_startup (APIServer pid=72781) raise RuntimeError( (APIServer pid=72781) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Root Cause

(z3) z@z3:~$ VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 vllm serve cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit --gpu-memory-utilization 0.6 --kv-cache-dtype fp8 --max-model-len 65536 --enable-prefix-caching --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --language-model-only --cpu-offload-gb 6
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] 
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.20.1
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]   █▄█▀ █     █     █     █  model   cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] 
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:233] non-default args: {'model_tag': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'max_model_len': 65536, 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.6, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'cpu_offload_gb': 6.0, 'language_model_only': True}
(APIServer pid=72781) INFO 05-09 16:20:50 [model.py:555] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=72781) INFO 05-09 16:20:50 [model.py:1680] Using max model len 65536
(APIServer pid=72781) INFO 05-09 16:20:51 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL.
(APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:34] NIXL is not available
(APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:44] NIXL agent config is not available
(APIServer pid=72781) INFO 05-09 16:20:51 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=72781) WARNING 05-09 16:20:51 [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=72781) INFO 05-09 16:20:51 [config.py:387] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=72781) INFO 05-09 16:20:51 [vllm.py:840] Asynchronous scheduling is enabled.
(APIServer pid=72781) INFO 05-09 16:20:51 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=72781) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=72781) INFO 05-09 16:20:56 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
WARNING 05-09 16:21:04 [nixl_utils.py:34] NIXL is not available
WARNING 05-09 16:21:04 [nixl_utils.py:44] NIXL agent config is not available
(EngineCore pid=72838) INFO 05-09 16:21:04 [core.py:109] Initializing a V1 LLM engine (v0.20.1) with config: model='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', speculative_config=None, tokenizer='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=72838) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=72838) INFO 05-09 16:21:10 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1402] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://139.24.23.9:48235 backend=nccl
(EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1715] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=72838) INFO 05-09 16:21:10 [base.py:123] Offloader set to UVAOffloader
(EngineCore pid=72838) INFO 05-09 16:21:10 [gpu_model_runner.py:4777] Starting to load model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit...
(EngineCore pid=72838) INFO 05-09 16:21:11 [cuda.py:423] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=72838) INFO 05-09 16:21:11 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=72838) INFO 05-09 16:21:11 [gdn_linear_attn.py:153] Using Triton/FLA GDN prefill kernel
(EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe.py:122] Using CompressedTensorsWNA16MarlinMoEMethod
(EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe_wna16_marlin.py:87] Using Marlin backend for WNA16 MoE (group_size=32, num_bits=4)
(EngineCore pid=72838) INFO 05-09 16:21:12 [cuda.py:368] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(EngineCore pid=72838) INFO 05-09 16:21:17 [uva.py:58] Total CPU offloaded parameters: 6.24
(EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:904] Filesystem type for checkpoints: FUSE.MERGERFS. Checkpoint size: 22.78 GiB. Available RAM: 114.37 GiB.
(EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:927] Auto-prefetch is disabled because the filesystem (FUSE.MERGERFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:11<00:45, 11.38s/it]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:19<00:27,  9.31s/it]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:27<00:17,  8.72s/it]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:37<00:09,  9.39s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00,  7.64s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00,  8.44s/it]
(EngineCore pid=72838) 
(EngineCore pid=72838) INFO 05-09 16:22:01 [default_loader.py:384] Loading weights took 42.24 seconds
(EngineCore pid=72838) INFO 05-09 16:22:08 [gpu_model_runner.py:4879] Model loading took 15.38 GiB memory and 57.027123 seconds
(EngineCore pid=72838) INFO 05-09 16:22:08 [interface.py:606] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1069] Using cache directory: /home/z/.cache/vllm/torch_compile_cache/3f9e87c5e2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1128] Dynamo bytecode transform time: 8.01 s
(EngineCore pid=72838) INFO 05-09 16:22:18 [backends.py:376] Cache the graph of compile range (1, 2048) for later use
(EngineCore pid=72838) INFO 05-09 16:22:48 [backends.py:391] Compiling a graph for compile range (1, 2048) takes 31.27 s
(EngineCore pid=72838) INFO 05-09 16:22:51 [decorators.py:668] saved AOT compiled function to /home/z/.cache/vllm/torch_compile_cache/torch_aot_compile/90a747e06d0e8c397ccb2c580c362fed44da87f2a8c85e0b79d323e7dd81626c/rank_0_0/model
(EngineCore pid=72838) INFO 05-09 16:22:51 [monitor.py:53] torch.compile took 43.17 s in total
(EngineCore pid=72838) INFO 05-09 16:23:57 [monitor.py:81] Initial profiling/warmup run took 65.63 s
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] EngineCore failed to start.
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] Traceback (most recent call last):
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     super().__init__(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     self._init_minimal_kv_cache_for_profiling()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     self.initialize_kv_cache(minimal_config, is_profiling=True)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_caches = self.initialize_kv_cache_tensors(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     tensor = torch.zeros(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]              ^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
(EngineCore pid=72838) Process EngineCore:
(EngineCore pid=72838) Traceback (most recent call last):
(EngineCore pid=72838)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=72838)     self.run()
(EngineCore pid=72838)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=72838)     self._target(*self._args, **self._kwargs)
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1140, in run_engine_core
(EngineCore pid=72838)     raise e
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=72838)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=72838)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=72838)     super().__init__(
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=72838)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=72838)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=72838)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=72838)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=72838)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=72838)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=72838)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory
(EngineCore pid=72838)     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=72838)                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory
(EngineCore pid=72838)     self._init_minimal_kv_cache_for_profiling()
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling
(EngineCore pid=72838)     self.initialize_kv_cache(minimal_config, is_profiling=True)
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache
(EngineCore pid=72838)     kv_caches = self.initialize_kv_cache_tensors(
(EngineCore pid=72838)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors
(EngineCore pid=72838)     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(EngineCore pid=72838)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors
(EngineCore pid=72838)     tensor = torch.zeros(
(EngineCore pid=72838)              ^^^^^^^^^^^^
(EngineCore pid=72838) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
[rank0]:[W509 16:24:06.962826627 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=72781) Traceback (most recent call last):
(APIServer pid=72781)   File "/home/z/vl/.venv/bin/vllm", line 10, in <module>
(APIServer pid=72781)     sys.exit(main())
(APIServer pid=72781)              ^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=72781)     args.dispatch_function(args)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=72781)     uvloop.run(run_server(args))
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=72781)     return __asyncio.run(
(APIServer pid=72781)            ^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=72781)     return runner.run(main)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=72781)     return self._loop.run_until_complete(task)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=72781)     return await main
(APIServer pid=72781)            ^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 678, in run_server
(APIServer pid=72781)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server_worker
(APIServer pid=72781)     async with build_async_engine_client(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=72781)     return await anext(self.gen)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=72781)     async with build_async_engine_client_from_engine_args(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=72781)     return await anext(self.gen)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=72781)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=72781)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=72781)     return cls(
(APIServer pid=72781)            ^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=72781)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=72781)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=72781)     return func(*args, **kwargs)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=72781)     return AsyncMPClient(*client_args)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=72781)     return func(*args, **kwargs)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 900, in __init__
(APIServer pid=72781)     super().__init__(
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=72781)     with launch_core_engines(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=72781)     next(self.gen)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1119, in launch_core_engines
(APIServer pid=72781)     wait_for_engine_startup(
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1178, in wait_for_engine_startup
(APIServer pid=72781)     raise RuntimeError(
(APIServer pid=72781) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: GenuineIntel Model name: Intel(R) Core(TM) i7-6850K CPU @ 3.60GHz CPU family: 6 Model: 79 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 Stepping: 1 CPU(s) scaling MHz: 70% CPU max MHz: 4000.0000 CPU min MHz: 1200.0000 BogoMIPS: 7199.15 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 pti intel_ppin ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d L1d cache: 192 KiB (6 instances) L1i cache: 192 KiB (6 instances) L2 cache: 1.5 MiB (6 instances) L3 cache: 15 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Vmscape: Mitigation; IBPB before exit to userspace

(z3) z@z3:~$ VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 vllm serve cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit --gpu-memory-utilization 0.6 --kv-cache-dtype fp8 --max-model-len 65536 --enable-prefix-caching --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --language-model-only --cpu-offload-gb 6
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] 
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.20.1
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]   █▄█▀ █     █     █     █  model   cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] 
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:233] non-default args: {'model_tag': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'max_model_len': 65536, 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.6, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'cpu_offload_gb': 6.0, 'language_model_only': True}
(APIServer pid=72781) INFO 05-09 16:20:50 [model.py:555] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=72781) INFO 05-09 16:20:50 [model.py:1680] Using max model len 65536
(APIServer pid=72781) INFO 05-09 16:20:51 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL.
(APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:34] NIXL is not available
(APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:44] NIXL agent config is not available
(APIServer pid=72781) INFO 05-09 16:20:51 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=72781) WARNING 05-09 16:20:51 [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=72781) INFO 05-09 16:20:51 [config.py:387] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=72781) INFO 05-09 16:20:51 [vllm.py:840] Asynchronous scheduling is enabled.
(APIServer pid=72781) INFO 05-09 16:20:51 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=72781) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=72781) INFO 05-09 16:20:56 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
WARNING 05-09 16:21:04 [nixl_utils.py:34] NIXL is not available
WARNING 05-09 16:21:04 [nixl_utils.py:44] NIXL agent config is not available
(EngineCore pid=72838) INFO 05-09 16:21:04 [core.py:109] Initializing a V1 LLM engine (v0.20.1) with config: model='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', speculative_config=None, tokenizer='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=72838) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=72838) INFO 05-09 16:21:10 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1402] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://139.24.23.9:48235 backend=nccl
(EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1715] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=72838) INFO 05-09 16:21:10 [base.py:123] Offloader set to UVAOffloader
(EngineCore pid=72838) INFO 05-09 16:21:10 [gpu_model_runner.py:4777] Starting to load model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit...
(EngineCore pid=72838) INFO 05-09 16:21:11 [cuda.py:423] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=72838) INFO 05-09 16:21:11 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=72838) INFO 05-09 16:21:11 [gdn_linear_attn.py:153] Using Triton/FLA GDN prefill kernel
(EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe.py:122] Using CompressedTensorsWNA16MarlinMoEMethod
(EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe_wna16_marlin.py:87] Using Marlin backend for WNA16 MoE (group_size=32, num_bits=4)
(EngineCore pid=72838) INFO 05-09 16:21:12 [cuda.py:368] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(EngineCore pid=72838) INFO 05-09 16:21:17 [uva.py:58] Total CPU offloaded parameters: 6.24
(EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:904] Filesystem type for checkpoints: FUSE.MERGERFS. Checkpoint size: 22.78 GiB. Available RAM: 114.37 GiB.
(EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:927] Auto-prefetch is disabled because the filesystem (FUSE.MERGERFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:11<00:45, 11.38s/it]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:19<00:27,  9.31s/it]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:27<00:17,  8.72s/it]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:37<00:09,  9.39s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00,  7.64s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00,  8.44s/it]
(EngineCore pid=72838) 
(EngineCore pid=72838) INFO 05-09 16:22:01 [default_loader.py:384] Loading weights took 42.24 seconds
(EngineCore pid=72838) INFO 05-09 16:22:08 [gpu_model_runner.py:4879] Model loading took 15.38 GiB memory and 57.027123 seconds
(EngineCore pid=72838) INFO 05-09 16:22:08 [interface.py:606] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1069] Using cache directory: /home/z/.cache/vllm/torch_compile_cache/3f9e87c5e2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1128] Dynamo bytecode transform time: 8.01 s
(EngineCore pid=72838) INFO 05-09 16:22:18 [backends.py:376] Cache the graph of compile range (1, 2048) for later use
(EngineCore pid=72838) INFO 05-09 16:22:48 [backends.py:391] Compiling a graph for compile range (1, 2048) takes 31.27 s
(EngineCore pid=72838) INFO 05-09 16:22:51 [decorators.py:668] saved AOT compiled function to /home/z/.cache/vllm/torch_compile_cache/torch_aot_compile/90a747e06d0e8c397ccb2c580c362fed44da87f2a8c85e0b79d323e7dd81626c/rank_0_0/model
(EngineCore pid=72838) INFO 05-09 16:22:51 [monitor.py:53] torch.compile took 43.17 s in total
(EngineCore pid=72838) INFO 05-09 16:23:57 [monitor.py:81] Initial profiling/warmup run took 65.63 s
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] EngineCore failed to start.
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] Traceback (most recent call last):
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     super().__init__(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     self._init_minimal_kv_cache_for_profiling()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     self.initialize_kv_cache(minimal_config, is_profiling=True)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_caches = self.initialize_kv_cache_tensors(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     tensor = torch.zeros(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]              ^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
(EngineCore pid=72838) Process EngineCore:
(EngineCore pid=72838) Traceback (most recent call last):
(EngineCore pid=72838)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=72838)     self.run()
(EngineCore pid=72838)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=72838)     self._target(*self._args, **self._kwargs)
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1140, in run_engine_core
(EngineCore pid=72838)     raise e
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=72838)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=72838)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=72838)     super().__init__(
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=72838)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=72838)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=72838)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=72838)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=72838)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=72838)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=72838)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory
(EngineCore pid=72838)     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=72838)                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory
(EngineCore pid=72838)     self._init_minimal_kv_cache_for_profiling()
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling
(EngineCore pid=72838)     self.initialize_kv_cache(minimal_config, is_profiling=True)
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache
(EngineCore pid=72838)     kv_caches = self.initialize_kv_cache_tensors(
(EngineCore pid=72838)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors
(EngineCore pid=72838)     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(EngineCore pid=72838)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors
(EngineCore pid=72838)     tensor = torch.zeros(
(EngineCore pid=72838)              ^^^^^^^^^^^^
(EngineCore pid=72838) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
[rank0]:[W509 16:24:06.962826627 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=72781) Traceback (most recent call last):
(APIServer pid=72781)   File "/home/z/vl/.venv/bin/vllm", line 10, in <module>
(APIServer pid=72781)     sys.exit(main())
(APIServer pid=72781)              ^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=72781)     args.dispatch_function(args)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=72781)     uvloop.run(run_server(args))
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=72781)     return __asyncio.run(
(APIServer pid=72781)            ^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=72781)     return runner.run(main)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=72781)     return self._loop.run_until_complete(task)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=72781)     return await main
(APIServer pid=72781)            ^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 678, in run_server
(APIServer pid=72781)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server_worker
(APIServer pid=72781)     async with build_async_engine_client(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=72781)     return await anext(self.gen)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=72781)     async with build_async_engine_client_from_engine_args(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=72781)     return await anext(self.gen)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=72781)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=72781)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=72781)     return cls(
(APIServer pid=72781)            ^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=72781)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=72781)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=72781)     return func(*args, **kwargs)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=72781)     return AsyncMPClient(*client_args)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=72781)     return func(*args, **kwargs)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 900, in __init__
(APIServer pid=72781)     super().__init__(
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=72781)     with launch_core_engines(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=72781)     next(self.gen)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1119, in launch_core_engines
(APIServer pid=72781)     wait_for_engine_startup(
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1178, in wait_for_engine_startup
(APIServer pid=72781)     raise RuntimeError(
(APIServer pid=72781) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Code Example

uv is set
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : version 3.28.3
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.17.0-23-generic-x86_64-with-glibc2.39
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.2.78
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version        : 595.58.03
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_tensor_ir.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.22.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  12
On-line CPU(s) list:                     0-11
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Core(TM) i7-6850K CPU @ 3.60GHz
CPU family:                              6
Model:                                   79
Thread(s) per core:                      2
Core(s) per socket:                      6
Socket(s):                               1
Stepping:                                1
CPU(s) scaling MHz:                      70%
CPU max MHz:                             4000.0000
CPU min MHz:                             1200.0000
BogoMIPS:                                7199.15
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 pti intel_ppin ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
L1d cache:                               192 KiB (6 instances)
L1i cache:                               192 KiB (6 instances)
L2 cache:                                1.5 MiB (6 instances)
L3 cache:                                15 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-11
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             KVM: Mitigation: VMX unsupported
Vulnerability L1tf:                      Mitigation; PTE Inversion
Vulnerability Mds:                       Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:                  Mitigation; PTI
Vulnerability Mmio stale data:           Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.0
[pip3] nvidia-cutlass-dsl-libs-base==4.5.0
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] torch==2.11.0
[pip3] torch-c-dlpack-ext==0.1.5
[pip3] torchaudio==2.11.0
[pip3] torchvision==0.26.0
[pip3] transformers==5.8.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.20.1
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
  	GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-11	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
MAX_JOBS=6
LD_LIBRARY_PATH=:/opt/MVS/lib/64:/opt/MVS/lib/32:/home/z/halcon/lib/x64-linux:/usr/local/cuda-13.2/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_z

---

(z3) z@z3:~$ VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 vllm serve cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit --gpu-memory-utilization 0.6 --kv-cache-dtype fp8 --max-model-len 65536 --enable-prefix-caching --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --language-model-only --cpu-offload-gb 6
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] 
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.20.1
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]   █▄█▀ █     █     █     █  model   cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] 
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:233] non-default args: {'model_tag': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'max_model_len': 65536, 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.6, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'cpu_offload_gb': 6.0, 'language_model_only': True}
(APIServer pid=72781) INFO 05-09 16:20:50 [model.py:555] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=72781) INFO 05-09 16:20:50 [model.py:1680] Using max model len 65536
(APIServer pid=72781) INFO 05-09 16:20:51 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL.
(APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:34] NIXL is not available
(APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:44] NIXL agent config is not available
(APIServer pid=72781) INFO 05-09 16:20:51 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=72781) WARNING 05-09 16:20:51 [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=72781) INFO 05-09 16:20:51 [config.py:387] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=72781) INFO 05-09 16:20:51 [vllm.py:840] Asynchronous scheduling is enabled.
(APIServer pid=72781) INFO 05-09 16:20:51 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=72781) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=72781) INFO 05-09 16:20:56 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
WARNING 05-09 16:21:04 [nixl_utils.py:34] NIXL is not available
WARNING 05-09 16:21:04 [nixl_utils.py:44] NIXL agent config is not available
(EngineCore pid=72838) INFO 05-09 16:21:04 [core.py:109] Initializing a V1 LLM engine (v0.20.1) with config: model='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', speculative_config=None, tokenizer='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=72838) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=72838) INFO 05-09 16:21:10 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1402] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://139.24.23.9:48235 backend=nccl
(EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1715] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=72838) INFO 05-09 16:21:10 [base.py:123] Offloader set to UVAOffloader
(EngineCore pid=72838) INFO 05-09 16:21:10 [gpu_model_runner.py:4777] Starting to load model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit...
(EngineCore pid=72838) INFO 05-09 16:21:11 [cuda.py:423] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=72838) INFO 05-09 16:21:11 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=72838) INFO 05-09 16:21:11 [gdn_linear_attn.py:153] Using Triton/FLA GDN prefill kernel
(EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe.py:122] Using CompressedTensorsWNA16MarlinMoEMethod
(EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe_wna16_marlin.py:87] Using Marlin backend for WNA16 MoE (group_size=32, num_bits=4)
(EngineCore pid=72838) INFO 05-09 16:21:12 [cuda.py:368] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(EngineCore pid=72838) INFO 05-09 16:21:17 [uva.py:58] Total CPU offloaded parameters: 6.24
(EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:904] Filesystem type for checkpoints: FUSE.MERGERFS. Checkpoint size: 22.78 GiB. Available RAM: 114.37 GiB.
(EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:927] Auto-prefetch is disabled because the filesystem (FUSE.MERGERFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:11<00:45, 11.38s/it]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:19<00:27,  9.31s/it]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:27<00:17,  8.72s/it]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:37<00:09,  9.39s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00,  7.64s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00,  8.44s/it]
(EngineCore pid=72838) 
(EngineCore pid=72838) INFO 05-09 16:22:01 [default_loader.py:384] Loading weights took 42.24 seconds
(EngineCore pid=72838) INFO 05-09 16:22:08 [gpu_model_runner.py:4879] Model loading took 15.38 GiB memory and 57.027123 seconds
(EngineCore pid=72838) INFO 05-09 16:22:08 [interface.py:606] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1069] Using cache directory: /home/z/.cache/vllm/torch_compile_cache/3f9e87c5e2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1128] Dynamo bytecode transform time: 8.01 s
(EngineCore pid=72838) INFO 05-09 16:22:18 [backends.py:376] Cache the graph of compile range (1, 2048) for later use
(EngineCore pid=72838) INFO 05-09 16:22:48 [backends.py:391] Compiling a graph for compile range (1, 2048) takes 31.27 s
(EngineCore pid=72838) INFO 05-09 16:22:51 [decorators.py:668] saved AOT compiled function to /home/z/.cache/vllm/torch_compile_cache/torch_aot_compile/90a747e06d0e8c397ccb2c580c362fed44da87f2a8c85e0b79d323e7dd81626c/rank_0_0/model
(EngineCore pid=72838) INFO 05-09 16:22:51 [monitor.py:53] torch.compile took 43.17 s in total
(EngineCore pid=72838) INFO 05-09 16:23:57 [monitor.py:81] Initial profiling/warmup run took 65.63 s
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] EngineCore failed to start.
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] Traceback (most recent call last):
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     super().__init__(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     self._init_minimal_kv_cache_for_profiling()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     self.initialize_kv_cache(minimal_config, is_profiling=True)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_caches = self.initialize_kv_cache_tensors(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     tensor = torch.zeros(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]              ^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
(EngineCore pid=72838) Process EngineCore:
(EngineCore pid=72838) Traceback (most recent call last):
(EngineCore pid=72838)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=72838)     self.run()
(EngineCore pid=72838)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=72838)     self._target(*self._args, **self._kwargs)
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1140, in run_engine_core
(EngineCore pid=72838)     raise e
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=72838)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=72838)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=72838)     super().__init__(
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=72838)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=72838)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=72838)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=72838)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=72838)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=72838)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=72838)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory
(EngineCore pid=72838)     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=72838)                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory
(EngineCore pid=72838)     self._init_minimal_kv_cache_for_profiling()
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling
(EngineCore pid=72838)     self.initialize_kv_cache(minimal_config, is_profiling=True)
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache
(EngineCore pid=72838)     kv_caches = self.initialize_kv_cache_tensors(
(EngineCore pid=72838)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors
(EngineCore pid=72838)     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(EngineCore pid=72838)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors
(EngineCore pid=72838)     tensor = torch.zeros(
(EngineCore pid=72838)              ^^^^^^^^^^^^
(EngineCore pid=72838) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
[rank0]:[W509 16:24:06.962826627 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=72781) Traceback (most recent call last):
(APIServer pid=72781)   File "/home/z/vl/.venv/bin/vllm", line 10, in <module>
(APIServer pid=72781)     sys.exit(main())
(APIServer pid=72781)              ^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=72781)     args.dispatch_function(args)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=72781)     uvloop.run(run_server(args))
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=72781)     return __asyncio.run(
(APIServer pid=72781)            ^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=72781)     return runner.run(main)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=72781)     return self._loop.run_until_complete(task)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=72781)     return await main
(APIServer pid=72781)            ^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 678, in run_server
(APIServer pid=72781)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server_worker
(APIServer pid=72781)     async with build_async_engine_client(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=72781)     return await anext(self.gen)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=72781)     async with build_async_engine_client_from_engine_args(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=72781)     return await anext(self.gen)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=72781)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=72781)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=72781)     return cls(
(APIServer pid=72781)            ^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=72781)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=72781)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=72781)     return func(*args, **kwargs)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=72781)     return AsyncMPClient(*client_args)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=72781)     return func(*args, **kwargs)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 900, in __init__
(APIServer pid=72781)     super().__init__(
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=72781)     with launch_core_engines(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=72781)     next(self.gen)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1119, in launch_core_engines
(APIServer pid=72781)     wait_for_engine_startup(
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1178, in wait_for_engine_startup
(APIServer pid=72781)     raise RuntimeError(
(APIServer pid=72781) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

RAW_BUFFERClick to expand / collapse

Your current environment

uv is set
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : version 3.28.3
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.17.0-23-generic-x86_64-with-glibc2.39
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.2.78
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version        : 595.58.03
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_tensor_ir.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.22.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.22.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  12
On-line CPU(s) list:                     0-11
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Core(TM) i7-6850K CPU @ 3.60GHz
CPU family:                              6
Model:                                   79
Thread(s) per core:                      2
Core(s) per socket:                      6
Socket(s):                               1
Stepping:                                1
CPU(s) scaling MHz:                      70%
CPU max MHz:                             4000.0000
CPU min MHz:                             1200.0000
BogoMIPS:                                7199.15
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 pti intel_ppin ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
L1d cache:                               192 KiB (6 instances)
L1i cache:                               192 KiB (6 instances)
L2 cache:                                1.5 MiB (6 instances)
L3 cache:                                15 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-11
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             KVM: Mitigation: VMX unsupported
Vulnerability L1tf:                      Mitigation; PTE Inversion
Vulnerability Mds:                       Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:                  Mitigation; PTI
Vulnerability Mmio stale data:           Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.0
[pip3] nvidia-cutlass-dsl-libs-base==4.5.0
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] torch==2.11.0
[pip3] torch-c-dlpack-ext==0.1.5
[pip3] torchaudio==2.11.0
[pip3] torchvision==0.26.0
[pip3] transformers==5.8.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.20.1
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
  	GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-11	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
MAX_JOBS=6
LD_LIBRARY_PATH=:/opt/MVS/lib/64:/opt/MVS/lib/32:/home/z/halcon/lib/x64-linux:/usr/local/cuda-13.2/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_z

🐛 Describe the bug

I'm trying to run cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit with vllm on 4090, but no matter what setting I set, it will always go out of memory even if I enabled huge cpu offload.

(z3) z@z3:~$ VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 vllm serve cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit --gpu-memory-utilization 0.6 --kv-cache-dtype fp8 --max-model-len 65536 --enable-prefix-caching --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --language-model-only --cpu-offload-gb 6
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] 
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.20.1
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]   █▄█▀ █     █     █     █  model   cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:299] 
(APIServer pid=72781) INFO 05-09 16:20:47 [utils.py:233] non-default args: {'model_tag': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': 'cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', 'max_model_len': 65536, 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.6, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'cpu_offload_gb': 6.0, 'language_model_only': True}
(APIServer pid=72781) INFO 05-09 16:20:50 [model.py:555] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=72781) INFO 05-09 16:20:50 [model.py:1680] Using max model len 65536
(APIServer pid=72781) INFO 05-09 16:20:51 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL.
(APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:34] NIXL is not available
(APIServer pid=72781) WARNING 05-09 16:20:51 [nixl_utils.py:44] NIXL agent config is not available
(APIServer pid=72781) INFO 05-09 16:20:51 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=72781) WARNING 05-09 16:20:51 [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=72781) INFO 05-09 16:20:51 [config.py:387] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=72781) INFO 05-09 16:20:51 [vllm.py:840] Asynchronous scheduling is enabled.
(APIServer pid=72781) INFO 05-09 16:20:51 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=72781) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=72781) INFO 05-09 16:20:56 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
WARNING 05-09 16:21:04 [nixl_utils.py:34] NIXL is not available
WARNING 05-09 16:21:04 [nixl_utils.py:44] NIXL agent config is not available
(EngineCore pid=72838) INFO 05-09 16:21:04 [core.py:109] Initializing a V1 LLM engine (v0.20.1) with config: model='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', speculative_config=None, tokenizer='cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=72838) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=72838) INFO 05-09 16:21:10 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1402] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://139.24.23.9:48235 backend=nccl
(EngineCore pid=72838) INFO 05-09 16:21:10 [parallel_state.py:1715] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=72838) INFO 05-09 16:21:10 [base.py:123] Offloader set to UVAOffloader
(EngineCore pid=72838) INFO 05-09 16:21:10 [gpu_model_runner.py:4777] Starting to load model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit...
(EngineCore pid=72838) INFO 05-09 16:21:11 [cuda.py:423] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=72838) INFO 05-09 16:21:11 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=72838) INFO 05-09 16:21:11 [gdn_linear_attn.py:153] Using Triton/FLA GDN prefill kernel
(EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe.py:122] Using CompressedTensorsWNA16MarlinMoEMethod
(EngineCore pid=72838) INFO 05-09 16:21:11 [compressed_tensors_moe_wna16_marlin.py:87] Using Marlin backend for WNA16 MoE (group_size=32, num_bits=4)
(EngineCore pid=72838) INFO 05-09 16:21:12 [cuda.py:368] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(EngineCore pid=72838) INFO 05-09 16:21:17 [uva.py:58] Total CPU offloaded parameters: 6.24
(EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:904] Filesystem type for checkpoints: FUSE.MERGERFS. Checkpoint size: 22.78 GiB. Available RAM: 114.37 GiB.
(EngineCore pid=72838) INFO 05-09 16:21:19 [weight_utils.py:927] Auto-prefetch is disabled because the filesystem (FUSE.MERGERFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:11<00:45, 11.38s/it]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:19<00:27,  9.31s/it]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:27<00:17,  8.72s/it]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:37<00:09,  9.39s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00,  7.64s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:42<00:00,  8.44s/it]
(EngineCore pid=72838) 
(EngineCore pid=72838) INFO 05-09 16:22:01 [default_loader.py:384] Loading weights took 42.24 seconds
(EngineCore pid=72838) INFO 05-09 16:22:08 [gpu_model_runner.py:4879] Model loading took 15.38 GiB memory and 57.027123 seconds
(EngineCore pid=72838) INFO 05-09 16:22:08 [interface.py:606] Setting attention block size to 2096 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1069] Using cache directory: /home/z/.cache/vllm/torch_compile_cache/3f9e87c5e2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=72838) INFO 05-09 16:22:16 [backends.py:1128] Dynamo bytecode transform time: 8.01 s
(EngineCore pid=72838) INFO 05-09 16:22:18 [backends.py:376] Cache the graph of compile range (1, 2048) for later use
(EngineCore pid=72838) INFO 05-09 16:22:48 [backends.py:391] Compiling a graph for compile range (1, 2048) takes 31.27 s
(EngineCore pid=72838) INFO 05-09 16:22:51 [decorators.py:668] saved AOT compiled function to /home/z/.cache/vllm/torch_compile_cache/torch_aot_compile/90a747e06d0e8c397ccb2c580c362fed44da87f2a8c85e0b79d323e7dd81626c/rank_0_0/model
(EngineCore pid=72838) INFO 05-09 16:22:51 [monitor.py:53] torch.compile took 43.17 s in total
(EngineCore pid=72838) INFO 05-09 16:23:57 [monitor.py:81] Initial profiling/warmup run took 65.63 s
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] EngineCore failed to start.
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] Traceback (most recent call last):
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     super().__init__(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     self._init_minimal_kv_cache_for_profiling()
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     self.initialize_kv_cache(minimal_config, is_profiling=True)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_caches = self.initialize_kv_cache_tensors(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]     tensor = torch.zeros(
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136]              ^^^^^^^^^^^^
(EngineCore pid=72838) ERROR 05-09 16:24:05 [core.py:1136] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
(EngineCore pid=72838) Process EngineCore:
(EngineCore pid=72838) Traceback (most recent call last):
(EngineCore pid=72838)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=72838)     self.run()
(EngineCore pid=72838)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=72838)     self._target(*self._args, **self._kwargs)
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1140, in run_engine_core
(EngineCore pid=72838)     raise e
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=72838)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=72838)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=72838)     super().__init__(
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=72838)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=72838)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=72838)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=72838)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=72838)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore pid=72838)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=72838)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 385, in determine_available_memory
(EngineCore pid=72838)     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(EngineCore pid=72838)                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=72838)     return func(*args, **kwargs)
(EngineCore pid=72838)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5951, in profile_cudagraph_memory
(EngineCore pid=72838)     self._init_minimal_kv_cache_for_profiling()
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5881, in _init_minimal_kv_cache_for_profiling
(EngineCore pid=72838)     self.initialize_kv_cache(minimal_config, is_profiling=True)
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6842, in initialize_kv_cache
(EngineCore pid=72838)     kv_caches = self.initialize_kv_cache_tensors(
(EngineCore pid=72838)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6756, in initialize_kv_cache_tensors
(EngineCore pid=72838)     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(EngineCore pid=72838)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=72838)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6537, in _allocate_kv_cache_tensors
(EngineCore pid=72838)     tensor = torch.zeros(
(EngineCore pid=72838)              ^^^^^^^^^^^^
(EngineCore pid=72838) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.02 GiB. GPU 0 has a total capacity of 23.52 GiB of which 115.69 MiB is free. Including non-PyTorch memory, this process has 23.37 GiB memory in use. Of the allocated memory 22.69 GiB is allocated by PyTorch, and 209.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
[rank0]:[W509 16:24:06.962826627 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=72781) Traceback (most recent call last):
(APIServer pid=72781)   File "/home/z/vl/.venv/bin/vllm", line 10, in <module>
(APIServer pid=72781)     sys.exit(main())
(APIServer pid=72781)              ^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=72781)     args.dispatch_function(args)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=72781)     uvloop.run(run_server(args))
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=72781)     return __asyncio.run(
(APIServer pid=72781)            ^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=72781)     return runner.run(main)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=72781)     return self._loop.run_until_complete(task)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=72781)     return await main
(APIServer pid=72781)            ^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 678, in run_server
(APIServer pid=72781)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server_worker
(APIServer pid=72781)     async with build_async_engine_client(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=72781)     return await anext(self.gen)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=72781)     async with build_async_engine_client_from_engine_args(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=72781)     return await anext(self.gen)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=72781)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=72781)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=72781)     return cls(
(APIServer pid=72781)            ^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=72781)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=72781)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=72781)     return func(*args, **kwargs)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=72781)     return AsyncMPClient(*client_args)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=72781)     return func(*args, **kwargs)
(APIServer pid=72781)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 900, in __init__
(APIServer pid=72781)     super().__init__(
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=72781)     with launch_core_engines(
(APIServer pid=72781)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=72781)     next(self.gen)
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1119, in launch_core_engines
(APIServer pid=72781)     wait_for_engine_startup(
(APIServer pid=72781)   File "/home/z/vl/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1178, in wait_for_engine_startup
(APIServer pid=72781)     raise RuntimeError(
(APIServer pid=72781) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The error is similar for --gpu-memory-utilization 0.95, both with or without offloading. Technically since the model itself without visual part should be ~21G there should be some space left, therefore offloading shouldn't even be neceesary?

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #memory management #model loading #environment variable #memory leak

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - 💡(How to fix) Fix [Bug]: Qwen 3.6 awq can't load, always OOM error

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

Still need to ship something?

TRENDING

vllm - 💡(How to fix) Fix [Bug]: Qwen 3.6 awq can't load, always OOM error

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

Still need to ship something?

RELATED_DISCOVERY

TRENDING