vllm - ✅(Solved) Fix [Bug]: does not have the attribute 'FakeTensorMode' [3 pull requests, 6 comments, 3 participants]

vllm2026-03-23 06:58:28

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

vllm-project/vllm#37858•Fetched 2026-04-08 01:17:29

View on GitHub

Comments

Participants

Timeline

Reactions

Author

Participants

Timeline (top)

commented ×6subscribed ×3cross-referenced ×2labeled ×1

Error Message

(APIServer pid=1008) WARNING 03-23 14:50:59 [system_utils.py:289] Found ulimit of 1024 and failed to automatically increase with error current limit exceeds maximum limit. This can cause fd limit errors like OSError: [Errno 24] Too many open files. Consider increasing with ulimit -n (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] WorkerProc hit an exception. (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] Traceback (most recent call last): (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 927, in worker_busy_loop (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] output = func(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 388, in determine_available_memory (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.model_runner.profile_run() (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] hidden_states, last_hidden_states = self._dummy_run( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5210, in _dummy_run (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] outputs = self.model( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/cuda_graph.py", line 251, in call (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self.runnable(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self._call_impl(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return forward_call(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_5.py", line 765, in forward (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] hidden_states = self.language_model.model( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 583, in call (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.aot_compiled_fn = self.aot_compile(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/wrapper.py", line 168, in aot_compile (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self._compiled_callable.aot_compile((args, kwargs)) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in aot_compile (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return aot_compile_fullgraph( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/_dynamo/aot_compile.py", line 239, in aot_compile_fullgraph

Root Cause

root@b20654c00b62:/app# python -m vllm.entrypoints.openai.api_server \

--model=/app/model \
--block-size=32 \
--max-model-len=120000 \
--gpu-memory-utilization 0.85 \
--served-model-name=Qwen3.5-27B-FP8 \
--port=10105 \
--host=0.0.0.0 \
--tensor-parallel-size=2 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--api-key=K7mP9xQ2vR4n

(APIServer pid=1008) INFO 03-23 14:50:59 [utils.py:297] (APIServer pid=1008) INFO 03-23 14:50:59 [utils.py:297] █ █ █▄ ▄█ (APIServer pid=1008) INFO 03-23 14:50:59 [utils.py:297] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.18.0 (APIServer pid=1008) INFO 03-23 14:50:59 [utils.py:297] █▄█▀ █ █ █ █ model /app/model (APIServer pid=1008) INFO 03-23 14:50:59 [utils.py:297] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1008) INFO 03-23 14:50:59 [utils.py:297] (APIServer pid=1008) INFO 03-23 14:50:59 [utils.py:233] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'hermes', 'host': '0.0.0.0', 'port': 10105, 'api_key': ['K7mP9xQ2vR4n'], 'model': '/app/model', 'max_model_len': 120000, 'served_model_name': ['Qwen3.5-27B-FP8'], 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'block_size': 32, 'gpu_memory_utilization': 0.85} (APIServer pid=1008) WARNING 03-23 14:50:59 [system_utils.py:289] Found ulimit of 1024 and failed to automatically increase with error current limit exceeds maximum limit. This can cause fd limit errors like OSError: [Errno 24] Too many open files. Consider increasing with ulimit -n (APIServer pid=1008) INFO 03-23 14:50:59 [model.py:533] Resolved architecture: Qwen3_5ForConditionalGeneration (APIServer pid=1008) INFO 03-23 14:50:59 [model.py:1582] Using max model len 120000 (APIServer pid=1008) INFO 03-23 14:51:01 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048. (APIServer pid=1008) INFO 03-23 14:51:01 [config.py:212] Setting attention block size to 784 tokens to ensure that attention page size is >= mamba page size. (APIServer pid=1008) INFO 03-23 14:51:01 [config.py:243] Padding mamba page size by 0.13% to ensure that mamba page size and attention page size are exactly equal. (APIServer pid=1008) INFO 03-23 14:51:01 [vllm.py:754] Asynchronous scheduling is enabled. (APIServer pid=1008) INFO 03-23 14:51:01 [compilation.py:289] Enabled custom fusions: norm_quant, act_quant (EngineCore pid=1179) INFO 03-23 14:51:16 [core.py:103] Initializing a V1 LLM engine (v0.18.0) with config: model='/app/model', speculative_config=None, tokenizer='/app/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=120000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.5-27B-FP8, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (EngineCore pid=1179) WARNING 03-23 14:51:16 [multiproc_executor.py:997] Reducing Torch parallelism from 48 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. (EngineCore pid=1179) INFO 03-23 14:51:16 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.17.0.6 (local), world_size=2, local_world_size=2 (Worker pid=1346) INFO 03-23 14:51:25 [parallel_state.py:1395] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:36235 backend=nccl (Worker pid=1345) INFO 03-23 14:51:25 [parallel_state.py:1395] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:36235 backend=nccl (Worker pid=1345) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. (Worker pid=1345) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. (Worker pid=1345) INFO 03-23 14:51:26 [pynccl.py:111] vLLM is using nccl==2.27.5 (Worker pid=1346) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. (Worker pid=1346) <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. (Worker pid=1345) WARNING 03-23 14:51:26 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available. (Worker pid=1346) WARNING 03-23 14:51:26 [symm_mem.py:67] SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available. (Worker pid=1345) INFO 03-23 14:51:26 [parallel_state.py:1717] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A (Worker pid=1346) INFO 03-23 14:51:26 [parallel_state.py:1717] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A, EPLB rank N/A (Worker_TP0 pid=1345) INFO 03-23 14:51:32 [gpu_model_runner.py:4481] Starting to load model /app/model... (Worker_TP1 pid=1346) INFO 03-23 14:51:32 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention (Worker_TP1 pid=1346) INFO 03-23 14:51:32 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (Worker_TP1 pid=1346) INFO 03-23 14:51:33 [qwen3_next.py:198] Using Triton/FLA GDN prefill kernel (Worker_TP0 pid=1345) INFO 03-23 14:51:33 [cuda.py:373] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention (Worker_TP0 pid=1345) INFO 03-23 14:51:33 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (Worker_TP0 pid=1345) INFO 03-23 14:51:33 [qwen3_next.py:198] Using Triton/FLA GDN prefill kernel (Worker_TP0 pid=1345) INFO 03-23 14:51:33 [cuda.py:317] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (Worker_TP0 pid=1345) INFO 03-23 14:51:33 [flash_attn.py:598] Using FlashAttention version 2 Loading safetensors checkpoint shards: 0% Completed | 0/11 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 9% Completed | 1/11 [00:00<00:09, 1.10it/s] Loading safetensors checkpoint shards: 18% Completed | 2/11 [00:01<00:07, 1.26it/s] Loading safetensors checkpoint shards: 27% Completed | 3/11 [00:02<00:06, 1.28it/s] Loading safetensors checkpoint shards: 36% Completed | 4/11 [00:03<00:05, 1.27it/s] Loading safetensors checkpoint shards: 45% Completed | 5/11 [00:03<00:04, 1.26it/s] Loading safetensors checkpoint shards: 55% Completed | 6/11 [00:04<00:03, 1.25it/s] Loading safetensors checkpoint shards: 64% Completed | 7/11 [00:05<00:03, 1.27it/s] Loading safetensors checkpoint shards: 73% Completed | 8/11 [00:06<00:02, 1.37it/s] Loading safetensors checkpoint shards: 82% Completed | 9/11 [00:06<00:01, 1.45it/s] (Worker_TP1 pid=1346) WARNING 03-23 14:51:40 [marlin_utils_fp8.py:97] Your GPU does not have native support for FP8 computation but FP8 quantization is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads. Loading safetensors checkpoint shards: 91% Completed | 10/11 [00:07<00:00, 1.39it/s] Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:08<00:00, 1.52it/s] Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:08<00:00, 1.36it/s] (Worker_TP0 pid=1345) (Worker_TP0 pid=1345) INFO 03-23 14:51:41 [default_loader.py:384] Loading weights took 8.12 seconds (Worker_TP0 pid=1345) WARNING 03-23 14:51:41 [marlin_utils_fp8.py:97] Your GPU does not have native support for FP8 computation but FP8 quantization is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads. (Worker_TP0 pid=1345) INFO 03-23 14:51:42 [gpu_model_runner.py:4566] Model loading took 14.61 GiB memory and 8.975880 seconds (Worker_TP0 pid=1345) INFO 03-23 14:51:42 [gpu_model_runner.py:5488] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. (Worker_TP1 pid=1346) INFO 03-23 14:51:42 [gpu_model_runner.py:5488] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. (Worker_TP0 pid=1345) INFO 03-23 14:52:00 [backends.py:988] Using cache directory: /root/.cache/vllm/torch_compile_cache/7efd4602e0/rank_0_0/backbone for vLLM's torch.compile (Worker_TP0 pid=1345) INFO 03-23 14:52:00 [backends.py:1048] Dynamo bytecode transform time: 15.55 s (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] WorkerProc hit an exception. (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] Traceback (most recent call last): (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 927, in worker_busy_loop (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] output = func(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 388, in determine_available_memory (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.model_runner.profile_run() (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] hidden_states, last_hidden_states = self._dummy_run( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5210, in _dummy_run (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] outputs = self.model( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/cuda_graph.py", line 251, in call (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self.runnable(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self._call_impl(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return forward_call(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_5.py", line 765, in forward (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] hidden_states = self.language_model.model( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 583, in call (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.aot_compiled_fn = self.aot_compile(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/wrapper.py", line 168, in aot_compile (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self._compiled_callable.aot_compile((args, kwargs)) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/dynamo/eval_frame.py", line 832, in aot_compile (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return aot_compile_fullgraph( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/dynamo/aot_compile.py", line 239, in aot_compile_fullgraph (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] compiled_fn = backend( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/init.py", line 2514, in call (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self.compiler_fn(model, inputs, **self.kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/contextlib.py", line 79, in inner (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwds) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/backends.py", line 1114, in call (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] PiecewiseCompileInterpreter( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/backends.py", line 640, in run (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return super().run(*args) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/fx/interpreter.py", line 200, in run (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.env[node] = self.run_node(node) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/fx/interpreter.py", line 295, in run_node (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return getattr(self, n.op)(n.target, args, kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/backends.py", line 667, in call_module (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] piecewise_backend = PiecewiseBackend( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/piecewise_backend.py", line 189, in init (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.compile_all_ranges() (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/piecewise_backend.py", line 265, in compile_all_ranges (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] range_entry.runnable = self.vllm_backend.compiler_manager.compile( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/backends.py", line 346, in compile (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] compiled_graph, handle = self.compiler.compile( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/compiler_interface.py", line 383, in compile (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] with pregrad_ctx, fake_mode_ctx, _patch_constrain_to_fx_strides(): (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/unittest/mock.py", line 1447, in enter (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] original, local = self.get_original() (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/unittest/mock.py", line 1420, in get_original (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] raise AttributeError( (Worker_TP1 pid=1346) ERROR 03-23 14:52:02 [multiproc_executor.py:932] AttributeError: <function standalone_compile at 0x7f508f73e8c0> does not have the attribute 'FakeTensorMode' (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] WorkerProc hit an exception. (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] Traceback (most recent call last): (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 927, in worker_busy_loop (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] output = func(*args, **kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 388, in determine_available_memory (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.model_runner.profile_run() (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] hidden_states, last_hidden_states = self._dummy_run( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5210, in _dummy_run (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] outputs = self.model( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/cuda_graph.py", line 251, in call (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self.runnable(*args, **kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self._call_impl(*args, **kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return forward_call(*args, **kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_5.py", line 765, in forward (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] hidden_states = self.language_model.model( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 583, in call (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.aot_compiled_fn = self.aot_compile(*args, **kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/wrapper.py", line 168, in aot_compile (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self._compiled_callable.aot_compile((args, kwargs)) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/dynamo/eval_frame.py", line 832, in aot_compile (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return aot_compile_fullgraph( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/dynamo/aot_compile.py", line 239, in aot_compile_fullgraph (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] compiled_fn = backend( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/init.py", line 2514, in call (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return self.compiler_fn(model, inputs, **self.kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/contextlib.py", line 79, in inner (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwds) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/backends.py", line 1114, in call (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] PiecewiseCompileInterpreter( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/backends.py", line 640, in run (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return super().run(*args) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/fx/interpreter.py", line 200, in run (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.env[node] = self.run_node(node) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/torch/fx/interpreter.py", line 295, in run_node (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return getattr(self, n.op)(n.target, args, kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/backends.py", line 667, in call_module (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] piecewise_backend = PiecewiseBackend( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/piecewise_backend.py", line 189, in init (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] self.compile_all_ranges() (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/piecewise_backend.py", line 265, in compile_all_ranges (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] range_entry.runnable = self.vllm_backend.compiler_manager.compile( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] return func(*args, **kwargs) (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/backends.py", line 346, in compile (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] compiled_graph, handle = self.compiler.compile( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/site-packages/vllm/compilation/compiler_interface.py", line 383, in compile (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] with pregrad_ctx, fake_mode_ctx, _patch_constrain_to_fx_strides(): (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/unittest/mock.py", line 1447, in enter (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] original, local = self.get_original() (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] File "/usr/local/lib/python3.10/unittest/mock.py", line 1420, in get_original (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] raise AttributeError( (Worker_TP0 pid=1345) ERROR 03-23 14:52:02 [multiproc_executor.py:932] AttributeError: <function standalone_compile at 0x7faca423a8c0> does not have the attribute 'FakeTensorMode' (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] EngineCore failed to start. (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] Traceback (most recent call last): (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1073, in run_engine_core (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] return func(*args, **kwargs) (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 839, in init (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] super().init( (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 122, in init (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] kv_cache_config = self._initialize_kv_caches(vllm_config) (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] return func(*args, **kwargs) (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] available_gpu_memory = self.model_executor.determine_available_memory() (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] File "/usr/local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 136, in determine_available_memory (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] return self.collective_rpc("determine_available_memory") (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] File "/usr/local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 397, in collective_rpc (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] return aggregate(get_response()) (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] File "/usr/local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 380, in get_response (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] raise RuntimeError( (EngineCore pid=1179) ERROR 03-23 14:52:02 [core.py:1099] RuntimeError: Worker failed with error '<function standalone_compile at 0x7faca423a8c0> does not have the attribute 'FakeTensorMode'', please check the stack trace above for the root cause (Worker_TP0 pid=1345) WARNING 03-23 14:52:02 [multiproc_executor.py:866] WorkerProc was terminated (Worker_TP1 pid=1346) WARNING 03-23 14:52:02 [multiproc_executor.py:866] WorkerProc was terminated (EngineCore pid=1179) ERROR 03-23 14:52:05 [multiproc_executor.py:273] Worker proc VllmWorker-0 died unexpectedly, shutting down executor. (EngineCore pid=1179) Process EngineCore: (EngineCore pid=1179) Traceback (most recent call last): (EngineCore pid=1179) File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap (EngineCore pid=1179) self.run() (EngineCore pid=1179) File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run (EngineCore pid=1179) self._target(*self._args, **self._kwargs) (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1103, in run_engine_core (EngineCore pid=1179) raise e (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1073, in run_engine_core (EngineCore pid=1179) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=1179) return func(*args, **kwargs) (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 839, in init (EngineCore pid=1179) super().init( (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 122, in init (EngineCore pid=1179) kv_cache_config = self._initialize_kv_caches(vllm_config) (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=1179) return func(*args, **kwargs) (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches (EngineCore pid=1179) available_gpu_memory = self.model_executor.determine_available_memory() (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 136, in determine_available_memory (EngineCore pid=1179) return self.collective_rpc("determine_available_memory") (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 397, in collective_rpc (EngineCore pid=1179) return aggregate(get_response()) (EngineCore pid=1179) File "/usr/local/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 380, in get_response (EngineCore pid=1179) raise RuntimeError( (EngineCore pid=1179) RuntimeError: Worker failed with error '<function standalone_compile at 0x7faca423a8c0> does not have the attribute 'FakeTensorMode'', please check the stack trace above for the root cause (APIServer pid=1008) Traceback (most recent call last): (APIServer pid=1008) File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main (APIServer pid=1008) return _run_code(code, main_globals, None, (APIServer pid=1008) File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code (APIServer pid=1008) exec(code, run_globals) (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 696, in <module> (APIServer pid=1008) uvloop.run(run_server(args)) (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/uvloop/init.py", line 69, in run (APIServer pid=1008) return loop.run_until_complete(wrapper()) (APIServer pid=1008) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/uvloop/init.py", line 48, in wrapper (APIServer pid=1008) return await main (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 656, in run_server (APIServer pid=1008) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 670, in run_server_worker (APIServer pid=1008) async with build_async_engine_client( (APIServer pid=1008) File "/usr/local/lib/python3.10/contextlib.py", line 199, in aenter (APIServer pid=1008) return await anext(self.gen) (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 103, in build_async_engine_client (APIServer pid=1008) async with build_async_engine_client_from_engine_args( (APIServer pid=1008) File "/usr/local/lib/python3.10/contextlib.py", line 199, in aenter (APIServer pid=1008) return await anext(self.gen) (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 144, in build_async_engine_client_from_engine_args (APIServer pid=1008) async_llm = AsyncLLM.from_vllm_config( (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config (APIServer pid=1008) return cls( (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 154, in init (APIServer pid=1008) self.engine_core = EngineCoreClient.make_async_mp_client( (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=1008) return func(*args, **kwargs) (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client (APIServer pid=1008) return AsyncMPClient(*client_args) (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=1008) return func(*args, **kwargs) (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 924, in init (APIServer pid=1008) super().init( (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 583, in init (APIServer pid=1008) with launch_core_engines( (APIServer pid=1008) File "/usr/local/lib/python3.10/contextlib.py", line 142, in exit (APIServer pid=1008) next(self.gen) (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 972, in launch_core_engines (APIServer pid=1008) wait_for_engine_startup( (APIServer pid=1008) File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup (APIServer pid=1008) raise RuntimeError( (APIServer pid=1008) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} /usr/local/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 2 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' root@b20654c00b62:/app#

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 96 On-line CPU(s) list: 0-95 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 2 Stepping: 6 BogoMIPS: 5600.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 invpcid_single ssbd mba rsb_ctxsw ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp_epp avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq md_clear pconfig spec_ctrl intel_stibp flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 2.3 MiB (48 instances) L1i cache: 1.5 MiB (48 instances) L2 cache: 60 MiB (48 instances) L3 cache: 72 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-23,48-71 NUMA node1 CPU(s): 24-47,72-95 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; Load fences, usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

root@b20654c00b62:/app# python -m vllm.entrypoints.openai.api_server \

--model=/app/model \
--block-size=32 \
--max-model-len=120000 \
--gpu-memory-utilization 0.85 \
--served-model-name=Qwen3.5-27B-FP8 \
--port=10105 \
--host=0.0.0.0 \
--tensor-parallel-size=2 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--api-key=K7mP9xQ2vR4n

PR fix notes

PR #37866: fix: handle FakeTensorMode patching for PyTorch compatibility

Repository: vllm-project/vllm
Author: CMLKevin
State: open | merged: False
Link: https://github.com/vllm-project/vllm/pull/37866

Description (problem / solution / changelog)

Fixes #37858

The current patch assumes torch._inductor.standalone_compile resolves to the standalone_compile module and patches FakeTensorMode there. In the failing PyTorch/Python combination from the issue, that lookup ends up on the wrapper function instead, which raises AttributeError: <function standalone_compile ...> does not have the attribute 'FakeTensorMode' before compilation starts.

This switches the override to patch the imported function's globals instead, which is where FakeTensorMode(...) is actually resolved at runtime. I also added a small regression test covering the function-global patch path.

Verification here was limited to py_compile plus a direct smoke test of the helper because the repo test harness imports tblib, which is missing in this environment.

Changed files

tests/compile/test_aot_compile.py (modified, +27/-0)
vllm/compilation/compiler_interface.py (modified, +33/-10)

PR #2108: [Qwen3TTS] [TTS] [Feat] Refactor voice cache manager

Repository: vllm-project/vllm-omni
Author: JuanPZuluaga
State: closed | merged: True
Link: https://github.com/vllm-project/vllm-omni/pull/2108

Description (problem / solution / changelog)

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

we add an in-memory cache for voice extraction artifacts in Qwen3-TTS Base voice clone requests. The system is simple enough that can be used by other models (will import it to the other models in a following PR).

When a user sends a voice clone request with ref_audio, the server runs two GPU ops: speaker embedding extraction (ECAPA-TDNN) and ref_audio encoding (SpeechTokenizer). These cost 60-250ms per request. If the same voice is used again, these results are identical, but today they're recomputed every time.

This PR caches the extraction results keyed by a SHA-256 hash of the audio content. On repeat requests with the same reference audio, the cached tensors are served from CPU memory instead of re-running GPU extraction. This gives a ~25% TTFP reduction on warm (repeated voice) requests.

Files deleted

vllm_omni/entrypoints/openai/metadata_manager.py: disk persistence replaced by in-memory dict (will add later something more robust)
vllm_omni/model_executor/models/qwen3_tts/voice_cache_manager.py: replaced by a simpler voice_cache.py

Known Issue: torch.compile crash on startup (vllm 0.18.0 + torch 2.10)

When using enforce_eager: false (required for stage 0), the server crashes during startup with:

AttributeError: <function standalone_compile at ...> does not have the attribute 'FakeTensorMode'

Root cause: torch._inductor.__init__ defines a wrapper function standalone_compile that shadows the submodule of the same name. unittest.mock.patch() resolves via getattr and finds the function instead of the module.

Fix: Apply a one-line patch to vllm/compilation/compiler_interface.py (line ~376). This is fixed upstream in vllm-project/vllm#37158 but not included in vllm 0.18.0.

-            fake_mode_ctx: Any = patch(
-                "torch._inductor.standalone_compile.FakeTensorMode",
+            import sys
+            fake_mode_ctx: Any = patch.object(
+                sys.modules["torch._inductor.standalone_compile"],
+                "FakeTensorMode",
                 lambda *a, **kw: input_fake_mode,
             )

this was merge to main already with bug in: vllm-project/vllm#37858

(we might need to pin to a version, please someone test this as well)

Test Plan

1. Start the server

vllm-omni serve Qwen/Qwen3-TTS-12Hz-0.6B-Base \
    --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
    --omni \
    --port 8000 \
    --trust-remote-code

2. Upload a voice (one can pass "ref_text" as well)

curl -X POST http://localhost:8000/v1/audio/voices \
  -F "audio_sample=@/path/to/your/reference.wav" \
  -F "name=my_voice" \
  -F "consent=test_consent"

3. First request (warmup, it extracts speaker embedding, and then caches it)

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, this is a voice cloning test.", "voice": "my_voice", "response_format": "wav"}' \
  --output cold.wav

4. Second request (embedding already cached, skips extraction)

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "This request reuses the cached speaker embedding.", "voice": "my_voice", "response_format": "wav"}' \
   --output warm.wav

5. Batch endpoint (all items share the cached embedding)

curl -X POST http://localhost:8000/v1/audio/speech/batch \
  -H "Content-Type: application/json" \
  -d '{
    "voice": "my_voice",
    "items": [
      {"input": "First sentence in the batch."},
      {"input": "Second sentence, same cached voice."},
      {"input": "Third sentence, still one extraction."}
    ],
    "response_format": "wav"
  }'

6. Delete voice (cache key removed, next upload gets fresh extraction)

curl -X DELETE http://localhost:8000/v1/audio/voices/my_voice

Test Result

Metric                          main        voice-cache     delta
─────────────────────────────────────────────────────────────────
Cold e2e mean (1st req/voice)   738 ms      734 ms          -4 ms
Warm con=1  median              812 ms      799 ms          -13 ms
Warm con=1  P95                 933 ms      928 ms          -6 ms
Warm con=4  median              1480 ms     1455 ms         -25 ms
Warm con=4  P95                 1688 ms     1835 ms         +147 ms
Warm con=16 median              3722 ms     3666 ms         -56 ms
Warm con=16 P95                 4576 ms     4544 ms         -32 ms
Throughput  con=16              4.0 req/s   4.0 req/s       0.0
Errors                          0           0               0
─────────────────────────────────────────────────────────────────

<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
The test results. Please paste the results comparison before and after, or the e2e results.
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
(Optional) Release notes update. If your change is user-facing, please update the release notes draft.

</details>

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Changed files

docs/serving/speech_api.md (modified, +24/-3)
docs/user_guide/examples/online_serving/qwen3_tts.md (modified, +22/-12)
examples/online_serving/qwen3_tts/README.md (modified, +8/-2)
tests/entrypoints/openai_api/test_serving_speech.py (modified, +65/-13)
tests/test_voice_cache.py (added, +129/-0)
vllm_omni/entrypoints/openai/api_server.py (modified, +28/-13)
vllm_omni/entrypoints/openai/metadata_manager.py (removed, +0/-243)
vllm_omni/entrypoints/openai/serving_speech.py (modified, +41/-57)
vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py (modified, +36/-0)
vllm_omni/model_executor/models/qwen3_tts/voice_cache_manager.py (removed, +0/-271)
vllm_omni/utils/voice_cache.py (added, +89/-0)

RAW_BUFFERClick to expand / collapse

Your current environment

root@b20654c00b62:/app/model# python collect_env.py Collecting environment information...

    System Info

============================== OS : Debian GNU/Linux 12 (bookworm) (x86_64) GCC version : (Debian 12.2.0-14+deb12u1) 12.2.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.36

============================== PyTorch Info

PyTorch version : 2.10.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A

============================== Python Environment

Python version : 3.10.19 (main, Oct 9 2025, 22:46:37) [GCC 12.2.0] (64-bit runtime) Python platform : Linux-3.10.0-1160.119.1.el7.x86_64-x86_64-with-glibc2.36

============================== CUDA / GPU Info

Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA A10 GPU 1: NVIDIA A10

Nvidia driver version : 550.90.12 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True

============================== CPU Info

============================== Versions of relevant libraries

[pip3] flashinfer-python==0.6.6 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.8.4.1 [pip3] nvidia-cuda-cupti-cu12==12.8.90 [pip3] nvidia-cuda-nvrtc-cu12==12.8.93 [pip3] nvidia-cuda-runtime-cu12==12.8.90 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.18.0 [pip3] nvidia-cufft-cu12==11.3.3.83 [pip3] nvidia-cufile-cu12==1.13.1.3 [pip3] nvidia-curand-cu12==10.3.9.90 [pip3] nvidia-cusolver-cu12==11.7.3.90 [pip3] nvidia-cusparse-cu12==12.5.8.93 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-cutlass-dsl==4.4.2 [pip3] nvidia-cutlass-dsl-libs-base==4.4.2 [pip3] nvidia-ml-py==13.595.45 [pip3] nvidia-nccl-cu12==2.27.5 [pip3] nvidia-nvjitlink-cu12==12.8.93 [pip3] nvidia-nvshmem-cu12==3.4.5 [pip3] nvidia-nvtx-cu12==12.8.90 [pip3] pyzmq==27.1.0 [pip3] torch==2.10.0 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.10.0 [pip3] torchvision==0.25.0 [pip3] transformers==4.57.6 [pip3] triton==3.6.0 [conda] Could not collect

============================== vLLM Info

ROCM Version : Could not collect vLLM Version : 0.18.0 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NODE 0-23,48-71 0 N/A GPU1 NODE X 0-23,48-71 0 N/A

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

============================== Environment Variables

NVIDIA_VISIBLE_DEVICES=1,3 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

🐛 Describe the bug

root@b20654c00b62:/app# python -m vllm.entrypoints.openai.api_server \

--model=/app/model \
--block-size=32 \
--max-model-len=120000 \
--gpu-memory-utilization 0.85 \
--served-model-name=Qwen3.5-27B-FP8 \
--port=10105 \
--host=0.0.0.0 \
--tensor-parallel-size=2 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--api-key=K7mP9xQ2vR4n

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

The error message indicates that the function standalone_compile does not have the attribute FakeTensorMode. This suggests a compatibility issue between the versions of PyTorch and the vllm library.

To fix this issue, you can try the following steps:

Update PyTorch: Make sure you are using the latest version of PyTorch. You can update PyTorch using pip: pip install --upgrade torch
Update vllm library: Try updating the vllm library to the latest version. You can do this by running pip install --upgrade vllm
Check compatibility: If updating the libraries does not resolve the issue, check the compatibility of the vllm library with your version of PyTorch. You can do this by checking the documentation of the vllm library or by searching for similar issues on GitHub.

Here is an example of how you can update PyTorch and the vllm library:

pip install --upgrade torch
pip install --upgrade vllm

If you are using a virtual environment, make sure to activate it before running the update commands.

Verification

To verify that the fix worked, you can try running the command that caused the error again:

python -m vllm.entrypoints.openai.api_server \
    --model=/app/model \
    --block-size=32 \
    --max-model-len=120000 \
    --gpu-memory-utilization 0.85 \
    --served-model-name=Qwen3.5-27B-FP8 \
    --port=10105 \
    --host=0.0.0.0 \
    --tensor-parallel-size=2 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --api-key=K7mP9xQ2vR4n

If the fix was successful, the command should run without errors.

Extra Tips

Make sure to check the documentation of the vllm library for any specific requirements or compatibility issues.
If you are using a virtual environment, make sure to activate it before running the update commands.
If you are still experiencing issues after updating the libraries, try searching for similar issues on GitHub or seeking help from the vllm community.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #model loading #environment variable #memory leak #API versioning

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

vllm - ✅(Solved) Fix [Bug]: does not have the attribute 'FakeTensorMode' [3 pull requests, 6 comments, 3 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

PR fix notes

PR #37866: fix: handle FakeTensorMode patching for PyTorch compatibility

Description (problem / solution / changelog)

Changed files

PR #2108: [Qwen3TTS] [TTS] [Feat] Refactor voice cache manager

Description (problem / solution / changelog)

Purpose

Files deleted

Known Issue: torch.compile crash on startup (vllm 0.18.0 + torch 2.10)

Test Plan

1. Start the server

2. Upload a voice (one can pass "ref_text" as well)

3. First request (warmup, it extracts speaker embedding, and then caches it)

4. Second request (embedding already cached, skips extraction)

5. Batch endpoint (all items share the cached embedding)

6. Delete voice (cache key removed, next upload gets fresh extraction)

Test Result

Changed files

Your current environment

root@b20654c00b62:/app/model# python collect_env.py Collecting environment information...

============================== PyTorch Info

============================== Python Environment

============================== CUDA / GPU Info

============================== CPU Info

============================== Versions of relevant libraries

============================== vLLM Info

============================== Environment Variables

🐛 Describe the bug

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING