vllm - 💡(How to fix) Fix [Bug]: deepseek v4 failed to work on R6000 GPU
ON THIS PAGE
Recommended Tools
×6Utilities matched from this issue’s tags and category — try them while you read without losing context.
GitHub issue graph ai analysis
Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.
The report is written in English Markdown for sharing and archival.
Helpful · Quick feedback
Error Message
(deepseekv4vllm) root@ubuntugpu-h2-96G-01:/wayne/deepseekv4vllm/vllm# CUDA_VISIBLE_DEVICES=0 VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve deepseek-ai/DeepSeek-V4-Flash --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 1 --max-model-len 4096 --kv-cache-dtype fp8 --block-size 256 --gpu-memory-utilization 0.70 --cpu-offload-gb 128 --max-num-seqs 1 --max-num-batched-tokens 1024 --enforce-eager --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █ █ █▄ ▄█
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.20.2rc1.dev254+ge1c8776e9
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █▄█▀ █ █ █ █ model deepseek-ai/DeepSeek-V4-Flash
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:240] non-default args: {'model_tag': 'deepseek-ai/DeepSeek-V4-Flash', 'host': '0.0.0.0', 'model': 'deepseek-ai/DeepSeek-V4-Flash', 'tokenizer_mode': 'deepseek_v4', 'trust_remote_code': True, 'max_model_len': 4096, 'enforce_eager': True, 'reasoning_parser': 'deepseek_v4', 'block_size': 256, 'gpu_memory_utilization': 0.7, 'kv_cache_dtype': 'fp8', 'cpu_offload_gb': 128.0, 'max_num_batched_tokens': 1024, 'max_num_seqs': 1}
(APIServer pid=396885) INFO 05-12 14:40:18 [config.py:800] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM.
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:568] Resolved architecture: DeepseekV4ForCausalLM
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:1697] Using max model len 4096
(APIServer pid=396885) INFO 05-12 14:40:20 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=396885) INFO 05-12 14:40:20 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=1024.
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:899] Asynchronous scheduling is enabled.
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:955] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:973] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=396885) INFO 05-12 14:40:20 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:1148] Cudagraph is disabled under eager mode
(APIServer pid=396885) WARNING 05-12 14:40:21 [vllm.py:1313] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implementedthe /wayne/deepseekv4vllm/vllm#reasoning_start_str and reasoning_end_str.
(APIServer pid=396885) INFO 05-12 14:40:21 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=396927) INFO 05-12 14:40:26 [core.py:109] Initializing a V1 LLM engine (v0.20.2rc1.dev254+ge1c8776e9) with config: model='deepseek-ai/DeepSeek-V4-Flash', speculative_config=None, tokenizer='deepseek-ai/DeepSeek-V4-Flash', skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_v4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=deepseek-ai/DeepSeek-V4-Flash, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [1024], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=False, moe_backend='auto')
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1410] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.74.124.32:44933 backend=nccl
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1723] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=396927) INFO 05-12 14:40:27 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=396927) INFO 05-12 14:40:27 [base.py:123] Offloader set to UVAOffloader
(EngineCore pid=396927) INFO 05-12 14:40:27 [gpu_model_runner.py:4863] Starting to load model deepseek-ai/DeepSeek-V4-Flash...
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4.py:168] DeepSeek V4 expert_dtype resolved to 'fp4'
(EngineCore pid=396927) INFO 05-12 14:40:28 [init.py:393] Selected CutlassFp8BlockScaledMMKernel for Fp8LinearMethod
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4_attention.py:710] Using DeepSeek's fp8_ds_mla KV cache format.
(EngineCore pid=396927) INFO 05-12 14:40:28 [mxfp4.py:551] Using 'MARLIN' Mxfp4 MoE backend.
(EngineCore pid=396927) INFO 05-12 14:40:34 [deepseek_v4_attention.py:1092] Using FP8 indexer cache for Lightning Indexer.
(EngineCore pid=396927) INFO 05-12 14:43:27 [uva.py:58] Total CPU offloaded parameters: 128.8
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:938] Filesystem type for checkpoints: EXT4. Checkpoint size: 148.66 GiB. Available RAM: 114.56 GiB.
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:968] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (148.66 GiB) exceeds 90% of available RAM (114.56 GiB).
Loading safetensors checkpoint shards: 0% Completed | 0/46 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 2% Completed | 1/46 [00:02<01:32, 2.06s/it]
Loading safetensors checkpoint shards: 4% Completed | 2/46 [00:09<03:55, 5.35s/it]
Loading safetensors checkpoint shards: 7% Completed | 3/46 [00:17<04:32, 6.34s/it]
Loading safetensors checkpoint shards: 9% Completed | 4/46 [00:24<04:47, 6.85s/it]
Loading safetensors checkpoint shards: 11% Completed | 5/46 [00:32<04:48, 7.03s/it]
Loading safetensors checkpoint shards: 13% Completed | 6/46 [00:39<04:44, 7.12s/it]
Loading safetensors checkpoint shards: 15% Completed | 7/46 [00:46<04:39, 7.16s/it]
Loading safetensors checkpoint shards: 17% Completed | 8/46 [00:49<03:43, 5.89s/it]
Loading safetensors checkpoint shards: 20% Completed | 9/46 [00:53<03:11, 5.17s/it]
Loading safetensors checkpoint shards: 22% Completed | 10/46 [00:56<02:45, 4.59s/it]
Loading safetensors checkpoint shards: 24% Completed | 11/46 [01:00<02:26, 4.19s/it]
Loading safetensors checkpoint shards: 26% Completed | 12/46 [01:03<02:14, 3.95s/it]
Loading safetensors checkpoint shards: 28% Completed | 13/46 [01:07<02:11, 3.99s/it]
Loading safetensors checkpoint shards: 30% Completed | 14/46 [01:11<02:06, 3.95s/it]
Loading safetensors checkpoint shards: 33% Completed | 15/46 [01:16<02:15, 4.37s/it]
Loading safetensors checkpoint shards: 35% Completed | 16/46 [01:22<02:19, 4.65s/it]
Loading safetensors checkpoint shards: 37% Completed | 17/46 [01:27<02:21, 4.87s/it]
Loading safetensors checkpoint shards: 39% Completed | 18/46 [01:32<02:14, 4.80s/it]
Loading safetensors checkpoint shards: 41% Completed | 19/46 [01:36<02:02, 4.55s/it]
Loading safetensors checkpoint shards: 43% Completed | 20/46 [01:40<02:00, 4.64s/it]
Loading safetensors checkpoint shards: 46% Completed | 21/46 [01:44<01:49, 4.38s/it]
Loading safetensors checkpoint shards: 48% Completed | 22/46 [01:48<01:41, 4.21s/it]
Loading safetensors checkpoint shards: 50% Completed | 23/46 [01:53<01:39, 4.33s/it]
Loading safetensors checkpoint shards: 52% Completed | 24/46 [01:57<01:35, 4.35s/it]
Loading safetensors checkpoint shards: 54% Completed | 25/46 [02:01<01:30, 4.32s/it]
Loading safetensors checkpoint shards: 57% Completed | 26/46 [02:05<01:25, 4.27s/it]
Loading safetensors checkpoint shards: 59% Completed | 27/46 [02:10<01:24, 4.44s/it]
Loading safetensors checkpoint shards: 61% Completed | 28/46 [02:15<01:22, 4.56s/it]
Loading safetensors checkpoint shards: 63% Completed | 29/46 [02:20<01:17, 4.57s/it]
Loading safetensors checkpoint shards: 65% Completed | 30/46 [02:24<01:10, 4.39s/it]
Loading safetensors checkpoint shards: 67% Completed | 31/46 [02:27<01:02, 4.15s/it]
Loading safetensors checkpoint shards: 70% Completed | 32/46 [02:32<01:00, 4.35s/it]
Loading safetensors checkpoint shards: 72% Completed | 33/46 [02:36<00:56, 4.38s/it]
Loading safetensors checkpoint shards: 74% Completed | 34/46 [02:40<00:50, 4.22s/it]
Loading safetensors checkpoint shards: 76% Completed | 35/46 [02:45<00:47, 4.36s/it]
Loading safetensors checkpoint shards: 78% Completed | 36/46 [02:55<01:01, 6.17s/it]
Loading safetensors checkpoint shards: 80% Completed | 37/46 [03:00<00:52, 5.79s/it]
Loading safetensors checkpoint shards: 83% Completed | 38/46 [03:06<00:45, 5.68s/it]
Loading safetensors checkpoint shards: 85% Completed | 39/46 [03:11<00:39, 5.66s/it]
Loading safetensors checkpoint shards: 87% Completed | 40/46 [03:17<00:33, 5.53s/it]
Loading safetensors checkpoint shards: 89% Completed | 41/46 [03:21<00:26, 5.30s/it]
Loading safetensors checkpoint shards: 91% Completed | 42/46 [03:27<00:21, 5.37s/it]
Loading safetensors checkpoint shards: 93% Completed | 43/46 [03:31<00:14, 4.88s/it]
Loading safetensors checkpoint shards: 96% Completed | 44/46 [03:34<00:08, 4.43s/it]
Loading safetensors checkpoint shards: 98% Completed | 45/46 [03:36<00:03, 3.73s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 3.01s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 4.74s/it]
(EngineCore pid=396927)
(EngineCore pid=396927) INFO 05-12 14:47:07 [default_loader.py:397] Loading weights took 217.96 seconds
(EngineCore pid=396927) INFO 05-12 14:47:11 [mxfp4.py:1497] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=396927) INFO 05-12 14:50:19 [gpu_model_runner.py:4965] Model loading took 17.03 GiB memory and 571.286208 seconds
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] EngineCore failed to start.
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] Traceback (most recent call last):
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in init
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] super().init(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in init
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] self.model_runner.profile_run()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5954, in profile_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] outputs = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in call
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.forward(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in call
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._op(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) ERROR
(EngineCore pid=396927) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=396927) self._target(*self._args, **self._kwargs)
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1144, in run_engine_core
(EngineCore pid=396927) raise e
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in init
(EngineCore pid=396927) super().init(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in init
(EngineCore pid=396927) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) return func(*args, **kwarg
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) outputs = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) hidden_states = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in call
(EngineCore pid=396927) return self.forward(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in call
(EngineCore pid=396927) return self._op(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) tf32_hc_prenorm_gemm(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/utils/deep_gemm.py", line 477, in tf32_hc_prenorm_gemm
(EngineCore pid=396927) return _tf32_hc_prenorm_gemm_impl(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) RuntimeError: Assertion error (csrc/apis/hyperconnection.hpp:56): Unsupported architecture
[rank0]:[W512 14:50:24.024626144 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=396885) Traceback (most recent call last):
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/bin/vllm", line 8, in <module>
(APIServer pid=396885) sys.exit(main())
(APIServer pid=396885) ^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=396885) args.dispatch_function(args)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=396885) uvloop.run(run_server(args))
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/init.py", line 96, in run
(APIServer pid=396885) return __asyncio.run(
(APIServer pid=396885) ^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=396885) return runner.run(main)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=396885) return self._loop.run_until_complete(task)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/init.py", line 48, in wrapper
(APIServer pid=396885) return await main
(APIServer pid=396885) ^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 679, in run_server
(APIServer pid=396885) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 693, in run_server_worker
(APIServer pid=396885) async with build_async_engine_client(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=396885) async with build_async_engine_client_from_engine_args(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=396885) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=396885) return cls(
(APIServer pid=396885) ^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 146, in init
(APIServer pid=396885) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=396885) return AsyncMPClient(*client_args)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 900, in init
(APIServer pid=396885) super().init(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 535, in init
(APIServer pid=396885) with launch_core_engines(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 144, in exit
(APIServer pid=396885) next(self.gen)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1133, in launch_core_engines
(APIServer pid=396885) wait_for_engine_startup(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1192, in wait_for_engine_startup
(APIServer pid=396885) raise RuntimeError(
(APIServer pid=396885) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(deepseekv4vllm) root@ubuntugpu-h2-96G-01:
Root Cause
here is error log
(deepseekv4vllm) root@ubuntugpu-h2-96G-01:~/wayne/deepseekv4vllm/vllm# CUDA_VISIBLE_DEVICES=0 VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve deepseek-ai/DeepSeek-V4-Flash --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 1 --max-model-len 4096 --kv-cache-dtype fp8 --block-size 256 --gpu-memory-utilization 0.70 --cpu-offload-gb 128 --max-num-seqs 1 --max-num-batched-tokens 1024 --enforce-eager --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █ █ █▄ ▄█
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.20.2rc1.dev254+ge1c8776e9
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █▄█▀ █ █ █ █ model deepseek-ai/DeepSeek-V4-Flash
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:240] non-default args: {'model_tag': 'deepseek-ai/DeepSeek-V4-Flash', 'host': '0.0.0.0', 'model': 'deepseek-ai/DeepSeek-V4-Flash', 'tokenizer_mode': 'deepseek_v4', 'trust_remote_code': True, 'max_model_len': 4096, 'enforce_eager': True, 'reasoning_parser': 'deepseek_v4', 'block_size': 256, 'gpu_memory_utilization': 0.7, 'kv_cache_dtype': 'fp8', 'cpu_offload_gb': 128.0, 'max_num_batched_tokens': 1024, 'max_num_seqs': 1}
(APIServer pid=396885) INFO 05-12 14:40:18 [config.py:800] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM.
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:568] Resolved architecture: DeepseekV4ForCausalLM
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:1697] Using max model len 4096
(APIServer pid=396885) INFO 05-12 14:40:20 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=396885) INFO 05-12 14:40:20 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=1024.
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:899] Asynchronous scheduling is enabled.
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:955] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:973] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=396885) INFO 05-12 14:40:20 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:1148] Cudagraph is disabled under eager mode
(APIServer pid=396885) WARNING 05-12 14:40:21 [vllm.py:1313] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implementedthe `reasoning_start_str` and `reasoning_end_str`.
(APIServer pid=396885) INFO 05-12 14:40:21 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=396927) INFO 05-12 14:40:26 [core.py:109] Initializing a V1 LLM engine (v0.20.2rc1.dev254+ge1c8776e9) with config: model='deepseek-ai/DeepSeek-V4-Flash', speculative_config=None, tokenizer='deepseek-ai/DeepSeek-V4-Flash', skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_v4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=deepseek-ai/DeepSeek-V4-Flash, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [1024], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=False, moe_backend='auto')
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1410] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.74.124.32:44933 backend=nccl
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1723] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=396927) INFO 05-12 14:40:27 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=396927) INFO 05-12 14:40:27 [base.py:123] Offloader set to UVAOffloader
(EngineCore pid=396927) INFO 05-12 14:40:27 [gpu_model_runner.py:4863] Starting to load model deepseek-ai/DeepSeek-V4-Flash...
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4.py:168] DeepSeek V4 expert_dtype resolved to 'fp4'
(EngineCore pid=396927) INFO 05-12 14:40:28 [__init__.py:393] Selected CutlassFp8BlockScaledMMKernel for Fp8LinearMethod
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4_attention.py:710] Using DeepSeek's fp8_ds_mla KV cache format.
(EngineCore pid=396927) INFO 05-12 14:40:28 [mxfp4.py:551] Using 'MARLIN' Mxfp4 MoE backend.
(EngineCore pid=396927) INFO 05-12 14:40:34 [deepseek_v4_attention.py:1092] Using FP8 indexer cache for Lightning Indexer.
(EngineCore pid=396927) INFO 05-12 14:43:27 [uva.py:58] Total CPU offloaded parameters: 128.8
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:938] Filesystem type for checkpoints: EXT4. Checkpoint size: 148.66 GiB. Available RAM: 114.56 GiB.
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:968] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (148.66 GiB) exceeds 90% of available RAM (114.56 GiB).
Loading safetensors checkpoint shards: 0% Completed | 0/46 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 2% Completed | 1/46 [00:02<01:32, 2.06s/it]
Loading safetensors checkpoint shards: 4% Completed | 2/46 [00:09<03:55, 5.35s/it]
Loading safetensors checkpoint shards: 7% Completed | 3/46 [00:17<04:32, 6.34s/it]
Loading safetensors checkpoint shards: 9% Completed | 4/46 [00:24<04:47, 6.85s/it]
Loading safetensors checkpoint shards: 11% Completed | 5/46 [00:32<04:48, 7.03s/it]
Loading safetensors checkpoint shards: 13% Completed | 6/46 [00:39<04:44, 7.12s/it]
Loading safetensors checkpoint shards: 15% Completed | 7/46 [00:46<04:39, 7.16s/it]
Loading safetensors checkpoint shards: 17% Completed | 8/46 [00:49<03:43, 5.89s/it]
Loading safetensors checkpoint shards: 20% Completed | 9/46 [00:53<03:11, 5.17s/it]
Loading safetensors checkpoint shards: 22% Completed | 10/46 [00:56<02:45, 4.59s/it]
Loading safetensors checkpoint shards: 24% Completed | 11/46 [01:00<02:26, 4.19s/it]
Loading safetensors checkpoint shards: 26% Completed | 12/46 [01:03<02:14, 3.95s/it]
Loading safetensors checkpoint shards: 28% Completed | 13/46 [01:07<02:11, 3.99s/it]
Loading safetensors checkpoint shards: 30% Completed | 14/46 [01:11<02:06, 3.95s/it]
Loading safetensors checkpoint shards: 33% Completed | 15/46 [01:16<02:15, 4.37s/it]
Loading safetensors checkpoint shards: 35% Completed | 16/46 [01:22<02:19, 4.65s/it]
Loading safetensors checkpoint shards: 37% Completed | 17/46 [01:27<02:21, 4.87s/it]
Loading safetensors checkpoint shards: 39% Completed | 18/46 [01:32<02:14, 4.80s/it]
Loading safetensors checkpoint shards: 41% Completed | 19/46 [01:36<02:02, 4.55s/it]
Loading safetensors checkpoint shards: 43% Completed | 20/46 [01:40<02:00, 4.64s/it]
Loading safetensors checkpoint shards: 46% Completed | 21/46 [01:44<01:49, 4.38s/it]
Loading safetensors checkpoint shards: 48% Completed | 22/46 [01:48<01:41, 4.21s/it]
Loading safetensors checkpoint shards: 50% Completed | 23/46 [01:53<01:39, 4.33s/it]
Loading safetensors checkpoint shards: 52% Completed | 24/46 [01:57<01:35, 4.35s/it]
Loading safetensors checkpoint shards: 54% Completed | 25/46 [02:01<01:30, 4.32s/it]
Loading safetensors checkpoint shards: 57% Completed | 26/46 [02:05<01:25, 4.27s/it]
Loading safetensors checkpoint shards: 59% Completed | 27/46 [02:10<01:24, 4.44s/it]
Loading safetensors checkpoint shards: 61% Completed | 28/46 [02:15<01:22, 4.56s/it]
Loading safetensors checkpoint shards: 63% Completed | 29/46 [02:20<01:17, 4.57s/it]
Loading safetensors checkpoint shards: 65% Completed | 30/46 [02:24<01:10, 4.39s/it]
Loading safetensors checkpoint shards: 67% Completed | 31/46 [02:27<01:02, 4.15s/it]
Loading safetensors checkpoint shards: 70% Completed | 32/46 [02:32<01:00, 4.35s/it]
Loading safetensors checkpoint shards: 72% Completed | 33/46 [02:36<00:56, 4.38s/it]
Loading safetensors checkpoint shards: 74% Completed | 34/46 [02:40<00:50, 4.22s/it]
Loading safetensors checkpoint shards: 76% Completed | 35/46 [02:45<00:47, 4.36s/it]
Loading safetensors checkpoint shards: 78% Completed | 36/46 [02:55<01:01, 6.17s/it]
Loading safetensors checkpoint shards: 80% Completed | 37/46 [03:00<00:52, 5.79s/it]
Loading safetensors checkpoint shards: 83% Completed | 38/46 [03:06<00:45, 5.68s/it]
Loading safetensors checkpoint shards: 85% Completed | 39/46 [03:11<00:39, 5.66s/it]
Loading safetensors checkpoint shards: 87% Completed | 40/46 [03:17<00:33, 5.53s/it]
Loading safetensors checkpoint shards: 89% Completed | 41/46 [03:21<00:26, 5.30s/it]
Loading safetensors checkpoint shards: 91% Completed | 42/46 [03:27<00:21, 5.37s/it]
Loading safetensors checkpoint shards: 93% Completed | 43/46 [03:31<00:14, 4.88s/it]
Loading safetensors checkpoint shards: 96% Completed | 44/46 [03:34<00:08, 4.43s/it]
Loading safetensors checkpoint shards: 98% Completed | 45/46 [03:36<00:03, 3.73s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 3.01s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 4.74s/it]
(EngineCore pid=396927)
(EngineCore pid=396927) INFO 05-12 14:47:07 [default_loader.py:397] Loading weights took 217.96 seconds
(EngineCore pid=396927) INFO 05-12 14:47:11 [mxfp4.py:1497] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=396927) INFO 05-12 14:50:19 [gpu_model_runner.py:4965] Model loading took 17.03 GiB memory and 571.286208 seconds
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] EngineCore failed to start.
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] Traceback (most recent call last):
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in __init__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] super().__init__(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] self.model_runner.profile_run()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5954, in profile_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] outputs = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in __call__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.forward(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in __call__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._op(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) ERROR
(EngineCore pid=396927) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=396927) self._target(*self._args, **self._kwargs)
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1144, in run_engine_core
(EngineCore pid=396927) raise e
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in __init__
(EngineCore pid=396927) super().__init__(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=396927) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) return func(*args, **kwarg
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) outputs = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) hidden_states = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in __call__
(EngineCore pid=396927) return self.forward(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in __call__
(EngineCore pid=396927) return self._op(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) tf32_hc_prenorm_gemm(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/utils/deep_gemm.py", line 477, in tf32_hc_prenorm_gemm
(EngineCore pid=396927) return _tf32_hc_prenorm_gemm_impl(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) RuntimeError: Assertion error (csrc/apis/hyperconnection.hpp:56): Unsupported architecture
[rank0]:[W512 14:50:24.024626144 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=396885) Traceback (most recent call last):
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/bin/vllm", line 8, in <module>
(APIServer pid=396885) sys.exit(main())
(APIServer pid=396885) ^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=396885) args.dispatch_function(args)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=396885) uvloop.run(run_server(args))
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=396885) return __asyncio.run(
(APIServer pid=396885) ^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=396885) return runner.run(main)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=396885) return self._loop.run_until_complete(task)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=396885) return await main
(APIServer pid=396885) ^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 679, in run_server
(APIServer pid=396885) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 693, in run_server_worker
(APIServer pid=396885) async with build_async_engine_client(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=396885) async with build_async_engine_client_from_engine_args(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=396885) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=396885) return cls(
(APIServer pid=396885) ^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=396885) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=396885) return AsyncMPClient(*client_args)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 900, in __init__
(APIServer pid=396885) super().__init__(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=396885) with launch_core_engines(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=396885) next(self.gen)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1133, in launch_core_engines
(APIServer pid=396885) wait_for_engine_startup(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1192, in wait_for_engine_startup
(APIServer pid=396885) raise RuntimeError(
(APIServer pid=396885) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(deepseekv4vllm) root@ubuntugpu-h2-96G-01:~/wayne/deepseekv4vllm/vllm#Fix Action
Fix / Workaround
BIOS Model name: INTEL(R) XEON(R) GOLD 6548Y+ CPU @ 2.3GHz BIOS CPU family: 2 CPU family: 6 Model: 207 Thread(s) per core: 1 Core(s) per socket: 16 Socket(s): 1 Stepping: 2 BogoMIPS: 5000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscplm constant_tsc arch_perfmon rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves user_shstk avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities Hypervisor vendor: VMware Virtualization type: full L1d cache: 768 KiB (16 instances) L1i cache: 512 KiB (16 instances) L2 cache: 32 MiB (16 instances) L3 cache: 60 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdow ROCM Version : Could not collect vLLM Version : 0.20.2rc1.dev254+ge1c8776e9 (git sha: e1c8776e9) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X 0-15 0 N/A
here is error log
(deepseekv4vllm) root@ubuntugpu-h2-96G-01:~/wayne/deepseekv4vllm/vllm# CUDA_VISIBLE_DEVICES=0 VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve deepseek-ai/DeepSeek-V4-Flash --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 1 --max-model-len 4096 --kv-cache-dtype fp8 --block-size 256 --gpu-memory-utilization 0.70 --cpu-offload-gb 128 --max-num-seqs 1 --max-num-batched-tokens 1024 --enforce-eager --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █ █ █▄ ▄█
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.20.2rc1.dev254+ge1c8776e9
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █▄█▀ █ █ █ █ model deepseek-ai/DeepSeek-V4-Flash
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:240] non-default args: {'model_tag': 'deepseek-ai/DeepSeek-V4-Flash', 'host': '0.0.0.0', 'model': 'deepseek-ai/DeepSeek-V4-Flash', 'tokenizer_mode': 'deepseek_v4', 'trust_remote_code': True, 'max_model_len': 4096, 'enforce_eager': True, 'reasoning_parser': 'deepseek_v4', 'block_size': 256, 'gpu_memory_utilization': 0.7, 'kv_cache_dtype': 'fp8', 'cpu_offload_gb': 128.0, 'max_num_batched_tokens': 1024, 'max_num_seqs': 1}
(APIServer pid=396885) INFO 05-12 14:40:18 [config.py:800] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM.
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:568] Resolved architecture: DeepseekV4ForCausalLM
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:1697] Using max model len 4096
(APIServer pid=396885) INFO 05-12 14:40:20 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=396885) INFO 05-12 14:40:20 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=1024.
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:899] Asynchronous scheduling is enabled.
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:955] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:973] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=396885) INFO 05-12 14:40:20 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:1148] Cudagraph is disabled under eager mode
(APIServer pid=396885) WARNING 05-12 14:40:21 [vllm.py:1313] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implementedthe `reasoning_start_str` and `reasoning_end_str`.
(APIServer pid=396885) INFO 05-12 14:40:21 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=396927) INFO 05-12 14:40:26 [core.py:109] Initializing a V1 LLM engine (v0.20.2rc1.dev254+ge1c8776e9) with config: model='deepseek-ai/DeepSeek-V4-Flash', speculative_config=None, tokenizer='deepseek-ai/DeepSeek-V4-Flash', skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_v4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=deepseek-ai/DeepSeek-V4-Flash, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [1024], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=False, moe_backend='auto')
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1410] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.74.124.32:44933 backend=nccl
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1723] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=396927) INFO 05-12 14:40:27 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=396927) INFO 05-12 14:40:27 [base.py:123] Offloader set to UVAOffloader
(EngineCore pid=396927) INFO 05-12 14:40:27 [gpu_model_runner.py:4863] Starting to load model deepseek-ai/DeepSeek-V4-Flash...
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4.py:168] DeepSeek V4 expert_dtype resolved to 'fp4'
(EngineCore pid=396927) INFO 05-12 14:40:28 [__init__.py:393] Selected CutlassFp8BlockScaledMMKernel for Fp8LinearMethod
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4_attention.py:710] Using DeepSeek's fp8_ds_mla KV cache format.
(EngineCore pid=396927) INFO 05-12 14:40:28 [mxfp4.py:551] Using 'MARLIN' Mxfp4 MoE backend.
(EngineCore pid=396927) INFO 05-12 14:40:34 [deepseek_v4_attention.py:1092] Using FP8 indexer cache for Lightning Indexer.
(EngineCore pid=396927) INFO 05-12 14:43:27 [uva.py:58] Total CPU offloaded parameters: 128.8
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:938] Filesystem type for checkpoints: EXT4. Checkpoint size: 148.66 GiB. Available RAM: 114.56 GiB.
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:968] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (148.66 GiB) exceeds 90% of available RAM (114.56 GiB).
Loading safetensors checkpoint shards: 0% Completed | 0/46 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 2% Completed | 1/46 [00:02<01:32, 2.06s/it]
Loading safetensors checkpoint shards: 4% Completed | 2/46 [00:09<03:55, 5.35s/it]
Loading safetensors checkpoint shards: 7% Completed | 3/46 [00:17<04:32, 6.34s/it]
Loading safetensors checkpoint shards: 9% Completed | 4/46 [00:24<04:47, 6.85s/it]
Loading safetensors checkpoint shards: 11% Completed | 5/46 [00:32<04:48, 7.03s/it]
Loading safetensors checkpoint shards: 13% Completed | 6/46 [00:39<04:44, 7.12s/it]
Loading safetensors checkpoint shards: 15% Completed | 7/46 [00:46<04:39, 7.16s/it]
Loading safetensors checkpoint shards: 17% Completed | 8/46 [00:49<03:43, 5.89s/it]
Loading safetensors checkpoint shards: 20% Completed | 9/46 [00:53<03:11, 5.17s/it]
Loading safetensors checkpoint shards: 22% Completed | 10/46 [00:56<02:45, 4.59s/it]
Loading safetensors checkpoint shards: 24% Completed | 11/46 [01:00<02:26, 4.19s/it]
Loading safetensors checkpoint shards: 26% Completed | 12/46 [01:03<02:14, 3.95s/it]
Loading safetensors checkpoint shards: 28% Completed | 13/46 [01:07<02:11, 3.99s/it]
Loading safetensors checkpoint shards: 30% Completed | 14/46 [01:11<02:06, 3.95s/it]
Loading safetensors checkpoint shards: 33% Completed | 15/46 [01:16<02:15, 4.37s/it]
Loading safetensors checkpoint shards: 35% Completed | 16/46 [01:22<02:19, 4.65s/it]
Loading safetensors checkpoint shards: 37% Completed | 17/46 [01:27<02:21, 4.87s/it]
Loading safetensors checkpoint shards: 39% Completed | 18/46 [01:32<02:14, 4.80s/it]
Loading safetensors checkpoint shards: 41% Completed | 19/46 [01:36<02:02, 4.55s/it]
Loading safetensors checkpoint shards: 43% Completed | 20/46 [01:40<02:00, 4.64s/it]
Loading safetensors checkpoint shards: 46% Completed | 21/46 [01:44<01:49, 4.38s/it]
Loading safetensors checkpoint shards: 48% Completed | 22/46 [01:48<01:41, 4.21s/it]
Loading safetensors checkpoint shards: 50% Completed | 23/46 [01:53<01:39, 4.33s/it]
Loading safetensors checkpoint shards: 52% Completed | 24/46 [01:57<01:35, 4.35s/it]
Loading safetensors checkpoint shards: 54% Completed | 25/46 [02:01<01:30, 4.32s/it]
Loading safetensors checkpoint shards: 57% Completed | 26/46 [02:05<01:25, 4.27s/it]
Loading safetensors checkpoint shards: 59% Completed | 27/46 [02:10<01:24, 4.44s/it]
Loading safetensors checkpoint shards: 61% Completed | 28/46 [02:15<01:22, 4.56s/it]
Loading safetensors checkpoint shards: 63% Completed | 29/46 [02:20<01:17, 4.57s/it]
Loading safetensors checkpoint shards: 65% Completed | 30/46 [02:24<01:10, 4.39s/it]
Loading safetensors checkpoint shards: 67% Completed | 31/46 [02:27<01:02, 4.15s/it]
Loading safetensors checkpoint shards: 70% Completed | 32/46 [02:32<01:00, 4.35s/it]
Loading safetensors checkpoint shards: 72% Completed | 33/46 [02:36<00:56, 4.38s/it]
Loading safetensors checkpoint shards: 74% Completed | 34/46 [02:40<00:50, 4.22s/it]
Loading safetensors checkpoint shards: 76% Completed | 35/46 [02:45<00:47, 4.36s/it]
Loading safetensors checkpoint shards: 78% Completed | 36/46 [02:55<01:01, 6.17s/it]
Loading safetensors checkpoint shards: 80% Completed | 37/46 [03:00<00:52, 5.79s/it]
Loading safetensors checkpoint shards: 83% Completed | 38/46 [03:06<00:45, 5.68s/it]
Loading safetensors checkpoint shards: 85% Completed | 39/46 [03:11<00:39, 5.66s/it]
Loading safetensors checkpoint shards: 87% Completed | 40/46 [03:17<00:33, 5.53s/it]
Loading safetensors checkpoint shards: 89% Completed | 41/46 [03:21<00:26, 5.30s/it]
Loading safetensors checkpoint shards: 91% Completed | 42/46 [03:27<00:21, 5.37s/it]
Loading safetensors checkpoint shards: 93% Completed | 43/46 [03:31<00:14, 4.88s/it]
Loading safetensors checkpoint shards: 96% Completed | 44/46 [03:34<00:08, 4.43s/it]
Loading safetensors checkpoint shards: 98% Completed | 45/46 [03:36<00:03, 3.73s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 3.01s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 4.74s/it]
(EngineCore pid=396927)
(EngineCore pid=396927) INFO 05-12 14:47:07 [default_loader.py:397] Loading weights took 217.96 seconds
(EngineCore pid=396927) INFO 05-12 14:47:11 [mxfp4.py:1497] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=396927) INFO 05-12 14:50:19 [gpu_model_runner.py:4965] Model loading took 17.03 GiB memory and 571.286208 seconds
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] EngineCore failed to start.
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] Traceback (most recent call last):
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in __init__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] super().__init__(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] self.model_runner.profile_run()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5954, in profile_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] outputs = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in __call__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.forward(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in __call__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._op(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) ERROR
(EngineCore pid=396927) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=396927) self._target(*self._args, **self._kwargs)
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1144, in run_engine_core
(EngineCore pid=396927) raise e
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in __init__
(EngineCore pid=396927) super().__init__(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=396927) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) return func(*args, **kwarg
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) outputs = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) hidden_states = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in __call__
(EngineCore pid=396927) return self.forward(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in __call__
(EngineCore pid=396927) return self._op(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) tf32_hc_prenorm_gemm(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/utils/deep_gemm.py", line 477, in tf32_hc_prenorm_gemm
(EngineCore pid=396927) return _tf32_hc_prenorm_gemm_impl(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) RuntimeError: Assertion error (csrc/apis/hyperconnection.hpp:56): Unsupported architecture
[rank0]:[W512 14:50:24.024626144 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=396885) Traceback (most recent call last):
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/bin/vllm", line 8, in <module>
(APIServer pid=396885) sys.exit(main())
(APIServer pid=396885) ^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=396885) args.dispatch_function(args)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=396885) uvloop.run(run_server(args))
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=396885) return __asyncio.run(
(APIServer pid=396885) ^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=396885) return runner.run(main)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=396885) return self._loop.run_until_complete(task)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=396885) return await main
(APIServer pid=396885) ^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 679, in run_server
(APIServer pid=396885) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 693, in run_server_worker
(APIServer pid=396885) async with build_async_engine_client(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=396885) async with build_async_engine_client_from_engine_args(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=396885) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=396885) return cls(
(APIServer pid=396885) ^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=396885) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=396885) return AsyncMPClient(*client_args)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 900, in __init__
(APIServer pid=396885) super().__init__(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=396885) with launch_core_engines(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=396885) next(self.gen)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1133, in launch_core_engines
(APIServer pid=396885) wait_for_engine_startup(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1192, in wait_for_engine_startup
(APIServer pid=396885) raise RuntimeError(
(APIServer pid=396885) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(deepseekv4vllm) root@ubuntugpu-h2-96G-01:~/wayne/deepseekv4vllm/vllm#Code Example
Your output of `python collect_env.py` here
---
Collecting environment information...
BIOS Model name: INTEL(R) XEON(R) GOLD 6548Y+ CPU @ 2.3GHz
BIOS CPU family: 2
CPU family: 6
Model: 207
Thread(s) per core: 1
Core(s) per socket: 16
Socket(s): 1
Stepping: 2
BogoMIPS: 5000.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscplm constant_tsc arch_perfmon rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves user_shstk avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 768 KiB (16 instances)
L1i cache: 512 KiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 60 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdow
ROCM Version : Could not collect
vLLM Version : 0.20.2rc1.dev254+ge1c8776e9 (git sha: e1c8776e9)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-15 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
==============================
Environment Variables
==============================
LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64:
CUDA_HOME=/usr/local/cuda-13.0
CUDA_HOME=/usr/local/cuda-13.0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
---
(deepseekv4vllm) root@ubuntugpu-h2-96G-01:~/wayne/deepseekv4vllm/vllm# CUDA_VISIBLE_DEVICES=0 VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve deepseek-ai/DeepSeek-V4-Flash --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 1 --max-model-len 4096 --kv-cache-dtype fp8 --block-size 256 --gpu-memory-utilization 0.70 --cpu-offload-gb 128 --max-num-seqs 1 --max-num-batched-tokens 1024 --enforce-eager --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █ █ █▄ ▄█
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.20.2rc1.dev254+ge1c8776e9
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █▄█▀ █ █ █ █ model deepseek-ai/DeepSeek-V4-Flash
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:240] non-default args: {'model_tag': 'deepseek-ai/DeepSeek-V4-Flash', 'host': '0.0.0.0', 'model': 'deepseek-ai/DeepSeek-V4-Flash', 'tokenizer_mode': 'deepseek_v4', 'trust_remote_code': True, 'max_model_len': 4096, 'enforce_eager': True, 'reasoning_parser': 'deepseek_v4', 'block_size': 256, 'gpu_memory_utilization': 0.7, 'kv_cache_dtype': 'fp8', 'cpu_offload_gb': 128.0, 'max_num_batched_tokens': 1024, 'max_num_seqs': 1}
(APIServer pid=396885) INFO 05-12 14:40:18 [config.py:800] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM.
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:568] Resolved architecture: DeepseekV4ForCausalLM
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:1697] Using max model len 4096
(APIServer pid=396885) INFO 05-12 14:40:20 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=396885) INFO 05-12 14:40:20 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=1024.
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:899] Asynchronous scheduling is enabled.
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:955] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:973] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=396885) INFO 05-12 14:40:20 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:1148] Cudagraph is disabled under eager mode
(APIServer pid=396885) WARNING 05-12 14:40:21 [vllm.py:1313] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implementedthe `reasoning_start_str` and `reasoning_end_str`.
(APIServer pid=396885) INFO 05-12 14:40:21 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=396927) INFO 05-12 14:40:26 [core.py:109] Initializing a V1 LLM engine (v0.20.2rc1.dev254+ge1c8776e9) with config: model='deepseek-ai/DeepSeek-V4-Flash', speculative_config=None, tokenizer='deepseek-ai/DeepSeek-V4-Flash', skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_v4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=deepseek-ai/DeepSeek-V4-Flash, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [1024], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=False, moe_backend='auto')
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1410] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.74.124.32:44933 backend=nccl
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1723] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=396927) INFO 05-12 14:40:27 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=396927) INFO 05-12 14:40:27 [base.py:123] Offloader set to UVAOffloader
(EngineCore pid=396927) INFO 05-12 14:40:27 [gpu_model_runner.py:4863] Starting to load model deepseek-ai/DeepSeek-V4-Flash...
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4.py:168] DeepSeek V4 expert_dtype resolved to 'fp4'
(EngineCore pid=396927) INFO 05-12 14:40:28 [__init__.py:393] Selected CutlassFp8BlockScaledMMKernel for Fp8LinearMethod
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4_attention.py:710] Using DeepSeek's fp8_ds_mla KV cache format.
(EngineCore pid=396927) INFO 05-12 14:40:28 [mxfp4.py:551] Using 'MARLIN' Mxfp4 MoE backend.
(EngineCore pid=396927) INFO 05-12 14:40:34 [deepseek_v4_attention.py:1092] Using FP8 indexer cache for Lightning Indexer.
(EngineCore pid=396927) INFO 05-12 14:43:27 [uva.py:58] Total CPU offloaded parameters: 128.8
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:938] Filesystem type for checkpoints: EXT4. Checkpoint size: 148.66 GiB. Available RAM: 114.56 GiB.
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:968] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (148.66 GiB) exceeds 90% of available RAM (114.56 GiB).
Loading safetensors checkpoint shards: 0% Completed | 0/46 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 2% Completed | 1/46 [00:02<01:32, 2.06s/it]
Loading safetensors checkpoint shards: 4% Completed | 2/46 [00:09<03:55, 5.35s/it]
Loading safetensors checkpoint shards: 7% Completed | 3/46 [00:17<04:32, 6.34s/it]
Loading safetensors checkpoint shards: 9% Completed | 4/46 [00:24<04:47, 6.85s/it]
Loading safetensors checkpoint shards: 11% Completed | 5/46 [00:32<04:48, 7.03s/it]
Loading safetensors checkpoint shards: 13% Completed | 6/46 [00:39<04:44, 7.12s/it]
Loading safetensors checkpoint shards: 15% Completed | 7/46 [00:46<04:39, 7.16s/it]
Loading safetensors checkpoint shards: 17% Completed | 8/46 [00:49<03:43, 5.89s/it]
Loading safetensors checkpoint shards: 20% Completed | 9/46 [00:53<03:11, 5.17s/it]
Loading safetensors checkpoint shards: 22% Completed | 10/46 [00:56<02:45, 4.59s/it]
Loading safetensors checkpoint shards: 24% Completed | 11/46 [01:00<02:26, 4.19s/it]
Loading safetensors checkpoint shards: 26% Completed | 12/46 [01:03<02:14, 3.95s/it]
Loading safetensors checkpoint shards: 28% Completed | 13/46 [01:07<02:11, 3.99s/it]
Loading safetensors checkpoint shards: 30% Completed | 14/46 [01:11<02:06, 3.95s/it]
Loading safetensors checkpoint shards: 33% Completed | 15/46 [01:16<02:15, 4.37s/it]
Loading safetensors checkpoint shards: 35% Completed | 16/46 [01:22<02:19, 4.65s/it]
Loading safetensors checkpoint shards: 37% Completed | 17/46 [01:27<02:21, 4.87s/it]
Loading safetensors checkpoint shards: 39% Completed | 18/46 [01:32<02:14, 4.80s/it]
Loading safetensors checkpoint shards: 41% Completed | 19/46 [01:36<02:02, 4.55s/it]
Loading safetensors checkpoint shards: 43% Completed | 20/46 [01:40<02:00, 4.64s/it]
Loading safetensors checkpoint shards: 46% Completed | 21/46 [01:44<01:49, 4.38s/it]
Loading safetensors checkpoint shards: 48% Completed | 22/46 [01:48<01:41, 4.21s/it]
Loading safetensors checkpoint shards: 50% Completed | 23/46 [01:53<01:39, 4.33s/it]
Loading safetensors checkpoint shards: 52% Completed | 24/46 [01:57<01:35, 4.35s/it]
Loading safetensors checkpoint shards: 54% Completed | 25/46 [02:01<01:30, 4.32s/it]
Loading safetensors checkpoint shards: 57% Completed | 26/46 [02:05<01:25, 4.27s/it]
Loading safetensors checkpoint shards: 59% Completed | 27/46 [02:10<01:24, 4.44s/it]
Loading safetensors checkpoint shards: 61% Completed | 28/46 [02:15<01:22, 4.56s/it]
Loading safetensors checkpoint shards: 63% Completed | 29/46 [02:20<01:17, 4.57s/it]
Loading safetensors checkpoint shards: 65% Completed | 30/46 [02:24<01:10, 4.39s/it]
Loading safetensors checkpoint shards: 67% Completed | 31/46 [02:27<01:02, 4.15s/it]
Loading safetensors checkpoint shards: 70% Completed | 32/46 [02:32<01:00, 4.35s/it]
Loading safetensors checkpoint shards: 72% Completed | 33/46 [02:36<00:56, 4.38s/it]
Loading safetensors checkpoint shards: 74% Completed | 34/46 [02:40<00:50, 4.22s/it]
Loading safetensors checkpoint shards: 76% Completed | 35/46 [02:45<00:47, 4.36s/it]
Loading safetensors checkpoint shards: 78% Completed | 36/46 [02:55<01:01, 6.17s/it]
Loading safetensors checkpoint shards: 80% Completed | 37/46 [03:00<00:52, 5.79s/it]
Loading safetensors checkpoint shards: 83% Completed | 38/46 [03:06<00:45, 5.68s/it]
Loading safetensors checkpoint shards: 85% Completed | 39/46 [03:11<00:39, 5.66s/it]
Loading safetensors checkpoint shards: 87% Completed | 40/46 [03:17<00:33, 5.53s/it]
Loading safetensors checkpoint shards: 89% Completed | 41/46 [03:21<00:26, 5.30s/it]
Loading safetensors checkpoint shards: 91% Completed | 42/46 [03:27<00:21, 5.37s/it]
Loading safetensors checkpoint shards: 93% Completed | 43/46 [03:31<00:14, 4.88s/it]
Loading safetensors checkpoint shards: 96% Completed | 44/46 [03:34<00:08, 4.43s/it]
Loading safetensors checkpoint shards: 98% Completed | 45/46 [03:36<00:03, 3.73s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 3.01s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 4.74s/it]
(EngineCore pid=396927)
(EngineCore pid=396927) INFO 05-12 14:47:07 [default_loader.py:397] Loading weights took 217.96 seconds
(EngineCore pid=396927) INFO 05-12 14:47:11 [mxfp4.py:1497] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=396927) INFO 05-12 14:50:19 [gpu_model_runner.py:4965] Model loading took 17.03 GiB memory and 571.286208 seconds
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] EngineCore failed to start.
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] Traceback (most recent call last):
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in __init__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] super().__init__(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] self.model_runner.profile_run()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5954, in profile_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] outputs = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in __call__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.forward(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in __call__
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._op(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) ERROR
(EngineCore pid=396927) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=396927) self._target(*self._args, **self._kwargs)
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1144, in run_engine_core
(EngineCore pid=396927) raise e
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in __init__
(EngineCore pid=396927) super().__init__(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=396927) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) return func(*args, **kwarg
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) outputs = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) hidden_states = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in __call__
(EngineCore pid=396927) return self.forward(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in __call__
(EngineCore pid=396927) return self._op(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) tf32_hc_prenorm_gemm(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/utils/deep_gemm.py", line 477, in tf32_hc_prenorm_gemm
(EngineCore pid=396927) return _tf32_hc_prenorm_gemm_impl(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) RuntimeError: Assertion error (csrc/apis/hyperconnection.hpp:56): Unsupported architecture
[rank0]:[W512 14:50:24.024626144 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=396885) Traceback (most recent call last):
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/bin/vllm", line 8, in <module>
(APIServer pid=396885) sys.exit(main())
(APIServer pid=396885) ^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=396885) args.dispatch_function(args)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=396885) uvloop.run(run_server(args))
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=396885) return __asyncio.run(
(APIServer pid=396885) ^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=396885) return runner.run(main)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=396885) return self._loop.run_until_complete(task)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=396885) return await main
(APIServer pid=396885) ^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 679, in run_server
(APIServer pid=396885) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 693, in run_server_worker
(APIServer pid=396885) async with build_async_engine_client(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=396885) async with build_async_engine_client_from_engine_args(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=396885) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=396885) return cls(
(APIServer pid=396885) ^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=396885) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=396885) return AsyncMPClient(*client_args)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 900, in __init__
(APIServer pid=396885) super().__init__(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=396885) with launch_core_engines(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=396885) next(self.gen)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1133, in launch_core_engines
(APIServer pid=396885) wait_for_engine_startup(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1192, in wait_for_engine_startup
(APIServer pid=396885) raise RuntimeError(
(APIServer pid=396885) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(deepseekv4vllm) root@ubuntugpu-h2-96G-01:~/wayne/deepseekv4vllm/vllm#RAW_BUFFERClick to expand / collapse
Your current environment
<details> <summary>The output of <code>python collect_env.py</code></summary>Your output of `python collect_env.py` hereBIOS Model name: INTEL(R) XEON(R) GOLD 6548Y+ CPU @ 2.3GHz BIOS CPU family: 2 CPU family: 6 Model: 207 Thread(s) per core: 1 Core(s) per socket: 16 Socket(s): 1 Stepping: 2 BogoMIPS: 5000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscplm constant_tsc arch_perfmon rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves user_shstk avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities Hypervisor vendor: VMware Virtualization type: full L1d cache: 768 KiB (16 instances) L1i cache: 512 KiB (16 instances) L2 cache: 32 MiB (16 instances) L3 cache: 60 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdow ROCM Version : Could not collect vLLM Version : 0.20.2rc1.dev254+ge1c8776e9 (git sha: e1c8776e9) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X 0-15 0 N/A
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks
============================== Environment Variables
LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64: CUDA_HOME=/usr/local/cuda-13.0 CUDA_HOME=/usr/local/cuda-13.0 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
### 🐛 Describe the bug
here is error log(deepseekv4vllm) root@ubuntugpu-h2-96G-01:/wayne/deepseekv4vllm/vllm# CUDA_VISIBLE_DEVICES=0 VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve deepseek-ai/DeepSeek-V4-Flash --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 1 --max-model-len 4096 --kv-cache-dtype fp8 --block-size 256 --gpu-memory-utilization 0.70 --cpu-offload-gb 128 --max-num-seqs 1 --max-num-batched-tokens 1024 --enforce-eager --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █ █ █▄ ▄█
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.20.2rc1.dev254+ge1c8776e9
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] █▄█▀ █ █ █ █ model deepseek-ai/DeepSeek-V4-Flash
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:306]
(APIServer pid=396885) INFO 05-12 14:40:18 [utils.py:240] non-default args: {'model_tag': 'deepseek-ai/DeepSeek-V4-Flash', 'host': '0.0.0.0', 'model': 'deepseek-ai/DeepSeek-V4-Flash', 'tokenizer_mode': 'deepseek_v4', 'trust_remote_code': True, 'max_model_len': 4096, 'enforce_eager': True, 'reasoning_parser': 'deepseek_v4', 'block_size': 256, 'gpu_memory_utilization': 0.7, 'kv_cache_dtype': 'fp8', 'cpu_offload_gb': 128.0, 'max_num_batched_tokens': 1024, 'max_num_seqs': 1}
(APIServer pid=396885) INFO 05-12 14:40:18 [config.py:800] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM.
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:568] Resolved architecture: DeepseekV4ForCausalLM
(APIServer pid=396885) INFO 05-12 14:40:19 [model.py:1697] Using max model len 4096
(APIServer pid=396885) INFO 05-12 14:40:20 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=396885) INFO 05-12 14:40:20 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=1024.
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:899] Asynchronous scheduling is enabled.
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:955] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=396885) WARNING 05-12 14:40:20 [vllm.py:973] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=396885) INFO 05-12 14:40:20 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=396885) INFO 05-12 14:40:20 [vllm.py:1148] Cudagraph is disabled under eager mode
(APIServer pid=396885) WARNING 05-12 14:40:21 [vllm.py:1313] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implementedthe /wayne/deepseekv4vllm/vllm#reasoning_start_str and reasoning_end_str.
(APIServer pid=396885) INFO 05-12 14:40:21 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=396927) INFO 05-12 14:40:26 [core.py:109] Initializing a V1 LLM engine (v0.20.2rc1.dev254+ge1c8776e9) with config: model='deepseek-ai/DeepSeek-V4-Flash', speculative_config=None, tokenizer='deepseek-ai/DeepSeek-V4-Flash', skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_v4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=deepseek-ai/DeepSeek-V4-Flash, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [1024], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=False, moe_backend='auto')
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1410] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.74.124.32:44933 backend=nccl
(EngineCore pid=396927) INFO 05-12 14:40:26 [parallel_state.py:1723] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=396927) INFO 05-12 14:40:27 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=396927) INFO 05-12 14:40:27 [base.py:123] Offloader set to UVAOffloader
(EngineCore pid=396927) INFO 05-12 14:40:27 [gpu_model_runner.py:4863] Starting to load model deepseek-ai/DeepSeek-V4-Flash...
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4.py:168] DeepSeek V4 expert_dtype resolved to 'fp4'
(EngineCore pid=396927) INFO 05-12 14:40:28 [init.py:393] Selected CutlassFp8BlockScaledMMKernel for Fp8LinearMethod
(EngineCore pid=396927) INFO 05-12 14:40:28 [deepseek_v4_attention.py:710] Using DeepSeek's fp8_ds_mla KV cache format.
(EngineCore pid=396927) INFO 05-12 14:40:28 [mxfp4.py:551] Using 'MARLIN' Mxfp4 MoE backend.
(EngineCore pid=396927) INFO 05-12 14:40:34 [deepseek_v4_attention.py:1092] Using FP8 indexer cache for Lightning Indexer.
(EngineCore pid=396927) INFO 05-12 14:43:27 [uva.py:58] Total CPU offloaded parameters: 128.8
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:938] Filesystem type for checkpoints: EXT4. Checkpoint size: 148.66 GiB. Available RAM: 114.56 GiB.
(EngineCore pid=396927) INFO 05-12 14:43:29 [weight_utils.py:968] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (148.66 GiB) exceeds 90% of available RAM (114.56 GiB).
Loading safetensors checkpoint shards: 0% Completed | 0/46 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 2% Completed | 1/46 [00:02<01:32, 2.06s/it]
Loading safetensors checkpoint shards: 4% Completed | 2/46 [00:09<03:55, 5.35s/it]
Loading safetensors checkpoint shards: 7% Completed | 3/46 [00:17<04:32, 6.34s/it]
Loading safetensors checkpoint shards: 9% Completed | 4/46 [00:24<04:47, 6.85s/it]
Loading safetensors checkpoint shards: 11% Completed | 5/46 [00:32<04:48, 7.03s/it]
Loading safetensors checkpoint shards: 13% Completed | 6/46 [00:39<04:44, 7.12s/it]
Loading safetensors checkpoint shards: 15% Completed | 7/46 [00:46<04:39, 7.16s/it]
Loading safetensors checkpoint shards: 17% Completed | 8/46 [00:49<03:43, 5.89s/it]
Loading safetensors checkpoint shards: 20% Completed | 9/46 [00:53<03:11, 5.17s/it]
Loading safetensors checkpoint shards: 22% Completed | 10/46 [00:56<02:45, 4.59s/it]
Loading safetensors checkpoint shards: 24% Completed | 11/46 [01:00<02:26, 4.19s/it]
Loading safetensors checkpoint shards: 26% Completed | 12/46 [01:03<02:14, 3.95s/it]
Loading safetensors checkpoint shards: 28% Completed | 13/46 [01:07<02:11, 3.99s/it]
Loading safetensors checkpoint shards: 30% Completed | 14/46 [01:11<02:06, 3.95s/it]
Loading safetensors checkpoint shards: 33% Completed | 15/46 [01:16<02:15, 4.37s/it]
Loading safetensors checkpoint shards: 35% Completed | 16/46 [01:22<02:19, 4.65s/it]
Loading safetensors checkpoint shards: 37% Completed | 17/46 [01:27<02:21, 4.87s/it]
Loading safetensors checkpoint shards: 39% Completed | 18/46 [01:32<02:14, 4.80s/it]
Loading safetensors checkpoint shards: 41% Completed | 19/46 [01:36<02:02, 4.55s/it]
Loading safetensors checkpoint shards: 43% Completed | 20/46 [01:40<02:00, 4.64s/it]
Loading safetensors checkpoint shards: 46% Completed | 21/46 [01:44<01:49, 4.38s/it]
Loading safetensors checkpoint shards: 48% Completed | 22/46 [01:48<01:41, 4.21s/it]
Loading safetensors checkpoint shards: 50% Completed | 23/46 [01:53<01:39, 4.33s/it]
Loading safetensors checkpoint shards: 52% Completed | 24/46 [01:57<01:35, 4.35s/it]
Loading safetensors checkpoint shards: 54% Completed | 25/46 [02:01<01:30, 4.32s/it]
Loading safetensors checkpoint shards: 57% Completed | 26/46 [02:05<01:25, 4.27s/it]
Loading safetensors checkpoint shards: 59% Completed | 27/46 [02:10<01:24, 4.44s/it]
Loading safetensors checkpoint shards: 61% Completed | 28/46 [02:15<01:22, 4.56s/it]
Loading safetensors checkpoint shards: 63% Completed | 29/46 [02:20<01:17, 4.57s/it]
Loading safetensors checkpoint shards: 65% Completed | 30/46 [02:24<01:10, 4.39s/it]
Loading safetensors checkpoint shards: 67% Completed | 31/46 [02:27<01:02, 4.15s/it]
Loading safetensors checkpoint shards: 70% Completed | 32/46 [02:32<01:00, 4.35s/it]
Loading safetensors checkpoint shards: 72% Completed | 33/46 [02:36<00:56, 4.38s/it]
Loading safetensors checkpoint shards: 74% Completed | 34/46 [02:40<00:50, 4.22s/it]
Loading safetensors checkpoint shards: 76% Completed | 35/46 [02:45<00:47, 4.36s/it]
Loading safetensors checkpoint shards: 78% Completed | 36/46 [02:55<01:01, 6.17s/it]
Loading safetensors checkpoint shards: 80% Completed | 37/46 [03:00<00:52, 5.79s/it]
Loading safetensors checkpoint shards: 83% Completed | 38/46 [03:06<00:45, 5.68s/it]
Loading safetensors checkpoint shards: 85% Completed | 39/46 [03:11<00:39, 5.66s/it]
Loading safetensors checkpoint shards: 87% Completed | 40/46 [03:17<00:33, 5.53s/it]
Loading safetensors checkpoint shards: 89% Completed | 41/46 [03:21<00:26, 5.30s/it]
Loading safetensors checkpoint shards: 91% Completed | 42/46 [03:27<00:21, 5.37s/it]
Loading safetensors checkpoint shards: 93% Completed | 43/46 [03:31<00:14, 4.88s/it]
Loading safetensors checkpoint shards: 96% Completed | 44/46 [03:34<00:08, 4.43s/it]
Loading safetensors checkpoint shards: 98% Completed | 45/46 [03:36<00:03, 3.73s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 3.01s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [03:37<00:00, 4.74s/it]
(EngineCore pid=396927)
(EngineCore pid=396927) INFO 05-12 14:47:07 [default_loader.py:397] Loading weights took 217.96 seconds
(EngineCore pid=396927) INFO 05-12 14:47:11 [mxfp4.py:1497] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=396927) INFO 05-12 14:50:19 [gpu_model_runner.py:4965] Model loading took 17.03 GiB memory and 571.286208 seconds
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] EngineCore failed to start.
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] Traceback (most recent call last):
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in init
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] super().init(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in init
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] self.model_runner.profile_run()
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5954, in profile_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] outputs = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states = self.model(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in call
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self.forward(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return forward_call(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in call
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] return self._op(*args, **kwargs)
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) ERROR 05-12 14:50:20 [core.py:1140] File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) ERROR
(EngineCore pid=396927) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=396927) self._target(*self._args, **self._kwargs)
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1144, in run_engine_core
(EngineCore pid=396927) raise e
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=396927) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 880, in init
(EngineCore pid=396927) super().init(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 128, in init
(EngineCore pid=396927) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=396927) return func(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=396927) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=396927) return self.collective_rpc("determine_available_memory")
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc
(EngineCore pid=396927) result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=396927) return func(*args, **kwarg
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/worker/gpu_model_runner.py", line 5622, in _dummy_run
(EngineCore pid=396927) outputs = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1669, in forward
(EngineCore pid=396927) hidden_states = self.model(
(EngineCore pid=396927) ^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/compilation/decorators.py", line 507, in call
(EngineCore pid=396927) return self.forward(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1405, in forward
(EngineCore pid=396927) hidden_states, residual, post_mix, res_mix = layer(
(EngineCore pid=396927) ^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=396927) return self._call_impl(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=396927) return forward_call(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1213, in forward
(EngineCore pid=396927) x, post_mix, res_mix = self.hc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/models/deepseek_v4.py", line 1179, in hc_pre
(EngineCore pid=396927) post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in call
(EngineCore pid=396927) return self._op(*args, **kwargs)
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/model_executor/layers/mhc.py", line 310, in mhc_pre
(EngineCore pid=396927) tf32_hc_prenorm_gemm(
(EngineCore pid=396927) File "/root/wayne/deepseekv4vllm/vllm/vllm/utils/deep_gemm.py", line 477, in tf32_hc_prenorm_gemm
(EngineCore pid=396927) return _tf32_hc_prenorm_gemm_impl(
(EngineCore pid=396927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396927) RuntimeError: Assertion error (csrc/apis/hyperconnection.hpp:56): Unsupported architecture
[rank0]:[W512 14:50:24.024626144 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=396885) Traceback (most recent call last):
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/bin/vllm", line 8, in <module>
(APIServer pid=396885) sys.exit(main())
(APIServer pid=396885) ^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=396885) args.dispatch_function(args)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=396885) uvloop.run(run_server(args))
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/init.py", line 96, in run
(APIServer pid=396885) return __asyncio.run(
(APIServer pid=396885) ^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=396885) return runner.run(main)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=396885) return self._loop.run_until_complete(task)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/lib/python3.12/site-packages/uvloop/init.py", line 48, in wrapper
(APIServer pid=396885) return await main
(APIServer pid=396885) ^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 679, in run_server
(APIServer pid=396885) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 693, in run_server_worker
(APIServer pid=396885) async with build_async_engine_client(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=396885) async with build_async_engine_client_from_engine_args(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=396885) return await anext(self.gen)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=396885) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=396885) return cls(
(APIServer pid=396885) ^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/async_llm.py", line 146, in init
(APIServer pid=396885) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=396885) return AsyncMPClient(*client_args)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=396885) return func(*args, **kwargs)
(APIServer pid=396885) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 900, in init
(APIServer pid=396885) super().init(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/core_client.py", line 535, in init
(APIServer pid=396885) with launch_core_engines(
(APIServer pid=396885) File "/usr/lib/python3.12/contextlib.py", line 144, in exit
(APIServer pid=396885) next(self.gen)
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1133, in launch_core_engines
(APIServer pid=396885) wait_for_engine_startup(
(APIServer pid=396885) File "/root/wayne/deepseekv4vllm/vllm/vllm/v1/engine/utils.py", line 1192, in wait_for_engine_startup
(APIServer pid=396885) raise RuntimeError(
(APIServer pid=396885) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(deepseekv4vllm) root@ubuntugpu-h2-96G-01:
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.Vote matrix · Quick signals
Still need to ship something?
×6Another batch ranked right after the header list — different links, same matching logic.
TRENDING
- Feature Request: Configurable per-minute rate limiting (RPM) for models to prevent 429 errors
- Android: Hermes App + Termux install share ~/.hermes and cause silent permission loops
- hermes update emits unicode-animations ANSI demo in non-interactive logs
- hermes update downgrades aiohttp from 3.13.4 to 3.13.3
- npm install warns about deprecated @babel/plugin-proposal-private-methods
- DingTalk inbound media URLs are skipped as unreadable native image paths
- fix(dashboard): ChatPage clears header action buttons on ALL pages, not just Sessions
- [Bug]: check_web_api_key() hardcodes built-in backends — third-party web search plugins silently disabled
- Hermes Web UI 修复经验：GatewayManager 补丁、进程 D 状态、数据库升级问题
- Telegram gateway can silently drop turn after /stop with response=0 chars while internal work continues
- Bug Report: v0.14.0 上下文污染 — 历史回复碎片回注到新请求
- Bug: hermes skills search table truncates Identifier column — install fails with copied value
- [skills-index-watchdog] Skills index is stale or degraded (degraded)
- Discord approval embed not rendering on web/mobile — embed data present in API but invisible
- Idea: Discord voice-channel participation / opt-in auto-join mode
- [Feature]: Claude Code--ultrawork
- build-arm64 job deterministically fails on cold cache (Azure SAS token expires mid-build)
- [Enhancement] computer_use: action=type should fall back to key events for terminal emulators (Ghostty/Terminal.app/iTerm2)
- Feature Request: Session Recovery on Temporary Provider Outage
- [Bug]: Hermes dashboard not working on NixOS (container)
- [Feature]: Add option to ignore @all/@everyone mentions in Feishu group chats
- QQ Bot WebSocket 频繁断开：长时间工具执行阻塞 asyncio 事件循环导致心跳超时
- patch tool: new_string escape sequences (\t) get written literally
- Feature Request: i18n / 多语言支持（国际化）
- Bug: web_crawl schema lets models auto-guess "instructions" instead of asking the user via clarify
- feat: `!command` prefix for direct shell execution (like Claude Code)
- Expose currently-running cron jobs via /api/jobs (or new endpoint)
- [Bug]: Kanban parent-child handoff: scratch workspace GC destroys artifacts before child can read them
- [Bug, Windows] hermes gateway restart loses session context — planned_stop_marker not written before SIGTERM
- [Bug]: Codex→DeepSeek fallback sends assistant turns without reasoning_content → HTTP 400 (require-side cross-provider failover)
- [Bug]: Update got stuck half way, reboot it, then ModuleNotFoundError: No module named 'hermes_cli'
- Kanban dispatcher corrupt-board handling and multi-profile gateway ownership ambiguity
- Gateway can resend a short fallback message when the real final Telegram response was already delivered
- [BUG] Bedrock: Fix 'Invalid API Key format' for presigned URL tokens
- Secret redaction corrupts code syntax in tool output (write_file, execute_code, terminal)
- Unable to connect Ollama Cloud with Pro Subscription to Hermes
- feat: fuzzy substring matching for /skill autocomplete
- PRD: Autonomous market-impact prediction briefing system
- Kanban dashboard should support task/card deep links
- [Feature] Native Feishu CardKit Streaming: consolidate best-in-class implementations
- [Feature]: Inject mental model into context when using Hindsight
- Interactive CLI hides tool output despite display.tool_progress=all, and hermes chat -v does not restore it
- fix(api_server): _handle_responses drops text.format JSON schema — structured output constraints silently ignored
- state.db FTS corruption goes undetected — no integrity check, no repair path
- bug: fallback routing can select text-only models for image requests and hide the primary failure
- feat(kanban): persist worker session_id per run and pass --resume on respawn after unblock
- feat(kanban): support GitHub/OMO lifecycle bridge for Xiyou-style automation
- Expose update-safe TUI/composer hooks for voice transcript and composer events
- Hide or configure voice transcript status rows in editable dictation mode
- [Feature]: Per-Tool / Per-Toolset Approval Policies
- Context compression creates orphan sessions missing from state.db
- messaging platform
- feat: Add read-only / silent monitoring mode for WhatsApp adapter
- double-.hermes path mismatch, the HOME env var leak, and the fallback-notification UX problem
- Bug: Plattform-Bundle name `hermes-yuanbao` in `agent.disabled_toolsets` silently kills ALL tools in gateway path (Telegram + cron), CLI unaffected
- CLI /yolo (in-chat) does not bypass dangerous command approvals — env var freeze + missing enable_session_yolo call
- OpenAI Codex provider crashes with "'NoneType' object is not iterable" (HTTP None)
- DEEPSEEK_API_KEY blocked by env blocklist in gateway process — cron jobs fail with deepseek provider
- fix(feishu): Card action callback routing issues - invalid message_id and unrecognized /card command
- Discord plugin: profiles without explicit `discord:` block silently get `require_mention=true` + `auto_thread=true` (regression in cc8e5ec2a)
- [Bug]: DISCORD_ALLOWED_ROLES ignored by gateway _is_user_authorized — role-authorized users get 'Unauthorized user' rejection
- [Bug]: /new, /clear, and /reset commands freeze the terminal session
- openai-codex subscription backend returns HTTP 200 with response.output=None, causing Slack/cron failures
- RFC: Centralized Model/Provider Registry
- bug: openai-codex provider — TypeError: 'NoneType' object is not iterable on every request (gpt-5.5)
- [Feature]: Source-aware instruction gate — architectural mitigation for indirect prompt injection
- Named custom provider stale_timeout_seconds ignored because runtime provider is normalized to `custom`
- guard test (ignore)
- [Feature]: per-platform LLM request_overrides (extra_body / reasoning_effort / service_tier)
- One-shot smoke: add Flue-backed orchestration fixture
- Gateway should not treat stale Codex app-server progress as final response after post-tool silence
- `docker_run_as_host_user: true` breaks bundled skills: Hermes home is mounted into `/root/.hermes` but the container runs as a non-root user (`HOME=/home/pn`)
- [Bug]: gateway api_server streaming bypasses server-side tool-call loop when chat_template_kwargs.enable_thinking=false (model emits tool name as plain text)
- [Feature]: Pre-install python-telegram-bot in Umbrel Hermes Docker image
- YouTube Shorts filter not working in youtube-content skill
- v0.15.0 PyPI release breaks ALL platforms — plugin.yaml manifests missing from package
- RFC: On-demand tool/skill/MCP discovery — decouple schema registration from process lifecycle
- Pixshelf: local-first stock photo workflow command center
- [Bug]: baoyu infographic skill should not silently bypass image_generate
- Pixshelf v1.5: manual submission tracking for stock agencies
- `hermes config set` silently accepts unknown keys, writing them where the runtime never reads
- Honcho memory prefetch hang on fresh CLI subprocess in v0.15.0 (regression from #27190)
- [Bug] v0.15.0 Docker image: stage2-hook.sh, main-wrapper.sh missing; container_boot module removed
- Feature: Reduce cache-read token overhead for DeepSeek providers — configurable cache_ttl, skills snapshot trimming, memory compaction
- Windows: three bugs from daily use (plugin discovery, gateway exit code, Unicode decode
- holographic memory: HRR silently degrades to FTS5 when numpy is missing
- Make max_tokens configurable for aux vision calls
- Conversation compression desynchronizes session ID between agent context and gateway routing, causing silent message loss
- [Bug]: v0.15.0 Docker image:The TUI cannot be used in the dashboard.
- cron: skip_memory=True blocks fact_store/memory tools from all cron jobs
- TUI: Node.js OOM crash when agent uses browser tools repeatedly
- feat: model_profiles — per-model toolset and memory config
- Automatic background skill patching disrupts active sessions (severe impact on local models)
- ensure_hermes_home() creates root-owned dirs in profile subdirectories when kanban workers are dispatched
- Feature: opt-in webhook bypass for DISCORD_ALLOW_BOTS — allow operator-initiated probes without weakening bot-loop guard
- v0.15.0: Codex requests fail HTTP 400 when participant display_name contains non-ASCII (emoji breaks input[].name pattern)
- Architecture: State Persistence Precedence (Memory vs Skills vs Hooks)
- [Bug]: cronjob tool: create action always fails with "schedule is required for create" even when parameters are provided
- codex-oauth: 'NoneType' object is not iterable in _run_codex_stream (gpt-5.5) — every turn fails non-retryably
- Docs/Config: Plugin local scope enablement ambiguity
- [Bug]: CLI freezes after using /new command (WSL)
- Profile Codex auth can ignore global credential pool when local state is stale
- [workflow-engine] CRITICAL: variable substitution crashes on regex metachars in user input
- [workflow-engine] HIGH: loop and bash nodes leak subprocesses on timeout
- [workflow-engine] HIGH: README documents config env vars the engine never reads
- [workflow-engine] MEDIUM: workflow_run rate limit bypassable via concurrent calls (TOCTOU)
- [workflow-engine] chore: manifest gaps, side-effectful register(), dead code, unauth kanban dispatch
- [mcp_lazy] HIGH: synthetic mcp_server_<name> stub collides with a real MCP server named 'server'
- [mcp_lazy] HIGH: promote_server eager flag documented but never persisted
- [mcp_lazy] MEDIUM: _prev_mode dict leaks and goes stale; not cleared on session evict
- [mcp_lazy] MEDIUM: get_pool has unlocked check-then-set race on pool creation
- [mcp_lazy] MEDIUM: pre_tool_call gives no guidance for unpromoted server-stub calls
- [mcp_lazy] chore: undeclared pre_tool_call hook, nonexistent 'mcp_load_tools' name in docs, missing tests
- [a2a_fleet] CRITICAL: server never auto-starts — register() runs outside an event loop
- [a2a_fleet] CRITICAL: auth_required defaults to false on a cross-machine surface
- [a2a_fleet] HIGH: remove invented disable() hook — loader never calls it, port leaks on reload
- [a2a_fleet] HIGH: plugin.yaml missing kind / provides_tools / requires_env (token env undeclared)
- [a2a_fleet] MEDIUM: tighten wide-open CORS, anonymous /health peer leak, and peer-URL SSRF
- [a2a_fleet] MEDIUM: relocate tests to tests/plugins/ and cover sync-register + auth-default paths
- xai-oauth auxiliary client incorrectly uses Responses API (CodexAuxiliaryClient), causing 403 on compression/vision/web_extract
- [Bug]: Direct Copilot gpt-5.5 large resumes are killed by 12s Codex TTFB watchdog
- [Bug]: `hermes uninstall` does not work on Windows
- TUI: Thinking block leaks raw JSON and Σ character
- Hostinger VPS: migration Hermes Agent → Hermes WebUI impossible (tini + UID mismatch + sessions)
- /goal judge over-continues exploratory goals unless the assistant explicitly says the goal is complete
- /goal auto-continuation can be amplified by preflight compression/session split and resurrect stale task state
- Dashboard infinite reload loop in loopback mode — GET /api/auth/me returns 401 on every page load
- [Bug]: Provider/LLM switch leaves stale encrypted_content causing 400 errors on Telegram sessions
- [Bug]: Infinite reload loop / React state loop on Sessions tab (Firefox + Chrome) — repeated 401 on /api/auth/me (v0.15.0)
- show_reasoning should work independently of streaming in CLI mode
- Feature Request: Strip reasoning/<think> blocks from TTS preprocessing
- mcp add / mcp test raise NameError when mcp package not installed
- v0.14.0 dashboard breaks behind reverse proxies — two regressions
- Skills hub creates empty category directories when no skills installed
- [Bug]: Custom endpoint: ChatCompletions returns content, but Hermes treats response as empty (v0.14.0)
- fix: atomic_replace() fails with EXDEV when HERMES_HOME is a cross-filesystem symlink
- fix(gateway): Feishu session cancellation orphans session guard, permanently blocking messages
- Custom endpoint pricing can overestimate Crof qwen3.5-9b cost by 1,000,000x
- MCP OAuth callback: module-level port global causes port collisions and structural weaknesses vs upstream
- Bug: send_message tool bypasses validate_media_delivery_path security check
- Proposal: Add Mnemosyne to official memory provider documentation
- feat(swarm): support custom verifier/synthesizer body + skills
- Template conversion failed
- Error occurred in the operation of the agent node in the workflow.
- PubSub client overrides Sentinel client when REDIS_USE_SENTINEL is enabled
- Frontend description of the Retrieval node output does not match the actual output
- JSON type input var raise Intenal server error
- cannot extract elements from a scalar
- 负载均衡 为模型配置多组凭据，并自动调用，此功能无法选择
- add models is error
- panic: could not create filter
- Persist partially generated messages when /chat-messages/:task_id/stop is called
- MCP server connection fails with 403 — request never leaves Dify (SSRF proxy suspected)
- Support durable async execution backends for long-running workflow steps
- [Xiaomi MiMo] Credentials validation fails with 400 "Not supported model mimo-v2-flash" when using Token Plan endpoint (v0.0.7)
- After clicking preview on a parent-child segmented knowledge base, it shows 0 chunks
- Retrieval score differs between UI upload (.docx) and API upload (.txt) despite identical chunk content and embedding model
- gemini cli crash again
- Xbox gift card code damage
- Damage caused by the gemini cli crash
- ioctl(2) failed, EBADF (Bad File Descriptor)
- Feat: Support Bun as an alternative runtime/package manager for updates and extensions
- fatal error again!!!!
- ioctl error
- Critical Crash: ioctl(2) failed, EBADF in ShellExecutionService.resizePty
- ioctl(2) failed, EBADF
- v0.44.0 Regression: Critical crash with ioctl(2) failed, EBADF during PTY resize
- Crash on startup: ioctl(2) failed, EBADF in UnixTerminal.resize
- Crash: `ioctl(2) failed, EBADF` in `node-pty` during PTY resize on macOS
- Gemini CLI crashes with `ioctl(2) failed, EBADF` in `node-pty` during `resizePty`
- Remote Role
- ERROR ioctl(2) failed, EBADF /home/mich
- RangeError: Maximum call stack size exceeded
- EBADF Error during folder creationg broke session and terminal glitches
- MAIP / Gargoub Project - Mediterania - North Coast
- Gemini cli crash again in this morning
- ERROR ioctl(2) failed, EBADF
- Verified node install fails — Checksum verification failed (Cloud)
- The extended debugging key did not arrive during registration.
- CollaborationPane unmounts collaboration store on single-user instances, causing permanent "No network connection" state
- Workflow cannot be saved when the name contains "->" (Potentially malicious string)
- automation does not work and does not show an error
- Raj Ai Automation
- Default Data Loader: DOMMatrix is not defined error
- Feature: Per-node execution timestamp overlay on canvas during workflow run
- AI Agent + Vertex `gemini-3.5-flash`: 400 "missing thought_signature" on sequential multi-turn tool calls (post-#24982)
- PDF Loader in Pinecone Vector Store fails due to pdf-parse version conflict (v2 not supported)
- emailReadImap: add UID deduplication, batch size cap, and numeric uid enforcement
- Manual node execution fails with "Could not find a node" when autosave is disabled (N8N_WORKFLOWS_AUTOSAVE_DISABLED)
- Schedule Trigger stopped firing — workflow Published & active, manual executions succeed, no automated fires for 2+ hours
- [MCP SDK] create_workflow_from_code intermittently returns HTTP 500, often as a false negative (workflow persists anyway, causing duplicates on retry)
- Credential-load wedge: workflows using googleApi/jwtAuth credentials silently fail to execute after key rotation
- Google Sheets Trigger every minute is not working manual Execute is working sent email
- [BUG] Plugin marketplace MCP connector remains stuck "still connecting" when mcp-remote requires OAuth
- [redacted at user request]
- Opus 4.7 behavioral regression: loaded instruction-following discipline degraded in recent Claude Code/Cowork updates
- [BUG] Tailscale via Homebrew CLI + Mac App Store GUI, both Macs on macOS, Cowork blocked by VPN detector despite Tailscale being a mesh VPN with no traffic interception
- stopShellPty on tab switch kills active sessions (exit 143) — regression in May 27 build
- [BUG] Long URLs are broken into multiple lines and become unclickable in terminal output
- [BUG] claude rm/stop/reap SIGKILLs background session tree without SIGTERM grace, orphaning git index.lock and similar
- [BUG] Default git workflow in the system prompt was pushed without context or consent
- [MODEL] Inconsistent output quality / Ignoring instructions (overfitting and inappropriate repetition of Korean vocabulary)
- You've hit your weekly limit · resets May 31 at 5pm (Asia/Shanghai)
- Paid yearly subscription silently downgraded to Free with no user action
- [Regression v2.1.153] Plugin bash hooks fail with "echo: write error: Permission denied" on Windows (claude-mem, shell: "bash")
- [BUG] Connector toggles in conversation are not clickable — must click text label instead
- [remote-control] Input from mobile app/browser not reaching host session — output works fine
- Model fails to read/reference CLAUDE.md contents despite being loaded in context
- [BUG] Claude Desktop reinstall destroys Code chat history (transcripts + Recents) while regular Chat history, project files, and memory all survive
- Bypass mode clamps to Accept Edits even with the toggle ON (Claude Code Desktop 1.9255.2 / CC 2.1.149)
- [BUG] TUI input freezes randomly mid-typing — entire prompt becomes unresponsive for minutes
- [BUG] Cowork downloads Linux ELF binary instead of macOS binary on macOS Sonoma 14.8.7 — exit code 132 (SIGILL) on every session
- [Feature Request] Persistent project memory — sessions forget everything on close, forcing users to keep many sessions open
- [Bug] Thread context stale after sleep/resume, returns outdated date and calendar data
- [FEATURE] Add context window usage indicator and warning before auto-compaction
- [BUG] Dictation error: Invalid character in header content ["x-config-keyterms"] on Windows
- [Bug] Anthropic API Error: Server rate limiting despite normal usage
- Does delegating work to `claude -p` subprocesses reduce context accumulation in the parent session?
- [BUG] Claude Code hangs on M1 Mac when terminal says "opening browser to sign in" and browser opens
- [BUG] Claude_Preview MCP preview_start spawns dev server with main-repo cwd instead of session's worktree cwd
- [Bug] Anthropic API Error: Server rate limiting during request execution
- [Bug] Anthropic API Error: Server rate limiting on concurrent requests
- [Bug] Ultraplan ready notification fires before cloud agent completes execution
- [BUG] API 500 ERROR ALL THROUGHOUT THE DAY
- [BUG] Cowork: Live Artifacts folder path changed in 1.9255.2, no automatic migration from Documents\Claude\Artifacts
- [Bug] Auto-compact never triggers despite statusline reporting "100% context used" (v2.1.153, Max sub, 200K mode)
- [BUG] [Desktop / macOS] 'Open in → New Window' detached session: font renders smaller than main, no per-window controls, Cmd+/Cmd- keystrokes routed to main window instead
- Feature request: option to switch between classic and new minimal UI
- [Feature Request] Show timestamps for each message
- [BUG] Terminal corruption when permission prompt appears while navigating Agent Teams agent selection menu
- [FEATURE] Allow users to customize the background color of the Claude desktop app beyond the current light/dark theme presets.
- [BUG] Statusline not displaying on Windows [fixed]
- Background agent UI Stop button is a no-op for stuck agents — process keeps consuming tokens
- Background agents silently die on session pause/resume — no completion notification, no work recovery
- Add option to hide email address from welcome banner
- [BUG] SSH Remote: `projects` field in remote ~/.claude.json becomes null after desktop restart — jsonl files intact, UI shows 'No messages yet' for every session
- [Bug] Claude Code not applying fixes despite claiming to complete tasks
- billing is unfair and poorly documented
- [BUG] Claude Code on the web: declared plugins inactive on first session, require restart to fully load
- [BUG] Restore from archive deleted sessions instead of restoring them
- [BUG] M365 connector fails with AADSTS50011 in Cowork — localhost vs 127.0.0.1 redirect URI mismatch
- claude agents: workflow slash-commands missing from dispatch-input completion (regression-adjacent to #61424)
- Claude Desktop's Info.plist missing TCC usage strings, blocks all EventKit-based MCP servers
- False-positive safety blocks on self-administered governance amendments — request for owner-authority mode for verified professional users
- [BUG] Stop pushing "AUTO"-mode
- [DOCS] Plugin marketplace guide omits `skipLfs` option for git-based sources
- [DOCS] MCP docs omit combined startup notification for MCP server and connector authentication
- [DOCS] Agent view docs omit macOS Privacy & Security identity for background agents
- [DOCS] Npm update docs do not explain release-channel behavior for `claude update`
- [DOCS] Agent SDK docs omit `subagent_type: "claude"` worktree and output persistence behavior
- [DOCS] Background session docs omit `$CLAUDE_JOB_DIR` temp-file behavior
- [FR] mask env-var values in 'claude mcp get <server>' output
- [FR] subagent worktrees should not inherit stale local 'user.email' from prior dispatches
- [BUG] Windows: Grep tool leaks rg.exe + conhost.exe processes (~2000 zombies / 14 GB RAM in long sessions)
- [BUG] Stats dashboard "Peak hour" appears off by one hour
- [BUG] Diff highlight (teal SGR background) bleeds past changed text in 2.1.150–2.1.153
- [FEATURE] confirm before deleting session
- Plugin PostToolUse hooks still silently skip in Claude Desktop / Cowork (re-filing closed #51904)
- /code-review skill: silent fallback to main...HEAD reviews other people's commits, and JSON-only output is hard to read
- Monitor tool doesn't source the shell snapshot like Bash does; PATH-dependent tools (jq, sleep, etc.) fail in Monitor commands on macOS/Nix
- [Bug] Long input lines truncated with ellipsis while typing instead of wrapping in terminal UI
- [FEATURE] VS Code extension: Render submitted user messages as Markdown in chat
- OSC 52 copy from Claude TUI doesn't reach clipboard inside tmux (regression in 2.1.146–2.1.153)
- [BUG] RemoteTrigger create/update returns HTTP 400 with circular error: "event_type is required" / "unknown field event_type"
- [BUG] Option to hide or minimize the built-in "status footer" (multi-line debug/cost panel) [re-raise of #31475]
- [Bug] Feedback submissions being closed without review or action
- [FEATURE] Word-jump cursor navigation in Chat input (option+arrow / bindable actions)
- [FEATURE] ! shell mode: filesystem tab completion
- [BUG] API Error: Usage credits required for 1M context
- claude agents: OSC 52 clipboard emission broken in tmux (regression in 2.1.146–2.1.153)
- CLI crashes on macOS 15 M3 - exit code 1
- [FEATURE] Support Cmd+V image paste from clipboard
- [FEATURE] Enhance claude.ai M365 connector to support MS Planner
- [BUG] Slash command autocomplete hijacks pasted absolute file paths starting with /
- PreToolUse hook `if` filter false-positives on complex Bash commands
- [BUG] Diff panel hangs/whites out
- Feature Request: Support drag-and-drop for binary documents (.wps, .doc, .docx, .xlsx, .pdf) in VS Code extension
- [BUG] activation of 1M context in VSCode
- [FEATURE] Support i18n / language localization for built-in slash command outputs
- Ctrl+V para colar imagens deixou de funcionar no CLI (Windows, PowerShell)
- [FEATURE] Please add Norwegian (Bokmål/Nynorsk) language support to the Claude Code interface
- [BUG] OTel log events (claude_code.user_prompt, api_request_body, tool_decision, hook_execution_complete) emitted with empty trace_id/span_id while sibling spans correlate correctly
- [BUG] Cowork crashes on every message, no VM logs generated, missing AppData\Roaming\Claude
- [FEATURE] first-class session handoff + per-session token budgets for unattended runs
- [FEATURE] Smart paste: convert clipboard code to file reference chips (like Cursor)
- [Feature Request] Restore chat pin functionality to title chat submenu
- [BUG] SIGILL issues with version 2.1.153
- [BUG] Cowork plugin upload fails with generic "Plugin validation failed" when a `description` field in any SKILL.md frontmatter contains angle brackets (`<…>`)
- [BUG] Desktop App 2.1.144+: startup scanner deletes cliSessionId from claude-code-sessions local files on every launch — session not found on disk
- [Feature Request] Add keyboard shortcut to copy last message with proper formatting
- [MODEL] Opus 4.7 not 1M
- Allow naming/renaming background agents in `claude agents` view
- Stale worktrees in .claude/worktrees/ are never cleaned up, consuming massive disk space
- Agent worktrees are never cleaned up, silently consuming disk space
- Subagent worktrees not auto-cleaned when reviewer writes scratch files
- [Bug] Skill initialization hangs for extended duration in Plan Mode
- Claude Desktop writes malformed registry Run entry (nested escaped quotes) - crashes Windows Task Manager and other Run-key parsers
- IME candidate window shows at bottom-right corner instead of caret position (Windows CMD)
- [BUG] Pressing 'Escape' doesn't close the /BTW conversation when the main conversation is asking for approval
- [BUG] Opus 4.7 (1M) intermittently emits empty-string values for tool_use.input fields, killing the session
- FleetView agent UI shows "running" with incrementing elapsed time after agent has returned
- /doctor flags context-scoped cmd+c binding as macOS conflict (false positive)
- [BUG] Text Rendering in Elvish
- Desktop app: Bypass Permissions mode flips to Accept Edits on first prompt (M5 / macOS 26.5)
- [Workaround] Date-Weekday Verification Hook — Prevents Claude from writing wrong weekdays
- [BUG] Claude Code create c:/memfs directory without asking me.
- [BUG] Claude Code's Bash execution waits forever with no processes running
- [BUG] usage stays stuck waiting for 5 hr limit after upgrading to premium seat in team plan
- [Workflow tool] resume cache is unreachable for nontrivial workflows because LLM dispatchers can't transcribe args byte-exactly
- Code review (Preview): "Add a repository" shows no results for private GitHub org repos
- [BUG] /context commands blows up context
- [Feature Request] Add precache expiry hook to enable proactive compaction before token eviction
- [BUG] Context indicator shows 0% at session start despite ~20K+ tokens already loaded
- [Feature Request] Add semantic search for --resume session history
- [Feature Request] Add session search, tagging, and filtering capabilities
- [BUG] Cowork Dispatch reports "desktop not available" on Windows 11 while standard Cowork works normally
- [Bug] Claude Code provides incorrect suggestions with high confidence despite errors
- defaultMode: acceptEdits silently overrides per-path permissions.ask rules for Write/Edit
- [FEATUR configurable tip interval (e.g. tipIntervalSeconds: 30 in settings)E]
- Plugin marketplace fails to load: schema rejects 'displayName' key (v2.1.153)
- claude agents: in-session copy uses broken OSC 52 path while overview correctly uses tmux buffer
- [BUG] Plugin agent descriptions (and custom agents) load unconditionally into context — no parity with disable-model-invocation for skills
- Crashed ultrareview consumed a free credit despite producing zero findings
- [Bug] Character rendering issue - invisible or missing text display
- [BUG] Cowork: processo Claude Code encerra com código 3 — .claude.json não contém token de autenticação (Windows 11 25H2)
- [BUG] 2.1.153 silently discards tools/list response from rmcp 0.12.0 HTTP MCP server (works in 2.1.152, wire-identical handshake)
- VS Code extension: option to auto-resume last session when reopening a workspace folder
- [Bug] Conversation continuation failure
- [BUG] Cowork crashes every time I start a new chat or attempt to continue an existing one in any project. The error displayed is: "Claude Code è andato in crash
- [Bug] Unannounced quota changes
- Native update/install fails with 'socket connection was closed unexpectedly' behind proxy — undici TLS incompatibility
- [BUG] Session name reverting after manual change
- [BUG] 非正常思考，上下文过长时，一直显示思考，点击interrupt按钮失效
- Honor `tools:` frontmatter when an agent is invoked via `@mention` — strip `Task` only when the agent did not declare it
- macOS TCC popup still recurring on v2.1.153 — "2.1.153" would like to access data from other apps
- Claude Code leaks pty handles — exhausts pseudo-terminals on macOS after long session
- [Bug] Agent fails to execute or respond to user input
- [BUG] Persistent "Expecting value: line 1 column 1 (char 0)" JSON parse error after tool execution
- [Feature Request] Implement proactive unit test coverage recommendations for recurring bugs
- VS Code panel lacks status line + terminal lacks image paste in Codespaces, forcing a tradeoff
- `/powerup` only shows ~10 lessons — allow viewing the full catalog
- [Bug] Context contamination after auto-compact with unrelated email draft of Tejo/Sado Basin
- [Bug] VSCode terminal output displays corrupted text with garbled symbols
- [Feature Request] Add LaTeX/KaTeX math rendering to TUI
- [Bug] Sub-agent PR review results not validated by orchestrating agent
- Subagents on Pro 1M tier: trivial probes pass, real workloads fail at first tool call (probe-vs-workload divergence)
- Path-scoped rules and subdirectory CLAUDE.md not loaded when creating new files matching the pattern
- AskUserQuestion: cancelling during extended thinking poisons the whole session with 400 'thinking blocks cannot be modified' (2.1.153); concurrent prompts overwrite each other
- Ideas Missing from Claude Cowork Menu (Windows)
- [BUG_BOUNTY_SAFE_POC_2026] Prompt Injection RCE Test - Command Execution Proof
- [BUG] Cowork scheduled task: execution history row not showing after successful run
- Resuming an extended-thinking session fails permanently with 400 "thinking blocks cannot be modified" (transcript stores thinking text as empty but keeps signature)
- [Bug] Plugin-registered CwdChanged and FileChanged hooks don't fire (settings.json works) — v2.1.153
- Auto-archive on PR merge / branch delete — clarify autoArchiveSessions semantics or add dedicated opt-out
- `claude mcp add` echoes Authorization header value verbatim to stdout, leaks bearer tokens to terminal and session transcripts
- [BUG] Bug report — /insights skill, Claude Code The /insights skill outputs a malformed file path.
- Plugin slash commands render with '*'-inline format instead of two-column, despite matching official plugin shape
- [Bug] Unexpected long text generation without user input or goal
- [Bug] Thinking blocks causing task progression blocked without user modification
- [BUG] (Critical!) contamination by an unknown session simirlar to the report => [Bug] Context contamination after auto-compact with unrelated email draft of Tejo/Sado Basin #63137
- [Critical] Opus 4.7 Korean output degeneration — Korean grammar itself collapses in long contexts
- [BUG] Title: Autocompact buffer persists across /clear — wastes tokens for irrelevant old context
- [Bug] Auto-Compact loses user input before processing in conversation history
- Feature: per-invocation effort parameter + runtime session-config introspection for skills
- Auto-mode classifier mislabels Azure DevOps vote -5 as "Reject" when denying PR vote actions
- [BUG] Claude Desktop and Claude Code CLI never re-register MCP tools after OAuth 2.1 handshake on a remote HTTP server
- [BUG] Workspace file tags leak across sessions
- [BUG] Ink renderer crashes on Windows 11 build 26200 (Canary) duplicate banners, terminal mode leaks, mid-operation aborts
- [BUG] Claude Code Desktop issue
- PTY master fd leak in Claude desktop app exhausts macOS kern.tty.ptmx_max after ~2-3 days
- [BUG] Claude Code — Session Management after Unexpected Interruption
- [Windows] Cowork OpenTelemetry exporter does not initialize - zero events emitted to any destination, including loopback
- [Bug] Opus 4.7: 400 `thinking blocks ... cannot be modified` on long extended-thinking sessions, triggered by history-altering events (scheduled prompts / parallel tool-call cancellation)
- [BUG] API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited
- Multi-plugin custom marketplace: only first plugin registered in installed_plugins.json, skills don't load
- [BUG] Git push through the SDK's git proxy fan-outs into ~500 GitHub REST API calls, exhausting the 5,000/hour budget after a handful of pushes
- [BUG] Claude took liberties it really shouldn't with my global config
- [BUG] Agent window focus lost after navigating with arrow keys, causing scroll deadlock
- [BUG] `--model` flag silently ignored in interactive sessions (works in `--print` only)
- [BUG] Dispatch permanently shows "desktop appears offline" on Windows 11 - never worked on first use
- feat: support per-command enableWeakerNetworkIsolation as safer alternative to dangerouslyDisableSandbox
- /code-review outputs a raw JSON array instead of readable findings
- [BUG] Cowork — Additional allowed domains ignored on Team plan; same domain works on Pro plan
- Haiku
- [Bug] False positive blocking beneficial outcomes in tool execution
- 3P Bedrock SSO: credentials silently expire without triggering re-auth on day 2+
- CLAUDE_AUTOCOMPACT_PCT_OVERRIDE in settings.json env block silently ignored by autocompact logic
- Auto-compaction deletes main session JSONL before verifying summary completion, causing data loss
- [Bug] Claude Code not executing stated actions or producing expected results
- [FEATURE] Deferred Messages — Queue Input for End of Turn
- [BUG] Up/Down arrows in input box navigate history instead of moving cursor — regression in 2.1.149+
- Cancelling a parallel tool-call batch corrupts thinking blocks -> 400 "thinking blocks cannot be modified" permanently wedges the session
- Claude Code caused data loss, then contradicted itself about recovery (two incidents, one session)
- [Bug] Unclear error messages from Claude Code CLI
- [Bug] Agent tool rejecting due to context size limit exceeded
- claude agents: daemon and bg-spare processes spin at ~100% CPU when idle
- [BUG] Compaction fails with "context window limit" error even when context usage is low (e.g., 20%) — regression in v2.1.153
- Remote Control entitlement lost after May 27-28 incident — `Error: Remote Control is not yet enabled for your account` on active Max subscription
- PreToolUse hook exit code 2 does not block Write tool
- [Bug] Thinking blocks in latest assistant message are immutable
- GUI: dispatch file:// and custom-scheme clicks to OS shell handler
- Show current model in statusLine by default
- [Bug] Agent console becomes unresponsive to keyboard input after multiple agents initialized
- [FEATURE] PreToolUse hooks should have a way of updating the environment
- [Bug] Unable to start or use Claude Code CLI
- [BUG] Repository not visible in Claude Code web repo picker
- Session permanently wedged on 400 "thinking blocks cannot be modified" after parallel tool_results
- [Bug] @ autocomplete loses sibling repos after a file edit in multi-repo workspace
- Unclear error message when creating sub-agent without authentication
- [Bug] Anthropic API errors causing frequent failures and high token usage
- [BUG] @ mention file picker only shows packages, not individual files (desktop app - Code tab)
- [Bug] TUI panel footer remains sticky and consumes excessive terminal space
- PR-status polling exhausts GitHub GraphQL rate limit on repos with many open PRs
- [BUG] Windows: welcome panel not shown in some project folders (2.1.153)
- [Bug] Anthropic API Error: thinking blocks corrupted during context compaction with extended thinking enabled
- API 400 "thinking blocks cannot be modified" permanently bricks session during agent activation (interleaved thinking + tool use)
- Right-click Copy copies the whole message instead of the selection; pasted text retains dark background
- Mid-session model switch corrupts conversation when extended thinking is enabled (API 400: 'thinking blocks cannot be modified')
- [BUG] Markdown file links in chat output do not open files when clicked (VS Code extension)
- Stuck retry loop: `400 thinking blocks cannot be modified` on large interleaved-thinking turns using AskUserQuestion
- [FEATURE] Prompt user for approval before auto-compaction proceeds
- Custom MCP connectors not attachable to scheduled routines — no UUID discovery path
- [BUG] Claude in Chrome — Navigation blocked for teams.cloud.microsoft and outlook.cloud.microsoft after Microsoft domain migration**
- [BUG] Claude Desktop — Personal plugins panel renders list but is entirely non-interactive (macOS, v1.9255.2)
- [Bug] error when using Workflows
- [BUG] Persistent "update available" notification despite being on latest version
- [BUG] Sweep Agent from /code-review never completes
- [Bug] Tool calls not executing or returning results
- [FEATURE] Cloud-synced memory and settings across machines
- [Bug] Terminal UI freezes when Ctrl+O view exits during interactive prompt in plan mode
- Continuous api errors when using claude code with Opus 4.7 with thinking on low
- [Feature Request] Add support for installing and using previous Claude Code versions
- [Bug] Extended Thinking: Summarized thinking blocks fail signature validation when resent to API
- [Bug] Anthropic API Error: 'thinking' blocks cannot be modified
- [Bug] Anthropic API Error: Thinking blocks cannot be modified with extended thinking mode
- Feature request: Lazy/on-demand MCP server connections
- [Bug] Tool Arguments Parsed as String Instead of Object
- [Bug] Anthropic API Error: Insufficient context provided
- [Bug] Claude Opus occasionally uses moskovian(russian) orthography instead of Ukrainian in system-prompted responses
- Opus 4.8: backgrounded task completions (subagents AND Bash) crash with 400 "thinking blocks cannot be modified"
- [Bug] Opus 4.7 fabricates stable preferences ("my default") to rationalize arbitrary choices when challenged
- [Bug] Unable to update Claude Code CLI
- [BUG] Desktop app: /remote-control mints link + connects bridge (main.log) but in-chat link/QR panel never renders
- Feature: sessionColor and sessionName in .claude/settings.json
- [BUG] Anthropic API error: thinking blocks
- [FEATURE] Support Remote MCPs in Cowork as in Claude Code
- [Bug] Anthropic API Error: 400 Bad Request with Redacted Thinking - 0 4.7 & 4.8
- [Bug] Anthropic API Error: Cannot modify thinking blocks from different model versions
- Interleaved thinking + multi-tool turn corrupts thinking block (text blanked, signature kept) → permanent 400 'blocks must remain as they were'
- [BUG] Mode/permission changes mid-tool-loop (effortLevel: xhigh) poisons entire session
- Session failure log: Opus 4.6 ignores its own rules for an entire session
- [BUG] "400 Guardrail was enabled" error when using Claude Opus 4.8 with AWS Bedrock
- [Feature Request] Add subagent approach selection option to avoid accidental feedback
- Persistent 400 'thinking blocks in the latest assistant message cannot be modified' — interleaved thinking persisted with empty text + signature bricks sessions
- [BUG] DesktopvsApp
- [BUG] Opus 4.7 cache hit rate collapse after May 27 incident — Messages 1.1k→88.9k in 9 minutes, $630/session
- [Bug] Anthropic API Error: Invalid thinking block format
- [BUG] FUCK CLAUDE
- Opus 4.8 extended thinking: Stop hook block re-entry corrupts thinking blocks → 400
- [Bug] 4.8 Fails when accessing previous model history
- [Bug] Unintended File Modifications During Execution
- [DOCS] Model configuration docs omit lean system prompt default scope and model exceptions
- Add "Always allow globally" option to permission prompts
- Server-side model upgrade (Opus 4.7→4.8) wedges in-flight sessions with `thinking blocks cannot be modified` 400
- [DOCS] AskUserQuestion docs missing multiple-choice prompt decision threshold
- [DOCS] Agent view docs omit shell-command background session launch syntax
- [DOCS] Agent view dispatch input docs incorrectly imply `/logout` dispatches as a prompt
- [DOCS] Claude in Chrome docs omit connected-browser selection behavior
- [DOCS] Plugin docs omit `defaultEnabled: false` for opt-in plugins
- Feature Request: Customizable chat text colors for user and assistant messages
- [DOCS] `/plugin` Discover tab docs omit directory-based suggested plugin pins
- VSCode Chrome integration silently fails: 3 distinct bugs
- [DOCS] MCP stdio docs omit session environment variables
- [Bug] Anthropic API error on second request within session with Claude Opus 4.8
- Cowork emits a blank session "index" handoff on focus when a CLI session is paused awaiting input
- [DOCS] MCP docs omit `claude mcp list/get` pending-approval output for unapproved project servers
- [BUG] /compact fails with 400 error when last assistant turn contains thinking blocks
- [DOCS] `/claude-api` docs omit Opus 4.8 migration guidance
- [DOCS] Fast mode docs still recommend deprecated Opus 4.6 override variable
- [DOCS] Bash tool docs omit `$TMPDIR` consistency across sandboxed and unsandboxed commands
- [Bug] Anthropic API Error: 400 Bad Request on Extended Thinking
- [DOCS] Background session docs omit worktree-isolation behavior for spawned subagents
- Built-in mechanistic self-verification of verifiable claims (symmetric to the auto permission gate)
- [DOCS] Worktree docs do not clarify `worktree.baseRef: "head"` inside linked worktrees
- [BUG] Excessive RAM usage with multiple parallel chats (~10 sessions → 30 GB memory pressure, macOS OOM)
- [DOCS] Managed MCP policy docs omit invalid `allowedMcpServers`/`deniedMcpServers` entry behavior
- [DOCS] Effort docs omit `CLAUDE_CODE_ALWAYS_ENABLE_EFFORT` unsupported-model behavior
- Regression (2.1.147–2.1.150?): resuming an extended-thinking session after a CC update/model-switch → unrecoverable 400, session bricked
- [DOCS] Windows updater docs omit `claude.exe` in-use recovery guidance
- [DOCS] VS Code auto mode docs still tie mode-picker visibility to bypass-permissions setting
- [DOCS] MCP docs omit `/mcp` tool list and detail rendering behavior
- [DOCS] Fine-grained tool streaming docs still describe provider opt-in behavior
- bypassPermissions: session startup reads flat pref, GUI toggle writes per-account pref — they never sync
- [BUG] Claude Desktop Code tab causes disk write limit violation — 8.5GB in 11 min, macOS kills app (M5, v1.9659.1)
- Ultrareview v2.1.96: docs describe /tasks command + claude ultrareview --json subcommand that don't exist; findings hard to read after completion
- I'd be happy to help create a GitHub issue title, but I don't see the error message in your message. Could you please share the specific error you're encountering? That way I can generate an accurate and descriptive issue title for you.
- [BUG] Claude in Chrome `file_upload` rejects all scheduled-task sessions with misleading error (real cause: INVALID_SESSION)
- Extended thinking: signed thinking block 'cannot be modified' (400) permanently wedges session
- RTL text support for Hebrew (and Arabic) in Claude Code
- [Bug] Random errors occurring across multiple operations