VLLM_USE_MODELSCOPE=true VLLM_DISABLE_DEEP_GEMM=1 vllm serve /mnt/nvme1n1/DeepSeek-V4-Flash/hub/models/deepseek-ai/DeepSeek-V4-Flash
--served-model-name "DeepSeek-V4-Flash"
--gpu-memory-utilization 0.96
--trust-remote-code
--tensor-parallel-size $(ls /proc/driver/nvidia/gpus | wc -l)
--kv-cache-dtype fp8
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--enable-auto-tool-choice
--reasoning-parser deepseek_v4
--disable-custom-all-reduce
--enforce-eager
--host 0.0.0.0
--port 8000 (APIServer pid=100113) INFO 04-27 23:39:58 [utils.py:299] (APIServer pid=100113) INFO 04-27 23:39:58 [utils.py:299] █ █ █▄ ▄█ (APIServer pid=100113) INFO 04-27 23:39:58 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.20.0 (APIServer pid=100113) INFO 04-27 23:39:58 [utils.py:299] █▄█▀ █ █ █ █ model /mnt/nvme1n1/DeepSeek-V4-Flash/hub/models/deepseek-ai/DeepSeek-V4-Flash (APIServer pid=100113) INFO 04-27 23:39:58 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=100113) INFO 04-27 23:39:58 [utils.py:299] (APIServer pid=100113) INFO 04-27 23:39:58 [utils.py:233] non-default args: {'model_tag': '/mnt/nvme1n1/DeepSeek-V4-Flash/hub/models/deepseek-ai/DeepSeek-V4-Flash', 'enable_auto_tool_choice': True, 'tool_call_parser': 'deepseek_v4', 'host': '0.0.0.0', 'model': '/mnt/nvme1n1/DeepSeek-V4-Flash/hub/models/deepseek-ai/DeepSeek-V4-Flash', 'tokenizer_mode': 'deepseek_v4', 'trust_remote_code': True, 'enforce_eager': True, 'served_model_name': ['DeepSeek-V4-Flash'], 'reasoning_parser': 'deepseek_v4', 'tensor_parallel_size': 4, 'disable_custom_all_reduce': True, 'gpu_memory_utilization': 0.96, 'kv_cache_dtype': 'fp8'} (APIServer pid=100113) WARNING 04-27 23:39:58 [envs.py:1818] Unknown vLLM environment variable detected: VLLM_DISABLE_DEEP_GEMM (APIServer pid=100113) WARNING 04-27 23:39:58 [envs.py:1818] Unknown vLLM environment variable detected: VLLM_VERSION (APIServer pid=100113) INFO 04-27 23:39:58 [config.py:775] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM. (APIServer pid=100113) INFO 04-27 23:40:05 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL. (APIServer pid=100113) WARNING 04-27 23:40:05 [nixl_utils.py:34] NIXL is not available (APIServer pid=100113) WARNING 04-27 23:40:05 [nixl_utils.py:44] NIXL agent config is not available (APIServer pid=100113) INFO 04-27 23:40:06 [model.py:555] Resolved architecture: DeepseekV4ForCausalLM (APIServer pid=100113) INFO 04-27 23:40:06 [model.py:1680] Using max model len 1048576 (APIServer pid=100113) INFO 04-27 23:40:06 [cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor (APIServer pid=100113) INFO 04-27 23:40:06 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=8192. (APIServer pid=100113) INFO 04-27 23:40:06 [vllm.py:840] Asynchronous scheduling is enabled. (APIServer pid=100113) WARNING 04-27 23:40:06 [vllm.py:896] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none (APIServer pid=100113) WARNING 04-27 23:40:06 [vllm.py:914] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (APIServer pid=100113) INFO 04-27 23:40:06 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native']) (APIServer pid=100113) INFO 04-27 23:40:06 [vllm.py:1089] Cudagraph is disabled under eager mode (APIServer pid=100113) WARNING 04-27 23:40:06 [vllm.py:1248] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implemented the reasoning_start_str and reasoning_end_str. (APIServer pid=100113) INFO 04-27 23:40:06 [compilation.py:303] Enabled custom fusions: norm_quant, act_quant WARNING 04-27 23:40:12 [nixl_utils.py:34] NIXL is not available WARNING 04-27 23:40:12 [nixl_utils.py:44] NIXL agent config is not available (EngineCore pid=100533) INFO 04-27 23:40:12 [core.py:109] Initializing a V1 LLM engine (v0.20.0) with config: model='/mnt/nvme1n1/DeepSeek-V4-Flash/hub/models/deepseek-ai/DeepSeek-V4-Flash', speculative_config=None, tokenizer='/mnt/nvme1n1/DeepSeek-V4-Flash/hub/models/deepseek-ai/DeepSeek-V4-Flash', skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=1048576, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_v4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=DeepSeek-V4-Flash, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, moe_backend='auto') (EngineCore pid=100533) WARNING 04-27 23:40:12 [multiproc_executor.py:1029] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. (EngineCore pid=100533) INFO 04-27 23:40:12 [multiproc_executor.py:139] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.16.23.100 (local), world_size=4, local_world_size=4 WARNING 04-27 23:40:18 [nixl_utils.py:34] NIXL is not available WARNING 04-27 23:40:18 [nixl_utils.py:44] NIXL agent config is not available WARNING 04-27 23:40:18 [nixl_utils.py:34] NIXL is not available WARNING 04-27 23:40:18 [nixl_utils.py:44] NIXL agent config is not available WARNING 04-27 23:40:18 [nixl_utils.py:34] NIXL is not available WARNING 04-27 23:40:18 [nixl_utils.py:44] NIXL agent config is not available WARNING 04-27 23:40:19 [nixl_utils.py:34] NIXL is not available WARNING 04-27 23:40:19 [nixl_utils.py:44] NIXL agent config is not available (Worker pid=100744) INFO 04-27 23:40:19 [parallel_state.py:1402] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:59417 backend=nccl (Worker pid=100745) INFO 04-27 23:40:19 [parallel_state.py:1402] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:59417 backend=nccl (Worker pid=100747) INFO 04-27 23:40:19 [parallel_state.py:1402] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:59417 backend=nccl (Worker pid=100746) INFO 04-27 23:40:19 [parallel_state.py:1402] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:59417 backend=nccl (Worker pid=100744) INFO 04-27 23:40:20 [pynccl.py:111] vLLM is using nccl==2.28.9 (Worker pid=100744) WARNING 04-27 23:40:20 [symm_mem.py:66] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker pid=100747) WARNING 04-27 23:40:20 [symm_mem.py:66] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker pid=100746) WARNING 04-27 23:40:20 [symm_mem.py:66] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker pid=100745) WARNING 04-27 23:40:20 [symm_mem.py:66] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker pid=100744) INFO 04-27 23:40:21 [parallel_state.py:1715] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A (Worker_TP0 pid=100744) INFO 04-27 23:40:23 [gpu_model_runner.py:4777] Starting to load model /mnt/nvme1n1/DeepSeek-V4-Flash/hub/models/deepseek-ai/DeepSeek-V4-Flash... (Worker_TP2 pid=100746) [transformers] torch_dtype is deprecated! Use dtype instead! (Worker_TP1 pid=100745) [transformers] torch_dtype is deprecated! Use dtype instead! (Worker_TP0 pid=100744) INFO 04-27 23:40:23 [init.py:389] Selected CutlassFp8BlockScaledMMKernel for Fp8LinearMethod (Worker_TP0 pid=100744) [transformers] torch_dtype is deprecated! Use dtype instead! (Worker_TP3 pid=100747) [transformers] torch_dtype is deprecated! Use dtype instead! (Worker_TP0 pid=100744) INFO 04-27 23:40:23 [deepseek_v4_attention.py:614] Using DeepSeek's fp8_ds_mla KV cache format. (Worker_TP0 pid=100744) INFO 04-27 23:40:23 [mxfp4.py:495] Using 'MARLIN' Mxfp4 MoE backend. (Worker_TP0 pid=100744) INFO 04-27 23:40:23 [deepseek_v4_attention.py:983] Using FP8 indexer cache for Lighening Indexer. (Worker_TP0 pid=100744) INFO 04-27 23:40:23 [weight_utils.py:904] Filesystem type for checkpoints: EXT4. Checkpoint size: 148.66 GiB. Available RAM: 460.46 GiB. (Worker_TP0 pid=100744) INFO 04-27 23:40:23 [weight_utils.py:927] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. Loading safetensors checkpoint shards: 0% Completed | 0/46 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 4% Completed | 2/46 [00:00<00:10, 4.22it/s] Loading safetensors checkpoint shards: 7% Completed | 3/46 [00:00<00:13, 3.13it/s] Loading safetensors checkpoint shards: 9% Completed | 4/46 [00:01<00:14, 2.85it/s] Loading safetensors checkpoint shards: 11% Completed | 5/46 [00:01<00:15, 2.62it/s] Loading safetensors checkpoint shards: 13% Completed | 6/46 [00:02<00:15, 2.54it/s] Loading safetensors checkpoint shards: 15% Completed | 7/46 [00:02<00:15, 2.53it/s] Loading safetensors checkpoint shards: 17% Completed | 8/46 [00:03<00:15, 2.46it/s] Loading safetensors checkpoint shards: 20% Completed | 9/46 [00:03<00:14, 2.47it/s] Loading safetensors checkpoint shards: 22% Completed | 10/46 [00:03<00:14, 2.43it/s] Loading safetensors checkpoint shards: 24% Completed | 11/46 [00:04<00:14, 2.45it/s] Loading safetensors checkpoint shards: 26% Completed | 12/46 [00:04<00:14, 2.43it/s] Loading safetensors checkpoint shards: 28% Completed | 13/46 [00:05<00:13, 2.40it/s] Loading safetensors checkpoint shards: 30% Completed | 14/46 [00:05<00:13, 2.41it/s] Loading safetensors checkpoint shards: 33% Completed | 15/46 [00:05<00:12, 2.40it/s] Loading safetensors checkpoint shards: 35% Completed | 16/46 [00:06<00:12, 2.41it/s] Loading safetensors checkpoint shards: 37% Completed | 17/46 [00:06<00:11, 2.43it/s] Loading safetensors checkpoint shards: 39% Completed | 18/46 [00:07<00:11, 2.42it/s] Loading safetensors checkpoint shards: 41% Completed | 19/46 [00:07<00:11, 2.42it/s] Loading safetensors checkpoint shards: 43% Completed | 20/46 [00:07<00:10, 2.39it/s] Loading safetensors checkpoint shards: 46% Completed | 21/46 [00:08<00:10, 2.41it/s] Loading safetensors checkpoint shards: 48% Completed | 22/46 [00:08<00:09, 2.41it/s] Loading safetensors checkpoint shards: 50% Completed | 23/46 [00:09<00:09, 2.40it/s] Loading safetensors checkpoint shards: 52% Completed | 24/46 [00:09<00:09, 2.37it/s] Loading safetensors checkpoint shards: 54% Completed | 25/46 [00:10<00:08, 2.35it/s] Loading safetensors checkpoint shards: 57% Completed | 26/46 [00:10<00:08, 2.34it/s] Loading safetensors checkpoint shards: 59% Completed | 27/46 [00:10<00:08, 2.33it/s] Loading safetensors checkpoint shards: 61% Completed | 28/46 [00:11<00:07, 2.35it/s] Loading safetensors checkpoint shards: 63% Completed | 29/46 [00:11<00:07, 2.36it/s] Loading safetensors checkpoint shards: 65% Completed | 30/46 [00:12<00:06, 2.35it/s] Loading safetensors checkpoint shards: 67% Completed | 31/46 [00:12<00:06, 2.36it/s] Loading safetensors checkpoint shards: 70% Completed | 32/46 [00:13<00:06, 2.30it/s] Loading safetensors checkpoint shards: 72% Completed | 33/46 [00:13<00:05, 2.32it/s] Loading safetensors checkpoint shards: 74% Completed | 34/46 [00:13<00:05, 2.32it/s] Loading safetensors checkpoint shards: 76% Completed | 35/46 [00:14<00:04, 2.32it/s] Loading safetensors checkpoint shards: 78% Completed | 36/46 [00:14<00:04, 2.33it/s] Loading safetensors checkpoint shards: 80% Completed | 37/46 [00:15<00:03, 2.31it/s] Loading safetensors checkpoint shards: 83% Completed | 38/46 [00:15<00:03, 2.30it/s] Loading safetensors checkpoint shards: 85% Completed | 39/46 [00:16<00:03, 2.30it/s] Loading safetensors checkpoint shards: 87% Completed | 40/46 [00:16<00:02, 2.29it/s] Loading safetensors checkpoint shards: 89% Completed | 41/46 [00:17<00:02, 2.30it/s] Loading safetensors checkpoint shards: 91% Completed | 42/46 [00:17<00:01, 2.29it/s] Loading safetensors checkpoint shards: 93% Completed | 43/46 [00:17<00:01, 2.29it/s] Loading safetensors checkpoint shards: 96% Completed | 44/46 [00:18<00:00, 2.31it/s] Loading safetensors checkpoint shards: 100% Completed | 46/46 [00:18<00:00, 2.50it/s] (Worker_TP0 pid=100744) (Worker_TP0 pid=100744) INFO 04-27 23:40:42 [default_loader.py:384] Loading weights took 18.41 seconds (Worker_TP0 pid=100744) INFO 04-27 23:40:42 [mxfp4.py:1263] Using MoEPrepareAndFinalizeNoDPEPModular (Worker_TP0 pid=100744) INFO 04-27 23:40:45 [gpu_model_runner.py:4879] Model loading took 38.04 GiB memory and 21.058687 seconds (Worker_TP0 pid=100744) INFO 04-27 23:40:45 [interface.py:489] Setting kv cache block size to 256 for DEEPSEEK_SPARSE_SWA backend. (Worker_TP3 pid=100747) INFO 04-27 23:40:45 [interface.py:489] Setting kv cache block size to 256 for DEEPSEEK_SPARSE_SWA backend. (Worker_TP1 pid=100745) INFO 04-27 23:40:45 [interface.py:489] Setting kv cache block size to 256 for DEEPSEEK_SPARSE_SWA backend. (Worker_TP2 pid=100746) INFO 04-27 23:40:48 [interface.py:489] Setting kv cache block size to 256 for DEEPSEEK_SPARSE_SWA backend. (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] WorkerProc hit an exception. (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] Traceback (most recent call last): (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 957, in worker_busy_loop (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] output = func(*args, **kwargs) (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] return func(*args, **kwargs) (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 370, in determine_available_memory (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] self.model_runner.profile_run() (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5848, in profile_run (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] hidden_states, last_hidden_states = self._dummy_run( (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] return func(*args, **kwargs) (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5537, in _dummy_run (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] outputs = self.model( (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] return self._call_impl(*args, **kwargs) (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] return forward_call(*args, **kwargs) (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/deepseek_v4.py", line 1474, in forward (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] hidden_states = self.model( (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 467, in call (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] return self.forward(*args, **kwargs) (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/deepseek_v4.py", line 1263, in forward (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] hidden_states = layer( (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] return self._call_impl(*args, **kwargs) (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] return forward_call(*args, **kwargs) (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/deepseek_v4.py", line 1153, in forward (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] x, post, comb = self.hc_pre( (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/deepseek_v4.py", line 1124, in hc_pre (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre( (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/_ops.py", line 1269, in call (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] return self._op(*args, **kwargs) (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/layers/mhc.py", line 277, in mhc_pre (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] tf32_hc_prenorm_gemm( (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] File "/home/yafeng_wang/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/utils/deep_gemm.py", line 477, in tf32_hc_prenorm_gemm (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] return _tf32_hc_prenorm_gemm_impl( (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP2 pid=100746) ERROR 04-27 23:40:50 [multiproc_executor.py:962] RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/hyperconnection.hpp:56): Unsupported architecture

TL;DR

The most likely fix is to check the compatibility of the architecture used in the model with the tf32_hc_prenorm_gemm function.

Guidance

• Check the architecture used in the model and ensure it is compatible with the tf32_hc_prenorm_gemm function.
• Review the documentation for tf32_hc_prenorm_gemm to see if there are any specific architecture requirements.
• Consider updating the model or the tf32_hc_prenorm_gemm function to support the used architecture.
• If the issue persists, try to reproduce the error with a minimal example to isolate the problem.

Example

No specific code example can be provided without more context, but the error message suggests that the issue is related to the tf32_hc_prenorm_gemm function in the vllm/utils/deep_gemm.py file.

Notes

The error message indicates an "Unsupported architecture" error, which suggests that the issue is related to the compatibility of the model architecture with the tf32_hc_prenorm_gemm function. Without more information about the model and the function, it is difficult to provide a more specific solution.

Recommendation

Apply a workaround by checking the architecture used in the model and ensuring it is compatible with the tf32_hc_prenorm_gemm function. If the issue persists, consider updating the model or the function to support the used architecture.