Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (aarch64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.8.0-1041-nvidia-64k-aarch64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   :
GPU models and configuration : GPU 0: NVIDIA GH200 480GB
Nvidia driver version        : 580.95.05
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         aarch64
CPU op-mode(s):                       64-bit
Byte Order:                           Little Endian
CPU(s):                               72
On-line CPU(s) list:                  0-71
Vendor ID:                            ARM
Model name:                           Neoverse-V2
Model:                                0
Thread(s) per core:                   1
Core(s) per cluster:                  72
Socket(s):                            -
Cluster(s):                           1
Stepping:                             r0p0
Frequency boost:                      disabled
CPU max MHz:                          3447.0000
CPU min MHz:                          81.0000
BogoMIPS:                             2000.00
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti
L1d cache:                            4.5 MiB (72 instances)
L1i cache:                            4.5 MiB (72 instances)
L2 cache:                             72 MiB (72 instances)
L3 cache:                             114 MiB (1 instance)
NUMA node(s):                         9
NUMA node0 CPU(s):                    0-71
NUMA node1 CPU(s):
NUMA node2 CPU(s):
NUMA node3 CPU(s):
NUMA node4 CPU(s):
NUMA node5 CPU(s):
NUMA node6 CPU(s):
NUMA node7 CPU(s):
NUMA node8 CPU(s):
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.4
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.1
[pip3] nvidia-cutlass-dsl-libs-base==4.4.1
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu129
[pip3] torchvision==0.25.0+cu129
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.17.1rc1.dev23+g76c6e6da0 (git sha: 76c6e6da0)
vLLM Build Flags:
  CUDA Archs: 8.7 8.9 9.0 10.0+PTX 12.0; ROCm: Disabled
GPU Topology:
        GPU0    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    0-71    0               1
NIC0    NODE     X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: rocep1s0f1

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=GPU-9302ac7f-9fb3-24c3-e0d1-7c60f4cf076d
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
TORCH_CUDA_ARCH_LIST=8.7 8.9 9.0 10.0+PTX 12.0
PYTORCH_ALLOC_CONF=expandable_segments:True
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.9.1
VLLM_WEIGHT_OFFLOADING_DISABLE_UVA=1
VLLM_ENABLE_CUDA_COMPATIBILITY=0
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64
VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

---

PYTORCH_ALLOC_CONF="expandable_segments:True" VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1 VLLM_WEIGHT_OFFLOADING_DISABLE_UVA=1 python3 -m vllm.entrypoints.openai.api_server --port 5000 --host 0.0.0.0 --download-dir /workspace/.cache/huggingface/hub --model zai-org/GLM-4.7-FP8 --tensor-parallel-size 1 --trust-remote-code --enable-chunked-prefill --enable-prefix-caching --max-num-seqs 32 --gpu-memory-utilization 0.75 --max-model-len 202752 --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45 --cpu-offload-gb 360

---

(EngineCore_DP0 pid=372) torch._dynamo.exc.Unsupported: Failed to trace builtin operator
(EngineCore_DP0 pid=372)   Explanation: Dynamo does not know how to trace builtin operator `setattr` with argument types ['OrderedDict', 'str', 'OrderedDict'] (has_kwargs False)
(EngineCore_DP0 pid=372)   Hint: Avoid calling builtin `setattr` with argument types ['OrderedDict', 'str', 'OrderedDict']. Consider using an equivalent alternative function/method to `setattr`.
(EngineCore_DP0 pid=372)   Hint: If you are attempting to call a logging function (e.g. `print`), you can try adding it to `torch._dynamo.config.reorderable_logging_functions`.
(EngineCore_DP0 pid=372)   Hint: Please report an issue to PyTorch.
(EngineCore_DP0 pid=372) 
(EngineCore_DP0 pid=372)   Developer debug context: builtin setattr [<class 'torch._dynamo.variables.dicts.ConstDictVariable'>, <class 'torch._dynamo.variables.constant.ConstantVariable'>, <class 'torch._dynamo.variables.dicts.ConstDictVariable'>] False
(EngineCore_DP0 pid=372) 
(EngineCore_DP0 pid=372)  For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0059.html

---

(EngineCore_DP0 pid=372) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=372) INFO 03-11 14:19:14 [weight_utils.py:561] Time spent downloading weights for zai-org/GLM-4.7-FP8: 3826.597504 seconds
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:   0% Completed | 0/93 [00:00<?, ?it/s]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:   1% Completed | 1/93 [00:05<08:26,  5.51s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:   2% Completed | 2/93 [00:06<03:56,  2.60s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:   3% Completed | 3/93 [00:06<02:31,  1.68s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:   4% Completed | 4/93 [02:31<1:26:32, 58.34s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:   5% Completed | 5/93 [02:45<1:01:52, 42.19s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:   6% Completed | 6/93 [03:00<47:38, 32.86s/it]
(EngineCore_DP0 pid=372)   
Loading safetensors checkpoint shards:   8% Completed | 7/93 [03:15<38:44, 27.03s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:   9% Completed | 8/93 [03:26<31:17, 22.08s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  10% Completed | 9/93 [03:39<26:50, 19.17s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  11% Completed | 10/93 [03:53<24:25, 17.65s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  12% Completed | 11/93 [04:08<22:52, 16.74s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  13% Completed | 12/93 [04:20<20:47, 15.40s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  14% Completed | 13/93 [04:32<19:08, 14.35s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  15% Completed | 14/93 [04:44<17:49, 13.54s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  16% Completed | 15/93 [04:57<17:24, 13.39s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  17% Completed | 16/93 [05:10<16:54, 13.17s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  18% Completed | 17/93 [05:23<16:54, 13.35s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  19% Completed | 18/93 [05:36<16:22, 13.11s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  20% Completed | 19/93 [05:49<16:02, 13.01s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  22% Completed | 20/93 [06:00<15:19, 12.60s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  23% Completed | 21/93 [06:14<15:30, 12.93s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  24% Completed | 22/93 [06:26<14:49, 12.52s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  25% Completed | 23/93 [06:38<14:31, 12.45s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  26% Completed | 24/93 [06:51<14:26, 12.56s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  27% Completed | 25/93 [07:05<14:45, 13.02s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  28% Completed | 26/93 [07:17<14:20, 12.84s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  29% Completed | 27/93 [07:30<14:05, 12.81s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  30% Completed | 28/93 [07:43<13:52, 12.81s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  31% Completed | 29/93 [07:54<13:13, 12.40s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  32% Completed | 30/93 [08:08<13:20, 12.70s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  33% Completed | 31/93 [08:19<12:45, 12.35s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  34% Completed | 32/93 [08:32<12:51, 12.65s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  35% Completed | 33/93 [08:44<12:23, 12.39s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  37% Completed | 34/93 [08:56<12:00, 12.22s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  38% Completed | 35/93 [09:08<11:47, 12.20s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  39% Completed | 36/93 [09:20<11:26, 12.05s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  40% Completed | 37/93 [09:32<11:21, 12.17s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  41% Completed | 38/93 [09:44<10:59, 11.99s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  42% Completed | 39/93 [09:56<10:50, 12.05s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  43% Completed | 40/93 [10:09<10:46, 12.20s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  44% Completed | 41/93 [10:23<11:01, 12.73s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  45% Completed | 42/93 [10:35<10:48, 12.72s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  46% Completed | 43/93 [10:48<10:28, 12.58s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  47% Completed | 44/93 [10:59<09:59, 12.24s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  48% Completed | 45/93 [11:11<09:46, 12.23s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  49% Completed | 46/93 [11:23<09:26, 12.05s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  51% Completed | 47/93 [11:36<09:25, 12.29s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  52% Completed | 48/93 [11:49<09:30, 12.68s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  53% Completed | 49/93 [12:01<08:59, 12.27s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  54% Completed | 50/93 [12:13<08:46, 12.25s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  55% Completed | 51/93 [12:26<08:40, 12.40s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  56% Completed | 52/93 [12:39<08:46, 12.85s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  57% Completed | 53/93 [12:52<08:35, 12.88s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  58% Completed | 54/93 [13:06<08:31, 13.12s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  59% Completed | 55/93 [13:18<08:08, 12.84s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  60% Completed | 56/93 [13:31<07:52, 12.76s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  61% Completed | 57/93 [13:44<07:42, 12.86s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  62% Completed | 58/93 [13:57<07:36, 13.04s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  63% Completed | 59/93 [14:10<07:18, 12.89s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  65% Completed | 60/93 [14:21<06:51, 12.48s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  66% Completed | 61/93 [14:35<06:47, 12.73s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  67% Completed | 62/93 [14:47<06:28, 12.54s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  68% Completed | 63/93 [15:01<06:27, 12.91s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  69% Completed | 64/93 [15:15<06:26, 13.33s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  70% Completed | 65/93 [15:29<06:19, 13.56s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  71% Completed | 66/93 [15:41<05:49, 12.95s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  72% Completed | 67/93 [15:53<05:32, 12.81s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  73% Completed | 68/93 [16:05<05:14, 12.58s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  74% Completed | 69/93 [16:18<05:02, 12.62s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  75% Completed | 70/93 [16:31<04:52, 12.71s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  76% Completed | 71/93 [16:44<04:40, 12.73s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  77% Completed | 72/93 [16:56<04:23, 12.53s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  78% Completed | 73/93 [17:09<04:14, 12.72s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  80% Completed | 74/93 [17:21<04:01, 12.70s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  81% Completed | 75/93 [17:33<03:43, 12.41s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  82% Completed | 76/93 [17:45<03:28, 12.24s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  83% Completed | 77/93 [17:57<03:17, 12.31s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  84% Completed | 78/93 [18:10<03:04, 12.33s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  85% Completed | 79/93 [18:23<02:57, 12.65s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  86% Completed | 80/93 [18:38<02:51, 13.20s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  87% Completed | 81/93 [18:52<02:41, 13.42s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  88% Completed | 82/93 [19:05<02:26, 13.27s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  89% Completed | 83/93 [19:17<02:10, 13.08s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  90% Completed | 84/93 [19:30<01:57, 13.11s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  91% Completed | 85/93 [19:44<01:45, 13.15s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  92% Completed | 86/93 [19:57<01:32, 13.18s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  94% Completed | 87/93 [20:08<01:15, 12.65s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  95% Completed | 88/93 [20:20<01:02, 12.40s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  96% Completed | 89/93 [20:33<00:49, 12.43s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  97% Completed | 90/93 [20:46<00:37, 12.60s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  98% Completed | 91/93 [20:58<00:24, 12.47s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards:  99% Completed | 92/93 [21:14<00:13, 13.66s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards: 100% Completed | 93/93 [21:35<00:00, 15.75s/it]
(EngineCore_DP0 pid=372) 
Loading safetensors checkpoint shards: 100% Completed | 93/93 [21:35<00:00, 13.93s/it]
(EngineCore_DP0 pid=372) 
(EngineCore_DP0 pid=372) INFO 03-11 14:40:50 [default_loader.py:293] Loading weights took 1295.29 seconds
(EngineCore_DP0 pid=372) INFO 03-11 14:40:50 [fp8.py:544] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore_DP0 pid=372) INFO 03-11 14:45:54 [gpu_model_runner.py:4579] Model loading took 2.92 GiB memory and 5596.184954 seconds
(EngineCore_DP0 pid=372) Traceback (most recent call last):
(EngineCore_DP0 pid=372)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=372)     self.run()
(EngineCore_DP0 pid=372)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=372)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1102, in run_engine_core
(EngineCore_DP0 pid=372)     raise e
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1088, in run_engine_core
(EngineCore_DP0 pid=372)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=372)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP0 pid=372)     return func(*args, **kwargs)
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 832, in __init__
(EngineCore_DP0 pid=372)     super().__init__(
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 120, in __init__
(EngineCore_DP0 pid=372)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore_DP0 pid=372)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP0 pid=372)     return func(*args, **kwargs)
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 243, in _initialize_kv_caches
(EngineCore_DP0 pid=372)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=372)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore_DP0 pid=372)     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 78, in collective_rpc
(EngineCore_DP0 pid=372)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=372)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore_DP0 pid=372)     return func(*args, **kwargs)
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore_DP0 pid=372)     return func(*args, **kwargs)
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 388, in determine_available_memory
(EngineCore_DP0 pid=372)     self.model_runner.profile_run()
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5529, in profile_run
(EngineCore_DP0 pid=372)     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore_DP0 pid=372)                                         ^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore_DP0 pid=372)     return func(*args, **kwargs)
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5223, in _dummy_run
(EngineCore_DP0 pid=372)     outputs = self.model(
(EngineCore_DP0 pid=372)               ^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 241, in __call__
(EngineCore_DP0 pid=372)     return self.runnable(*args, **kwargs)
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(EngineCore_DP0 pid=372)     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(EngineCore_DP0 pid=372)     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/glm4_moe.py", line 695, in forward
(EngineCore_DP0 pid=372)     hidden_states = self.model(
(EngineCore_DP0 pid=372)                     ^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 572, in __call__
(EngineCore_DP0 pid=372)     self.aot_compiled_fn = self.aot_compile(*args, **kwargs)
(EngineCore_DP0 pid=372)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/wrapper.py", line 206, in aot_compile
(EngineCore_DP0 pid=372)     return self._compiled_callable.aot_compile((args, kwargs))
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 832, in aot_compile
(EngineCore_DP0 pid=372)     return aot_compile_fullgraph(
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/aot_compile.py", line 195, in aot_compile_fullgraph
(EngineCore_DP0 pid=372)     capture_output = convert_frame.fullgraph_capture(model, args, kwargs)
(EngineCore_DP0 pid=372)                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/convert_frame.py", line 1208, in fullgraph_capture
(EngineCore_DP0 pid=372)     return _fullgraph_capture_frame(
(EngineCore_DP0 pid=372)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/convert_frame.py", line 1273, in _fullgraph_capture_frame
(EngineCore_DP0 pid=372)     raise e.with_traceback(None) from e.__cause__  # User compiler error
(EngineCore_DP0 pid=372)     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=372) torch._dynamo.exc.Unsupported: Failed to trace builtin operator
(EngineCore_DP0 pid=372)   Explanation: Dynamo does not know how to trace builtin operator `setattr` with argument types ['OrderedDict', 'str', 'OrderedDict'] (has_kwargs False)
(EngineCore_DP0 pid=372)   Hint: Avoid calling builtin `setattr` with argument types ['OrderedDict', 'str', 'OrderedDict']. Consider using an equivalent alternative function/method to `setattr`.
(EngineCore_DP0 pid=372)   Hint: If you are attempting to call a logging function (e.g. `print`), you can try adding it to `torch._dynamo.config.reorderable_logging_functions`.
(EngineCore_DP0 pid=372)   Hint: Please report an issue to PyTorch.
(EngineCore_DP0 pid=372) 
(EngineCore_DP0 pid=372)   Developer debug context: builtin setattr [<class 'torch._dynamo.variables.dicts.ConstDictVariable'>, <class 'torch._dynamo.variables.constant.ConstantVariable'>, <class 'torch._dynamo.variables.dicts.ConstDictVariable'>] False
(EngineCore_DP0 pid=372) 
(EngineCore_DP0 pid=372)  For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0059.html
(EngineCore_DP0 pid=372) 
(EngineCore_DP0 pid=372) from user code:
(EngineCore_DP0 pid=372)    File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/glm4_moe.py", line 471, in forward
(EngineCore_DP0 pid=372)     hidden_states, residual = layer(positions, hidden_states, residual)
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(EngineCore_DP0 pid=372)     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/offloader/uva.py", line 123, in forward
(EngineCore_DP0 pid=372)     for k, v in module.state_dict().items()
(EngineCore_DP0 pid=372)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 2256, in state_dict
(EngineCore_DP0 pid=372)     destination._metadata = OrderedDict()
(EngineCore_DP0 pid=372) 
(EngineCore_DP0 pid=372) Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(EngineCore_DP0 pid=372) 
[rank0[]:[W311 14:45:55.192536574 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "<frozen runpy>", line 198, in _run_module_as_main
(APIServer pid=1)   File "<frozen runpy>", line 88, in _run_code
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 682, in <module>
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=1)     return __asyncio.run(
(APIServer pid=1)            ^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=1)     return runner.run(main)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 642, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 656, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 101, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 142, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 912, in __init__
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 570, in __init__
(APIServer pid=1)     with launch_core_engines(
(APIServer pid=1)          ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 951, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1010, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

---

PYTORCH_ALLOC_CONF="expandable_segments:True" VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1 python3 -m vllm.entrypoints.openai.api_server --port 5000 --host 0.0.0.0 --download-dir /workspace/.cache/huggingface/hub --model zai-org/GLM-4.7-FP8 --tensor-parallel-size 1 --trust-remote-code --enable-chunked-prefill --enable-prefix-caching --max-num-seqs 32 --gpu-memory-utilization 0.75 --max-model-len 202752 --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45 --cpu-offload-gb 360

---

RuntimeError: Expected b.stride(0) == 1 to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

---

(APIServer pid=1) INFO 03-11 14:53:20 [utils.py:292]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.17.1rc1.dev23+g76c6e6da0
(APIServer pid=1) INFO 03-11 14:53:20 [utils.py:292]   █▄█▀ █     █     █     █  model   zai-org/GLM-4.7-FP8
(APIServer pid=1) INFO 03-11 14:53:20 [utils.py:292]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 03-11 14:53:20 [utils.py:292] 
(APIServer pid=1) INFO 03-11 14:53:20 [utils.py:228] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'glm47', 'host': '0.0.0.0', 'port': 5000, 'model': 'zai-org/GLM-4.7-FP8', 'trust_remote_code': True, 'max_model_len': 202752, 'download_dir': '/workspace/.cache/huggingface/hub', 'reasoning_parser': 'glm45', 'gpu_memory_utilization': 0.75, 'enable_prefix_caching': True, 'cpu_offload_gb': 360.0, 'max_num_seqs': 32, 'enable_chunked_prefill': True}
(APIServer pid=1) INFO 03-11 14:53:28 [model.py:532] Resolved architecture: Glm4MoeForCausalLM
(APIServer pid=1) INFO 03-11 14:53:28 [model.py:1562] Using max model len 202752
(APIServer pid=1) INFO 03-11 14:53:28 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1) INFO 03-11 14:53:28 [vllm.py:748] Asynchronous scheduling is enabled.
(EngineCore_DP0 pid=371) INFO 03-11 14:53:35 [core.py:101[] Initializing a V1 LLM engine (v0.17.1rc1.dev23+g76c6e6da0) with config: model='zai-org/GLM-4.7-FP8', speculative_config=None, tokenizer='zai-org/GLM-4.7-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=202752, download_dir='/workspace/.cache/huggingface/hub', load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='glm45', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=zai-org/GLM-4.7-FP8, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [8192[], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64[], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': ]}
(EngineCore_DP0 pid=371) INFO 03-11 14:53:36 [parallel_state.py:1395] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.244.231.84:60153 backend=nccl
(EngineCore_DP0 pid=371) INFO 03-11 14:53:36 [parallel_state.py:1717] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore_DP0 pid=371) INFO 03-11 14:53:36 [base.py:111] Offloader set to UVAOffloader
(EngineCore_DP0 pid=371) INFO 03-11 14:53:36 [gpu_model_runner.py:4496] Starting to load model zai-org/GLM-4.7-FP8...
(EngineCore_DP0 pid=371) INFO 03-11 14:53:36 [__init__.py:257] Selected CutlassFP8ScaledMMLinearKernel for CompressedTensorsW8A8Fp8
(EngineCore_DP0 pid=371) INFO 03-11 14:53:36 [deep_gemm.py:100] DeepGEMM E8M0 enabled on current platform.
(EngineCore_DP0 pid=371) INFO 03-11 14:53:37 [cuda.py:317] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore_DP0 pid=371) INFO 03-11 14:53:37 [flash_attn.py:593] Using FlashAttention version 3
(EngineCore_DP0 pid=371) INFO 03-11 14:53:37 [fp8.py:395] Using VLLM_CUTLASS Fp8 MoE backend out of potential backends: ['AITER', 'FLASHINFER_TRTLLM', 'FLASHINFER_CUTLASS', 'DEEPGEMM', 'VLLM_CUTLASS', 'TRITON', 'MARLIN', 'BATCHED_DEEPGEMM', 'BATCHED_VLLM_CUTLASS', 'BATCHED_TRITON', 'XPU'].
(EngineCore_DP0 pid=371) INFO 03-11 14:54:27 [uva.py:61] Total CPU offloaded parameters: 327.77
(EngineCore_DP0 pid=371) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore_DP0 pid=371) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore_DP0 pid=371) 
(EngineCore_DP0 pid=371) 
(EngineCore_DP0 pid=371) INFO 03-11 15:12:42 [default_loader.py:293] Loading weights took 1093.12 seconds
(EngineCore_DP0 pid=371) INFO 03-11 15:12:43 [fp8.py:544] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore_DP0 pid=371) INFO 03-11 15:14:02 [gpu_model_runner.py:4579] Model loading took 2.92 GiB memory and 1225.038495 seconds
(EngineCore_DP0 pid=371) INFO 03-11 15:14:24 [backends.py:988] Using cache directory: /root/.cache/vllm/torch_compile_cache/943d487e60/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=371) INFO 03-11 15:14:24 [backends.py:1048] Dynamo bytecode transform time: 21.82 s
(EngineCore_DP0 pid=371) INFO 03-11 15:14:31 [backends.py:371] Cache the graph of compile range (1, 8192) for later use
(EngineCore_DP0 pid=371) /usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:321: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
(EngineCore_DP0 pid=371)   warnings.warn(
(EngineCore_DP0 pid=371) INFO 03-11 15:14:42 [backends.py:387] Compiling a graph for compile range (1, 8192) takes 17.58 s
(EngineCore_DP0 pid=371)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore_DP0 pid=371)     return func(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore_DP0 pid=371)     return func(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 388, in determine_available_memory
(EngineCore_DP0 pid=371)     self.model_runner.profile_run()
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5529, in profile_run
(EngineCore_DP0 pid=371)     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore_DP0 pid=371)                                         ^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore_DP0 pid=371)     return func(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5223, in _dummy_run
(EngineCore_DP0 pid=371)     outputs = self.model(
(EngineCore_DP0 pid=371)               ^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 241, in __call__
(EngineCore_DP0 pid=371)     return self.runnable(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(EngineCore_DP0 pid=371)     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(EngineCore_DP0 pid=371)     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/glm4_moe.py", line 695, in forward
(EngineCore_DP0 pid=371)     hidden_states = self.model(
(EngineCore_DP0 pid=371)                     ^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 578, in __call__
(EngineCore_DP0 pid=371)     output = self.aot_compiled_fn(self, *args, **kwargs)
(EngineCore_DP0 pid=371)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/aot_compile.py", line 124, in __call__
(EngineCore_DP0 pid=371)     return self.fn(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/glm4_moe.py", line 452, in forward
(EngineCore_DP0 pid=371)     def forward(
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/caching.py", line 206, in __call__
(EngineCore_DP0 pid=371)     return self.optimized_call(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(EngineCore_DP0 pid=371)     return self._wrapped_call(self, *args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 455, in __call__
(EngineCore_DP0 pid=371)     raise e
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 442, in __call__
(EngineCore_DP0 pid=371)     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(EngineCore_DP0 pid=371)     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(EngineCore_DP0 pid=371)     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "<eval_with_key>.187", line 939, in forward
(EngineCore_DP0 pid=371)     submod_0 = self.submod_0(l_input_ids_, s72, l_self_modules_embed_tokens_parameters_weight_, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_scale_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_bias_, l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  l_input_ids_ = l_self_modules_embed_tokens_parameters_weight_ = l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_scale_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_bias_ = l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_ = None
(EngineCore_DP0 pid=371)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 241, in __call__
(EngineCore_DP0 pid=371)     return self.runnable(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 363, in __call__
(EngineCore_DP0 pid=371)     return range_entry.runnable(*args)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/standalone_compile.py", line 122, in __call__
(EngineCore_DP0 pid=371)     return self._compiled_fn(*args)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1181, in _fn
(EngineCore_DP0 pid=371)     return fn(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1148, in forward
(EngineCore_DP0 pid=371)     return compiled_fn(full_args)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 357, in runtime_wrapper
(EngineCore_DP0 pid=371)     all_outs = call_func_at_runtime_with_args(
(EngineCore_DP0 pid=371)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 134, in call_func_at_runtime_with_args
(EngineCore_DP0 pid=371)     out = normalize_as_list(f(args))
(EngineCore_DP0 pid=371)                             ^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1962, in __call__
(EngineCore_DP0 pid=371)     return self.compiled_fn(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 531, in wrapper
(EngineCore_DP0 pid=371)     return compiled_fn(runtime_args)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 729, in inner_fn
(EngineCore_DP0 pid=371)     outs = compiled_fn(args)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/output_code.py", line 638, in __call__
(EngineCore_DP0 pid=371)     return self.current_callable(inputs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 3220, in run
(EngineCore_DP0 pid=371)     out = model(new_inputs)
(EngineCore_DP0 pid=371)           ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371)   File "/tmp/torchinductor_root/nt/cntppjs763uam2bxo6pmdk6ahgr7vz444mfegphu6rkqztkfonvd.py", line 1225, in call
(EngineCore_DP0 pid=371)     torch.ops._C.cutlass_scaled_mm.default(buf4, buf5, arg4_1, buf6, arg5_1, arg6_1)
(EngineCore_DP0 pid=371)   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 819, in __call__
(EngineCore_DP0 pid=371)     return self._op(*args, **kwargs)
(EngineCore_DP0 pid=371)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=371) RuntimeError: Expected b.stride(0) == 1 to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
[rank0[]:[W311 15:14:48.056050248 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "<frozen runpy>", line 198, in _run_module_as_main
(APIServer pid=1)   File "<frozen runpy>", line 88, in _run_code
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 682, in <module>
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=1)     return __asyncio.run(
(APIServer pid=1)            ^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=1)     return runner.run(main)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 642, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 656, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 101, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 142, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 912, in __init__
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 570, in __init__
(APIServer pid=1)     with launch_core_engines(
(APIServer pid=1)          ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 951, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1010, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
stream closed: EOF for nrp-llm/gh200-vllm-inference-8469b458b-x97s5 (gh200-vllm-inference)