==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.14.0-570.98.1.el9_6.x86_64-x86_64-with-glibc2.35
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H200 NVL
GPU 1: NVIDIA H200 NVL

Nvidia driver version        : 580.126.20
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           52 bits physical, 57 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  256
On-line CPU(s) list:                     0-255
Vendor ID:                               AuthenticAMD
Model name:                              AMD EPYC 9555 64-Core Processor
CPU family:                              26
Model:                                   2
Thread(s) per core:                      2
Core(s) per socket:                      64
Socket(s):                               2
Stepping:                                1
Frequency boost:                         enabled
CPU max MHz:                             4409.3750
CPU min MHz:                             1500.0000
BogoMIPS:                                6400.32
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid bus_lock_detect movdiri movdir64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d debug_swap amd_lbr_pmc_freeze
Virtualization:                          AMD-V
L1d cache:                               6 MiB (128 instances)
L1i cache:                               4 MiB (128 instances)
L2 cache:                                128 MiB (128 instances)
L3 cache:                                512 MiB (16 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-63,128-191
NUMA node1 CPU(s):                       64-127,192-255
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Reduced Speculation
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.6
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu129
[pip3] torchvision==0.25.0+cu129
[pip3] transformers==5.5.4
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.19.1
vLLM Build Flags:
  CUDA Archs: 7.0 7.5 8.0 8.9 9.0 10.0 12.0; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    GPU1    NIC0    NIC1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    NODE    NODE    64-127,192-255  1               N/A
GPU1    NODE     X      NODE    NODE    64-127,192-255  1               N/A
NIC0    NODE    NODE     X      PIX
NIC1    NODE    NODE    PIX      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1

==============================
     Environment Variables
==============================
CUDA_VERSION=12.9.1
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64
NVIDIA_CTK_LIBCUDA_DIR=/usr/lib64
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_GDRCOPY=enabled
NVIDIA_GDS=enabled
NVIDIA_MOFED=enabled
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
NVIDIA_VISIBLE_DEVICES=void
TORCH_CUDA_ARCH_LIST=7.0 7.5 8.0 8.9 9.0 10.0 12.0
VLLM_ENABLE_CUDA_COMPATIBILITY=0
VLLM_LOGGING_LEVEL=DEBUG
VLLM_USAGE_SOURCE=production-docker-image
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

---

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: genai-models
  name: google-gemma-4-31b-it-h200-vllm-deploy
  labels:
    app:  google-gemma-4-31b-it-3-deploy
    app.kubernetes.io/part-of:  google-gemma-4-31b-it-vllm
spec:
  replicas: 1
  revisionHistoryLimit: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app:  google-gemma-4-31b-it-vllm-pod
  template:
    metadata:
      labels:
        app:  google-gemma-4-31b-it-vllm-pod
    spec:
      serviceAccountName: genai-models-anyuid-serviceaccount
      securityContext:
        privileged: true
      volumes:
      - name: model
        persistentVolumeClaim:
          claimName: huggingface-pv-filesystem
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: "32G"
      containers:
      - name: google-gemma-4-31b-it-vllm-pod
        command: ["/bin/sh", "-c"]
        args:
          - >
            vllm serve google/gemma-4-31B-it
            --served-model-name google/gemma-4-31B-it
            --kv-cache-dtype fp8
            --max-model-len 200000
            --dtype auto
            --async-scheduling
            --tensor-parallel-size 2
            --trust-remote-code
            --gpu-memory-utilization 0.90
            --enable-chunked-prefill
            --enable-auto-tool-choice 
            --reasoning-parser gemma4 
            --tool-call-parser gemma4 
            --limit-mm-per-prompt '{"image": 2, "audio": 1}'
            --chat-template examples/tool_chat_template_gemma4.jinja
            --max-num-seqs 64
        env:
        - name: VLLM_LOGGING_LEVEL
          value: DEBUG

---

(Worker pid=647) DEBUG 05-04 14:38:15 [utils/nccl.py:34] Found nccl from library libnccl.so.2
(Worker pid=647) INFO 05-04 14:38:15 [distributed/device_communicators/pynccl.py:111] vLLM is using nccl==2.27.5
(APIServer pid=434) DEBUG 05-04 14:38:17 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.
(APIServer pid=434) DEBUG 05-04 14:38:27 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.

---

env:
        - name: NCCL_P2P_LEVEL
          value: "NVL"
        - name: NCCL_P2P_DISABLE
          value: "1"

---

(EngineCore pid=591) ERROR 05-04 08:51:40 [logging_utils/dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, num_skipped_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.002550310374067055, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=2182, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138] EngineCore encountered a fatal error.
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138] Traceback (most recent call last):
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 386, in get_response
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     status, result = mq.dequeue(timeout=dequeue_timeout)
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 755, in dequeue
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     with self.acquire_read(timeout, indefinite) as buf:
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     return next(self.gen)
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]            ^^^^^^^^^^^^^^
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 674, in acquire_read
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     self._spin_condition.wait(timeout_ms=read_timeout.timeout_ms())
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]                                          ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 631, in timeout_ms
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     raise TimeoutError
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138] TimeoutError
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138] 
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138] The above exception was the direct cause of the following exception:
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138] 
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138] Traceback (most recent call last):
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1129, in run_engine_core
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     engine_core.run_busy_loop()
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1170, in run_busy_loop
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     self._process_engine_step()
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1209, in _process_engine_step
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     outputs, model_executed = self.step_fn()
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]                               ^^^^^^^^^^^^^^
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 521, in step_with_batch_queue
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     model_output = future.result()
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]                    ^^^^^^^^^^^^^^^
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 90, in result
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     return super().result()
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]            ^^^^^^^^^^^^^^^^
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     return self.__get_result()
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     raise self._exception
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 94, in _wait_for_response
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     response = self.aggregate(self.get_response())
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]                               ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 388, in get_response
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138]     raise TimeoutError(f"RPC call to {method} timed out.") from e
(EngineCore pid=591) ERROR 05-04 08:51:40 [v1/engine/core.py:1138] TimeoutError: RPC call to sample_tokens timed out.
(EngineCore pid=591) DEBUG 05-04 08:51:40 [v1/executor/multiproc_executor.py:444] Triggering shutdown of workers
(EngineCore pid=591) DEBUG 05-04 08:51:40 [v1/executor/multiproc_executor.py:425] Worker Termination: allow workers to gracefully shutdown
(APIServer pid=433) ERROR 05-04 08:51:40 [v1/engine/async_llm.py:699] AsyncLLM output_handler failed.
(APIServer pid=433) ERROR 05-04 08:51:40 [v1/engine/async_llm.py:699] Traceback (most recent call last):
(APIServer pid=433) ERROR 05-04 08:51:40 [v1/engine/async_llm.py:699]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 655, in output_handler
(APIServer pid=433) ERROR 05-04 08:51:40 [v1/engine/async_llm.py:699]     outputs = await engine_core.get_output_async()
(APIServer pid=433) ERROR 05-04 08:51:40 [v1/engine/async_llm.py:699]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [v1/engine/async_llm.py:699]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 998, in get_output_async
(APIServer pid=433) ERROR 05-04 08:51:40 [v1/engine/async_llm.py:699]     raise self._format_exception(outputs) from None
(APIServer pid=433) ERROR 05-04 08:51:40 [v1/engine/async_llm.py:699] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(Worker_TP1 pid=664) DEBUG 05-04 08:51:40 [distributed/device_communicators/shm_broadcast.py:168] Canceling waiting reads on SHM Buffer
(Worker_TP0 pid=663) INFO 05-04 08:51:40 [v1/executor/multiproc_executor.py:775] Parent process exited, terminating worker queues
(Worker_TP0 pid=663) DEBUG 05-04 08:51:40 [distributed/device_communicators/shm_broadcast.py:168] Canceling waiting reads on SHM Buffer
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267] Error in chat completion stream generator.
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267] Traceback (most recent call last):
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 635, in chat_completion_stream_generator
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     async for res in result_generator:
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 574, in generate
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     out = q.get_nowait() or await q.get()
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]                             ^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 85, in get
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     raise output
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 655, in output_handler
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     outputs = await engine_core.get_output_async()
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 998, in get_output_async
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     raise self._format_exception(outputs) from None
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=433) DEBUG 05-04 08:51:40 [entrypoints/utils.py:312] create_error_response called with EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267] Error in chat completion stream generator.
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267] Traceback (most recent call last):
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 635, in chat_completion_stream_generator
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     async for res in result_generator:
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 574, in generate
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     out = q.get_nowait() or await q.get()
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]                             ^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 85, in get
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     raise output
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 635, in chat_completion_stream_generator
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     async for res in result_generator:
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 574, in generate
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     out = q.get_nowait() or await q.get()
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]                             ^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 85, in get
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     raise output
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 655, in output_handler
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     outputs = await engine_core.get_output_async()
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 998, in get_output_async
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     raise self._format_exception(outputs) from None
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=433) DEBUG 05-04 08:51:40 [entrypoints/utils.py:312] create_error_response called with EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267] Error in chat completion stream generator.
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267] Traceback (most recent call last):
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 635, in chat_completion_stream_generator
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     async for res in result_generator:
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 574, in generate
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     out = q.get_nowait() or await q.get()
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]                             ^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 85, in get
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     raise output
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 635, in chat_completion_stream_generator
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     async for res in result_generator:
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 574, in generate
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     out = q.get_nowait() or await q.get()
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]                             ^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 85, in get
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     raise output
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 635, in chat_completion_stream_generator
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     async for res in result_generator:
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 574, in generate
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     out = q.get_nowait() or await q.get()
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]                             ^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 85, in get
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     raise output
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 655, in output_handler
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     outputs = await engine_core.get_output_async()
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 998, in get_output_async
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267]     raise self._format_exception(outputs) from None
(APIServer pid=433) ERROR 05-04 08:51:40 [entrypoints/.../chat_completion/serving.py:1267] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=433) DEBUG 05-04 08:51:40 [entrypoints/utils.py:312] create_error_response called with EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.