Solo mode enabled. Skipping node detection.
Head Node: 127.0.0.1
Worker Nodes: 
Container Name: vllm_node
Image Name: vllm-node
Action: exec
Starting Head Node on 127.0.0.1...
e992683792b6536b09be675f825eda0be2df078699af16408353a02a7ba19411
Solo mode active: Skipping Ray cluster readiness check.
Executing command on head node: bash 
root@gx10-e9a9:/workspace/vllm# wget https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/collect_env.py
# For security purposes, please feel free to check the contents of collect_env.py before running it.
python collect_env.py
--2026-03-11 21:59:50--  https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/collect_env.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27835 (27K) [text/plain]
Saving to: ‘collect_env.py’

collect_env.py                                                                            100%[===================================================================================================================================================================================================================================>]  27.18K  --.-KB/s    in 0.007s  

2026-03-11 21:59:50 (3.72 MB/s) - ‘collect_env.py’ saved [27835/27835]

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.3 LTS (aarch64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version                : Could not collect
CMake version                : version 3.31.6
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0a0+a36e1d39eb.nv26.01.42222806
Is debug build               : False
CUDA used to build PyTorch   : 13.1
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Nov  6 2025, 13:44:16) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.17.0-1008-nvidia-aarch64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.1.115
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : GPU 0: NVIDIA GB10
Nvidia driver version        : 580.126.09
cuDNN version                : Probably one of the following:
/usr/lib/aarch64-linux-gnu/libcudnn.so.9.17.1
/usr/lib/aarch64-linux-gnu/libcudnn_adv.so.9.17.1
/usr/lib/aarch64-linux-gnu/libcudnn_cnn.so.9.17.1
/usr/lib/aarch64-linux-gnu/libcudnn_engines_precompiled.so.9.17.1
/usr/lib/aarch64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.17.1
/usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.17.1
/usr/lib/aarch64-linux-gnu/libcudnn_heuristic.so.9.17.1
/usr/lib/aarch64-linux-gnu/libcudnn_ops.so.9.17.1
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  20
On-line CPU(s) list:                     0-19
Vendor ID:                               ARM
BIOS Vendor ID:                          NVIDIA
Model name:                              Cortex-X925
BIOS Model name:                         GB10 Unknown CPU @ 3.9GHz
BIOS CPU family:                         258
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      10
Socket(s):                               1
Stepping:                                r0p1
Frequency boost:                         disabled
CPU(s) scaling MHz:                      100%
CPU max MHz:                             3900.0000
CPU min MHz:                             1378.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
Model name:                              Cortex-A725
BIOS Model name:                         GB10 Unknown CPU @ 3.9GHz
BIOS CPU family:                         258
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      10
Socket(s):                               1
Stepping:                                r0p1
CPU(s) scaling MHz:                      100%
CPU max MHz:                             2808.0000
CPU min MHz:                             338.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
L1d cache:                               1.3 MiB (20 instances)
L1i cache:                               1.3 MiB (20 instances)
L2 cache:                                25 MiB (20 instances)
L3 cache:                                24 MiB (2 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-19
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.5
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.1.0
[pip3] nvidia-cuda-cccl==13.1.115
[pip3] nvidia-cuda-runtime-cu13==0.0.0a0
[pip3] nvidia-cudnn-frontend==1.17.0
[pip3] nvidia-cutlass-dsl==4.4.1
[pip3] nvidia-cutlass-dsl-libs-base==4.4.1
[pip3] nvidia-dali-cuda130==1.53.0
[pip3] nvidia-libnvcomp-cu13==5.1.0.21
[pip3] nvidia-ml-py==13.590.44
[pip3] nvidia-modelopt==0.40.0
[pip3] nvidia-nvimgcodec-cu13==0.7.0.49
[pip3] nvidia-nvjpeg==13.0.2.28
[pip3] nvidia-nvjpeg2k-cu13==0.9.1.47
[pip3] nvidia-nvshmem-cu13==3.5.21
[pip3] nvidia-nvtiff-cu13==0.6.0.78
[pip3] nvidia-resiliency-ext==0.5.0
[pip3] onnx==1.18.0
[pip3] onnx-ir==0.1.14
[pip3] onnxscript==0.5.7
[pip3] optree==0.18.0
[pip3] pytorch-triton==3.6.0+git5261b273.nv26.1
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0a0+a36e1d39eb.nv26.1.42222806
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torch_tensorrt==2.10.0a0
[pip3] torchao==0.15.0+git1272f3cf
[pip3] torchdata==0.11.0
[pip3] torchprofile==0.0.4
[pip3] torchtitan==0.2.0+gite98ae995
[pip3] torchvision==0.25.0a0+6b56de1c.nv26.1.42222806
[pip3] transformers==5.3.0
[pip3] triton_kernels==1.0.0+git5261b273.nv26.1
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.17.0rc1.dev125+gc188749bc.d20260306 (git sha: c188749bc, date: 20260306)
vLLM Build Flags:
  CUDA Archs: 12.1a; ROCm: Disabled
GPU Topology:
  	GPU0	NIC0	NIC1	NIC2	NIC3	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NODE	NODE	NODE	NODE	0-19	0		N/A
NIC0	NODE	 X 	PIX	NODE	NODE				
NIC1	NODE	PIX	 X 	NODE	NODE				
NIC2	NODE	NODE	NODE	 X 	PIX				
NIC3	NODE	NODE	NODE	PIX	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: rocep1s0f0
  NIC1: rocep1s0f1
  NIC2: roceP2p1s0f0
  NIC3: roceP2p1s0f1

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
CUBLAS_VERSION=13.2.1.1
TORCHAO_BUILD_VERSION=+git1272f3cf
NVIDIA_REQUIRE_CUDA=cuda>=9.0
TORCHINDUCTOR_LOOP_ORDERING_AFTER_FUSION=0
TORCH_CUDA_ARCH_LIST=12.1a
NCCL_VERSION=2.29.stable.20260109
NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
TORCH_NCCL_USE_COMM_NONBLOCKING=0
CUDA_ARCH_LIST=8.0 8.6 9.0 10.0 11.0 12.0
NVIDIA_PRODUCT_NAME=PyTorch
TORCHTITAN_BUILD_VERSION=0.2.0+gite98ae995
CUDA_VERSION=13.1.1.006
PYTORCH_VERSION=2.10.0a0+a36e1d3
PYTORCH_BUILD_NUMBER=0
CUBLASMP_VERSION=0.7.0.125
CUDNN_FRONTEND_VERSION=1.17.0
CUDA_BINARY_LOADER_THREAD_COUNT=8
MAX_JOBS=16
CUDA_COMPONENT_LIST=cccl crt nvrtc driver-dev culibos-dev cudart cudart-dev nvcc tileiras
CUDNN_VERSION=9.17.1.4
PYTORCH_HOME=/opt/pytorch/pytorch
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/torch/lib:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NVIDIA_BUILD_ID=256811084
CUDA_DRIVER_VERSION=590.48.01
PYTORCH_BUILD_VERSION=2.10.0a0+a36e1d3
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
CUDA_MODULE_LOADING=LAZY
NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS=
NCCL_IGNORE_CPU_AFFINITY=1
NVIDIA_PYTORCH_VERSION=26.01
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

---

AsyncLLM output_handler failed
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [6,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.17.0rc1.dev125+gc188749bc.d20260306) with config: model='Sehyo/Qwen3.5-35B-A3B-NVFP4', speculative_config=SpeculativeConfig(method='mtp', model='Sehyo/Qwen3.5-35B-A3B-NVFP4', num_spec_tokens=1), tokenizer='Sehyo/Qwen3.5-35B-A3B-NVFP4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3-35B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '/root/.cache/vllm/torch_compile_cache/5dbc8b65ab', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [1024], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': '/root/.cache/vllm/torch_compile_cache/5dbc8b65ab/rank_0_0/eagle_head', 'fast_moe_cold_start': False, 'static_all_moe_layers': ['language_model.model.layers.0.mlp.experts', 'language_model.model.layers.1.mlp.experts', 'language_model.model.layers.2.mlp.experts', 'language_model.model.layers.3.mlp.experts', 'language_model.model.layers.4.mlp.experts', 'language_model.model.layers.5.mlp.experts', 'language_model.model.layers.6.mlp.experts', 'language_model.model.layers.7.mlp.experts', 'language_model.model.layers.8.mlp.experts', 'language_model.model.layers.9.mlp.experts', 'language_model.model.layers.10.mlp.experts', 'language_model.model.layers.11.mlp.experts', 'language_model.model.layers.12.mlp.experts', 'language_model.model.layers.13.mlp.experts', 'language_model.model.layers.14.mlp.experts', 'language_model.model.layers.15.mlp.experts', 'language_model.model.layers.16.mlp.experts', 'language_model.model.layers.17.mlp.experts', 'language_model.model.layers.18.mlp.experts', 'language_model.model.layers.19.mlp.experts', 'language_model.model.layers.20.mlp.experts', 'language_model.model.layers.21.mlp.experts', 'language_model.model.layers.22.mlp.experts', 'language_model.model.layers.23.mlp.experts', 'language_model.model.layers.24.mlp.experts', 'language_model.model.layers.25.mlp.experts', 'language_model.model.layers.26.mlp.experts', 'language_model.model.layers.27.mlp.experts', 'language_model.model.layers.28.mlp.experts', 'language_model.model.layers.29.mlp.experts', 'language_model.model.layers.30.mlp.experts', 'language_model.model.layers.31.mlp.experts', 'language_model.model.layers.32.mlp.experts', 'language_model.model.layers.33.mlp.experts', 'language_model.model.layers.34.mlp.experts', 'language_model.model.layers.35.mlp.experts', 'language_model.model.layers.36.mlp.experts', 'language_model.model.layers.37.mlp.experts', 'language_model.model.layers.38.mlp.experts', 'language_model.model.layers.39.mlp.experts', 'mtp.layers.0.mlp.experts']}, 
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-bdd97e729f194753-940909e7'],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[None],num_computed_tokens=[2163],num_output_tokens=[1801]), num_scheduled_tokens={chatcmpl-bdd97e729f194753-940909e7: 2}, total_num_scheduled_tokens=2, scheduled_spec_decode_tokens={chatcmpl-bdd97e729f194753-940909e7: [-1]}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0, 0, 0, 0], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=null, ec_connector_metadata=null)
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.00520833333333337, encoder_cache_usage=0.0, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113] Traceback (most recent call last):
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1104, in run_engine_core
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]     engine_core.run_busy_loop()
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1145, in run_busy_loop
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]     self._process_engine_step()
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1184, in _process_engine_step
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 499, in step_with_batch_queue
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]     model_output = future.result()
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]                    ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]     return self.__get_result()
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]            ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]     raise self._exception
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]     result = self.fn(*self.args, **self.kwargs)
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 250, in get_output
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113]     self.async_copy_ready_event.synchronize()
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113] torch.AcceleratorError: CUDA error: device-side assert triggered
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113] Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=277) ERROR 03-11 21:46:43 [core.py:1113] 
(EngineCore_DP0 pid=277) Process EngineCore_DP0:
(EngineCore_DP0 pid=277) Traceback (most recent call last):
(EngineCore_DP0 pid=277)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=277)     self.run()
(EngineCore_DP0 pid=277)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=277)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=277)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1115, in run_engine_core
(EngineCore_DP0 pid=277)     raise e
(EngineCore_DP0 pid=277)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1104, in run_engine_core
(EngineCore_DP0 pid=277)     engine_core.run_busy_loop()
(EngineCore_DP0 pid=277)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1145, in run_busy_loop
(EngineCore_DP0 pid=277)     self._process_engine_step()
(EngineCore_DP0 pid=277)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1184, in _process_engine_step
(EngineCore_DP0 pid=277)     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=277)                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=277)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 499, in step_with_batch_queue
(EngineCore_DP0 pid=277)     model_output = future.result()
(EngineCore_DP0 pid=277)                    ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=277)   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(EngineCore_DP0 pid=277)     return self.__get_result()
(EngineCore_DP0 pid=277)            ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=277)   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP0 pid=277)     raise self._exception
(EngineCore_DP0 pid=277)   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
(EngineCore_DP0 pid=277)     result = self.fn(*self.args, **self.kwargs)
(EngineCore_DP0 pid=277)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=277)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 250, in get_output
(EngineCore_DP0 pid=277)     self.async_copy_ready_event.synchronize()
(EngineCore_DP0 pid=277) torch.AcceleratorError: CUDA error: device-side assert triggered
(EngineCore_DP0 pid=277) Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=277) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=277) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=277) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=277) 
(APIServer pid=116) ERROR 03-11 21:46:43 [async_llm.py:707] AsyncLLM output_handler failed.
(APIServer pid=116) ERROR 03-11 21:46:43 [async_llm.py:707] Traceback (most recent call last):
(APIServer pid=116) ERROR 03-11 21:46:43 [async_llm.py:707]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 663, in output_handler
(APIServer pid=116) ERROR 03-11 21:46:43 [async_llm.py:707]     outputs = await engine_core.get_output_async()
(APIServer pid=116) ERROR 03-11 21:46:43 [async_llm.py:707]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=116) ERROR 03-11 21:46:43 [async_llm.py:707]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 1012, in get_output_async
(APIServer pid=116) ERROR 03-11 21:46:43 [async_llm.py:707]     raise self._format_exception(outputs) from None
(APIServer pid=116) ERROR 03-11 21:46:43 [async_llm.py:707] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362] Error in chat completion stream generator.
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362] Traceback (most recent call last):
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 686, in chat_completion_stream_generator
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]     async for res in result_generator:
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 582, in generate
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]     out = q.get_nowait() or await q.get()
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]                             ^^^^^^^^^^^^^
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 85, in get
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]     raise output
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 663, in output_handler
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]     outputs = await engine_core.get_output_async()
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 1012, in get_output_async
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362]     raise self._format_exception(outputs) from None
(APIServer pid=116) ERROR 03-11 21:46:43 [serving.py:1362] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
[rank0]:[W311 21:46:43.098829339 ProcessGroupNCCL.cpp:1565] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

---

./launch-cluster.sh --solo exec bash -lc '
export VLLM_USE_FLASHINFER_MOE_FP4=0;
SPEC="{\"method\":\"mtp\",\"num_speculative_tokens\":1}";
vllm serve Sehyo/Qwen3.5-35B-A3B-NVFP4 \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name Qwen3-35B \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 1024 \
  --kv-cache-dtype auto \
  --gpu-memory-utilization 0.55 \
  --generation-config vllm \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config "$SPEC"
'