Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.14.11-8-bpo12-pve-x86_64-with-glibc2.35
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition
Nvidia driver version        : 595.71.05
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           52 bits physical, 57 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  32
On-line CPU(s) list:                     0-31
Vendor ID:                               AuthenticAMD
Model name:                              AMD EPYC 9115 16-Core Processor
CPU family:                              26
Model:                                   2
Thread(s) per core:                      2
Core(s) per socket:                      16
Socket(s):                               1
Stepping:                                1
Frequency boost:                         enabled
CPU max MHz:                             4118.0000
CPU min MHz:                             1200.0000
BogoMIPS:                                5200.13
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid bus_lock_detect movdiri movdir64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d debug_swap amd_lbr_pmc_freeze
Virtualization:                          AMD-V
L1d cache:                               768 KiB (16 instances)
L1i cache:                               512 KiB (16 instances)
L2 cache:                                16 MiB (16 instances)
L3 cache:                                64 MiB (2 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-31
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; IBPB on VMEXIT only
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB on VMEXIT

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.11.post2
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.0
[pip3] nvidia-cutlass-dsl-libs-base==4.5.0
[pip3] nvidia-cutlass-dsl-libs-cu13==4.5.0
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] tokenspeed-triton==3.7.10.post20260505
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.8.1
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.21.1rc1.dev46+gb50646e5e (git sha: b50646e5e)
vLLM Build Flags:
  CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-31    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576
TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX
VLLM_NVFP4_GEMM_BACKEND=cutlass
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=13.0.2
VLLM_ENABLE_CUDA_COMPATIBILITY=0
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

---

(APIServer) INFO 05-18 17:24:44 [utils.py:306]        █     █     █▄   ▄█
(APIServer) INFO 05-18 17:24:44 [utils.py:306]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.21.1rc1.dev46+gb50646e5e
(APIServer) INFO 05-18 17:24:44 [utils.py:306]   █▄█▀ █     █     █     █  model   Qwen/Qwen3.6-27B-FP8
(APIServer) INFO 05-18 17:24:44 [utils.py:306]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer) INFO 05-18 17:24:44 [utils.py:306] 
(APIServer) INFO 05-18 17:24:44 [utils.py:240] non-default args: {'model_tag': 'Qwen/Qwen3.6-27B-FP8', 'default_chat_template_kwargs': {'preserve_thinking': False}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'disable_access_log_for_endpoints': '/health,/metrics,/ping,/v1/models', 'model': 'Qwen/Qwen3.6-27B-FP8', 'max_model_len': 147456, 'served_model_name': ['Q36_27B_FP8'], 'override_generation_config': {'temperature': 0.6, 'top_p': 0.95, 'top_k': 20, 'min_p': 0.0, 'presence_penalty': 0.0, 'repetition_penalty': 1.0}, 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.94, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': True, 'max_num_seqs': 32, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 1}}
(APIServer) WARNING 05-18 17:24:44 [envs.py:1895] Unknown vLLM environment variable detected: VLLM_BUILD_COMMIT
(APIServer) WARNING 05-18 17:24:44 [envs.py:1895] Unknown vLLM environment variable detected: VLLM_BUILD_PIPELINE
(APIServer) WARNING 05-18 17:24:44 [envs.py:1895] Unknown vLLM environment variable detected: VLLM_BUILD_URL
(APIServer) WARNING 05-18 17:24:44 [envs.py:1895] Unknown vLLM environment variable detected: VLLM_IMAGE_TAG
(APIServer) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(APIServer) INFO 05-18 17:24:45 [model.py:574] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer) INFO 05-18 17:24:45 [model.py:1705] Using max model len 147456
(APIServer) INFO 05-18 17:24:47 [cache.py:261] Using fp8_e4m3 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer) INFO 05-18 17:24:54 [model.py:574] Resolved architecture: Qwen3_5MTP
(APIServer) INFO 05-18 17:24:54 [model.py:1705] Using max model len 262144
(APIServer) INFO 05-18 17:24:54 [speculative.py:882] Overriding draft model max model len from 262144 to 147456
(APIServer) INFO 05-18 17:24:54 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer) WARNING 05-18 17:24:54 [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
(APIServer) INFO 05-18 17:24:54 [config.py:387] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer) INFO 05-18 17:24:54 [vllm.py:968] Asynchronous scheduling is enabled.
(APIServer) INFO 05-18 17:24:54 [kernel.py:267] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer) INFO 05-18 17:24:55 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(APIServer) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore) INFO 05-18 17:25:12 [core.py:109] Initializing a V1 LLM engine (v0.21.1rc1.dev46+gb50646e5e) with config: model='Qwen/Qwen3.6-27B-FP8', speculative_config=SpeculativeConfig(method='mtp', model='Qwen/Qwen3.6-27B-FP8', num_spec_tokens=1), tokenizer='Qwen/Qwen3.6-27B-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=147456, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e4m3, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Q36_27B_FP8, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto', linear_backend='auto')
(EngineCore) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(EngineCore) INFO 05-18 17:25:14 [parallel_state.py:1410] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.18.0.5:47893 backend=nccl
(EngineCore) INFO 05-18 17:25:14 [parallel_state.py:1723] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore) INFO 05-18 17:25:15 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(EngineCore) WARNING 05-18 17:25:15 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(EngineCore) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore) INFO 05-18 17:25:24 [gpu_model_runner.py:4976] Starting to load model Qwen/Qwen3.6-27B-FP8...
(EngineCore) INFO 05-18 17:25:24 [cuda.py:427] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore) INFO 05-18 17:25:24 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore) INFO 05-18 17:25:24 [__init__.py:520] Selected CutlassFp8BlockScaledMMKernel for Fp8LinearMethod
(EngineCore) INFO 05-18 17:25:24 [gdn_linear_attn.py:169] Using Triton/FLA GDN prefill kernel
(EngineCore) INFO 05-18 17:25:24 [cuda.py:372] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(EngineCore) INFO 05-18 17:25:26 [weight_utils.py:922] Filesystem type for checkpoints: ZFS. Checkpoint size: 28.75 GiB. Available RAM: 222.45 GiB.
(EngineCore) INFO 05-18 17:25:26 [weight_utils.py:945] Auto-prefetch is disabled because the filesystem (ZFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/66 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   6% Completed | 4/66 [00:00<00:01, 35.10it/s]
Loading safetensors checkpoint shards:  12% Completed | 8/66 [00:00<00:01, 34.98it/s]
Loading safetensors checkpoint shards:  18% Completed | 12/66 [00:00<00:01, 35.15it/s]
Loading safetensors checkpoint shards:  24% Completed | 16/66 [00:00<00:01, 35.28it/s]
Loading safetensors checkpoint shards:  30% Completed | 20/66 [00:00<00:01, 35.49it/s]
Loading safetensors checkpoint shards:  36% Completed | 24/66 [00:00<00:01, 35.30it/s]
Loading safetensors checkpoint shards:  42% Completed | 28/66 [00:00<00:01, 35.29it/s]
Loading safetensors checkpoint shards:  48% Completed | 32/66 [00:00<00:00, 35.21it/s]
Loading safetensors checkpoint shards:  55% Completed | 36/66 [00:01<00:00, 35.31it/s]
Loading safetensors checkpoint shards:  61% Completed | 40/66 [00:01<00:00, 35.24it/s]
Loading safetensors checkpoint shards:  67% Completed | 44/66 [00:01<00:00, 35.26it/s]
Loading safetensors checkpoint shards:  73% Completed | 48/66 [00:01<00:00, 35.37it/s]
Loading safetensors checkpoint shards:  79% Completed | 52/66 [00:01<00:00, 35.40it/s]
Loading safetensors checkpoint shards:  85% Completed | 56/66 [00:01<00:00, 35.38it/s]
Loading safetensors checkpoint shards:  91% Completed | 60/66 [00:01<00:00, 35.34it/s]
Loading safetensors checkpoint shards:  97% Completed | 64/66 [00:01<00:00, 35.27it/s]
Loading safetensors checkpoint shards: 100% Completed | 66/66 [00:02<00:00, 30.47it/s]
(EngineCore) 
(EngineCore) INFO 05-18 17:25:28 [default_loader.py:397] Loading weights took 2.22 seconds
(EngineCore) WARNING 05-18 17:25:28 [kv_cache.py:109] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(EngineCore) WARNING 05-18 17:25:28 [kv_cache.py:123] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(EngineCore) INFO 05-18 17:25:28 [gpu_model_runner.py:5000] Loading drafter model...
(EngineCore) INFO 05-18 17:25:28 [vllm.py:968] Asynchronous scheduling is enabled.
(EngineCore) INFO 05-18 17:25:28 [kernel.py:267] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore) INFO 05-18 17:25:28 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(EngineCore) INFO 05-18 17:25:28 [weight_utils.py:922] Filesystem type for checkpoints: ZFS. Checkpoint size: 28.75 GiB. Available RAM: 222.45 GiB.
Loading safetensors checkpoint shards:   0% Completed | 0/66 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  98% Completed | 65/66 [00:00<00:00, 578.97it/s]
Loading safetensors checkpoint shards: 100% Completed | 66/66 [00:00<00:00, 164.71it/s]
(EngineCore) 
(EngineCore) INFO 05-18 17:25:29 [default_loader.py:397] Loading weights took 0.44 seconds
(EngineCore) INFO 05-18 17:25:29 [llm_base_proposer.py:1321] Detected MTP model. Sharing target model embedding weights with the draft model.
(EngineCore) INFO 05-18 17:25:29 [llm_base_proposer.py:1377] Detected MTP model. Sharing target model lm_head weights with the draft model.
(EngineCore) INFO 05-18 17:25:29 [gpu_model_runner.py:5091] Model loading took 28.95 GiB memory and 4.777628 seconds
(EngineCore) INFO 05-18 17:25:29 [interface.py:649] Setting attention block size to 1584 tokens to ensure that attention page size is >= mamba page size.
(EngineCore) INFO 05-18 17:25:29 [interface.py:673] Padding mamba page size by 0.51% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore) INFO 05-18 17:25:30 [gpu_model_runner.py:6066] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore) INFO 05-18 17:25:48 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/f05144b57c/rank_0_0/backbone for vLLM's torch.compile
(EngineCore) INFO 05-18 17:25:48 [backends.py:1148] Dynamo bytecode transform time: 11.07 s
(EngineCore) INFO 05-18 17:25:51 [backends.py:378] Cache the graph of compile range (1, 8192) for later use
(EngineCore) INFO 05-18 17:26:25 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 35.70 s
(EngineCore) INFO 05-18 17:26:31 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/96e51fc61a83568e63631b14ba6b0c250d9ad7ee92c0eab0e392816a5e750732/rank_0_0/model
(EngineCore) INFO 05-18 17:26:31 [monitor.py:53] torch.compile took 54.07 s in total
(EngineCore) INFO 05-18 17:27:16 [monitor.py:81] Initial profiling/warmup run took 45.53 s
(EngineCore) INFO 05-18 17:27:17 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/f05144b57c/rank_0_0/eagle_head for vLLM's torch.compile
(EngineCore) INFO 05-18 17:27:17 [backends.py:1148] Dynamo bytecode transform time: 0.45 s
(EngineCore) INFO 05-18 17:27:25 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 7.19 s
(EngineCore) INFO 05-18 17:27:25 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/b3228728524f05b0328f6a99fe3c094f1fda9888915081cc5c3deee757f55578/rank_0_0/model
(EngineCore) INFO 05-18 17:27:25 [monitor.py:53] torch.compile took 8.38 s in total
(EngineCore) INFO 05-18 17:27:25 [monitor.py:81] Initial profiling/warmup run took 0.69 s
(EngineCore) WARNING 05-18 17:27:31 [kv_cache_utils.py:1157] Add 3 padding layers, may waste at most 6.25% KV cache memory
(EngineCore) WARNING 05-18 17:27:31 [compilation.py:1407] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention backend FlashInferBackend (support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE); setting cudagraph_mode=PIECEWISE
(EngineCore) INFO 05-18 17:27:31 [gpu_model_runner.py:6209] Profiling CUDA graph memory: PIECEWISE=19 (largest=128)
(EngineCore) INFO 05-18 17:27:33 [gpu_model_runner.py:6295] Estimated CUDA graph memory: 0.21 GiB total
(EngineCore) INFO 05-18 17:27:34 [gpu_worker.py:462] Available KV cache memory: 57.99 GiB
(EngineCore) INFO 05-18 17:27:34 [gpu_worker.py:477] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9400 is equivalent to --gpu-memory-utilization=0.9378 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9422. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore) WARNING 05-18 17:27:34 [kv_cache_utils.py:1157] Add 3 padding layers, may waste at most 6.25% KV cache memory
(EngineCore) INFO 05-18 17:27:34 [kv_cache_utils.py:1733] GPU KV cache size: 1,616,289 tokens
(EngineCore) INFO 05-18 17:27:34 [kv_cache_utils.py:1734] Maximum concurrency for 147,456 tokens per request: 10.96x
(EngineCore) INFO 05-18 17:27:34 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:00<00:00, 22.87it/s]
(EngineCore) INFO 05-18 17:27:37 [gpu_model_runner.py:6386] Graph capturing finished in 2 secs, took 0.23 GiB
(EngineCore) INFO 05-18 17:27:37 [gpu_worker.py:621] CUDA graph pool memory: 0.23 GiB (actual), 0.21 GiB (estimated), difference: 0.01 GiB (6.0%).
(EngineCore) INFO 05-18 17:27:37 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore) INFO 05-18 17:27:37 [core.py:299] init engine (profile, create kv cache, warmup model) took 127.39 s (compilation: 62.45 s)
(EngineCore) INFO 05-18 17:27:38 [kernel.py:267] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer) INFO 05-18 17:27:38 [api_server.py:599] Supported tasks: ['generate']
(APIServer) INFO 05-18 17:27:38 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer) WARNING 05-18 17:27:38 [model.py:1462] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'repetition_penalty': 1.0, 'temperature': 0.6, 'top_k': 20, 'top_p': 0.95, 'min_p': 0.0}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer) INFO 05-18 17:27:43 [hf.py:488] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer) INFO 05-18 17:27:55 [base.py:224] Multi-modal warmup completed in 12.594s
(APIServer) INFO 05-18 17:27:56 [base.py:224] Readonly multi-modal warmup completed in 0.400s
(APIServer) INFO 05-18 17:27:56 [api_server.py:603] Starting vLLM server on http://0.0.0.0:8000
(APIServer) INFO 05-18 17:27:56 [launcher.py:37] Available routes are:
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /load, Methods: GET
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /version, Methods: GET
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /health, Methods: GET
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /ping, Methods: GET
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /ping, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer) INFO 05-18 17:27:56 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer) INFO:     Started server process [1]
(APIServer) INFO:     Waiting for application startup.
(APIServer) INFO:     Application startup complete.
(APIServer) INFO:     192.168.128.3:59630 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer) INFO:     192.168.128.3:40312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore) WARNING 05-18 19:12:15 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _zero_kv_blocks_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:16 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:16 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _copy_page_indices_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:16 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _causal_conv1d_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:17 [jit_monitor.py:103] Triton kernel JIT compilation during inference: eagle_prepare_next_token_padded_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:17 [jit_monitor.py:103] Triton kernel JIT compilation during inference: batch_memcpy_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:26 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _fused_post_conv_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:27 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _causal_conv1d_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:28 [jit_monitor.py:103] Triton kernel JIT compilation during inference: fused_sigmoid_gating_delta_rule_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:28 [jit_monitor.py:103] Triton kernel JIT compilation during inference: expand_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore) WARNING 05-18 19:12:28 [jit_monitor.py:103] Triton kernel JIT compilation during inference: eagle_prepare_inputs_padded_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.