Your current environment

============================== PyTorch Info

PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A

============================== Python Environment

Python version : 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 05:01:18) [GCC 11.3.0] (64-bit runtime) Python platform : Linux-6.17.0-1014-nvidia-aarch64-with-glibc2.39

============================== CUDA / GPU Info

Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GB10 Nvidia driver version : 580.142 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True

============================== CPU Info

Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Vendor ID: ARM Model name: Cortex-X925 Model: 1 Thread(s) per core: 1 Core(s) per socket: 10 Socket(s): 1 Stepping: r0p1 Frequency boost: disabled CPU(s) scaling MHz: 100% CPU max MHz: 3900.0000 CPU min MHz: 1378.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt Model name: Cortex-A725 Model: 1 Thread(s) per core: 1 Core(s) per socket: 10 Socket(s): 1 Stepping: r0p1 CPU(s) scaling MHz: 100% CPU max MHz: 2808.0000 CPU min MHz: 338.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt L1d cache: 1.3 MiB (20 instances) L1i cache: 1.3 MiB (20 instances) L2 cache: 25 MiB (20 instances) L3 cache: 24 MiB (2 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-19 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected

============================== Versions of relevant libraries

[pip3] flashinfer-python==0.6.8.post1 [pip3] numpy==2.2.0 [pip3] nvidia-cublas==13.1.0.3 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.19.0.56 [pip3] nvidia-cudnn-frontend==1.18.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.0 [pip3] nvidia-cutlass-dsl==4.4.2 [pip3] nvidia-cutlass-dsl-libs-base==4.4.2 [pip3] nvidia-ml-py==13.595.45 [pip3] nvidia-nccl-cu13==2.28.9 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] onnxruntime==1.24.4 [pip3] pyzmq==27.1.0 [pip3] sentence-transformers==5.3.0 [pip3] torch==2.11.0 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0 [pip3] torchcodec==0.11.0+cu130 [pip3] torchvision==0.26.0 [pip3] transformers==4.57.6 [pip3] triton==3.6.0 [conda] flashinfer-python 0.6.8.post1 pypi_0 pypi [conda] numpy 2.2.0 pypi_0 pypi [conda] nvidia-cublas 13.1.0.3 pypi_0 pypi [conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi [conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi [conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi [conda] nvidia-cudnn-cu13 9.19.0.56 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.18.0 pypi_0 pypi [conda] nvidia-cufft 12.0.0.61 pypi_0 pypi [conda] nvidia-cufile 1.15.1.6 pypi_0 pypi [conda] nvidia-curand 10.4.0.35 pypi_0 pypi [conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi [conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi [conda] nvidia-cusparselt-cu13 0.8.0 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.4.2 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-base 4.4.2 pypi_0 pypi [conda] nvidia-ml-py 13.595.45 pypi_0 pypi [conda] nvidia-nccl-cu13 2.28.9 pypi_0 pypi [conda] nvidia-nvjitlink 13.0.88 pypi_0 pypi [conda] nvidia-nvshmem-cu13 3.4.5 pypi_0 pypi [conda] nvidia-nvtx 13.0.85 pypi_0 pypi [conda] pyzmq 27.1.0 pypi_0 pypi [conda] sentence-transformers 5.3.0 pypi_0 pypi [conda] torch 2.11.0 pypi_0 pypi [conda] torch-c-dlpack-ext 0.1.5 pypi_0 pypi [conda] torchaudio 2.11.0 pypi_0 pypi [conda] torchcodec 0.11.0+cu130 pypi_0 pypi [conda] torchvision 0.26.0 pypi_0 pypi [conda] transformers 4.57.6 pypi_0 pypi [conda] triton 3.6.0 pypi_0 pypi

============================== vLLM Info

ROCM Version : Could not collect vLLM Version : 0.20.0 vLLM Build Flags: CUDA Archs: 12.0f; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NODE NODE NODE NODE 0-19 0 N/A NIC0 NODE X PIX NODE NODE NIC1 NODE PIX X NODE NODE NIC2 NODE NODE NODE X PIX NIC3 NODE NODE NODE PIX X

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: rocep1s0f0 NIC1: rocep1s0f1 NIC2: roceP2p1s0f0 NIC3: roceP2p1s0f1

============================== Environment Variables

TORCH_CUDA_ARCH_LIST=12.0f CUDA_VISIBLE_DEVICES=0 CUDA_VISIBLE_DEVICES=0 MAX_JOBS=4 VLLM_TARGET_DEVICE=cuda LD_LIBRARY_PATH=/usr/local/cuda-13.0/targets/sbsa-linux/lib:/usr/lib/aarch64-linux-gnu: CUDA_HOME=/usr/local/cuda-13.0 CUDA_HOME=/usr/local/cuda-13.0 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_wnm3

🐛 Describe the bug

vLLM 0.20.0 Regression: granite-speech-4.1-2b Audio Processing Broken

Summary

ibm-granite/granite-speech-4.1-2b audio input is silently ignored in vLLM 0.20.0. The same model, quantization, and request format works correctly with vLLM 0.17.x. All audio API paths are affected, including the dedicated /v1/audio/transcriptions endpoint.

I had built a version of vllm 0.17.0 using these instructions and installed it in my conda environment by copying the miniforge3/envs/vllm/bin/vllm into miniforge3/envs/msp/bin/.:

BUILD_VLLM.md

Environment

• vLLM version: 0.20.0 (regression — works on 0.17.x)
• Model: ibm-granite/granite-speech-4.1-2b
• Platform: NVIDIA DGX Spark, ARM64 (aarch64), Linux 6.17.0
• GPU: NVIDIA GB10 (120 GB)
• Quantization: --quantization fp8
• Launch command:

  vllm serve ibm-granite/granite-speech-4.1-2b \
      --api-key vllm_token_here \
      --quantization fp8 \
      --max-model-len 2048 \
      --port 8083 \
      --gpu-memory-utilization 0.25

Reproduction

1. Chat completions with input_audio — audio silently ignored

AUDIO_B64=$(base64 < test_audio.wav | tr -d '\n')

curl -s -X POST http://localhost:8083/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer vllm-token-here" \
  -d "{
    \"model\": \"ibm-granite/granite-speech-4.1-2b\",
    \"stream\": false,
    \"temperature\": 0.2,
    \"max_completion_tokens\": 200,
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"text\", \"text\": \"transcribe speech to text\"},
        {\"type\": \"input_audio\", \"input_audio\": {\"data\": \"$AUDIO_B64\", \"format\": \"wav\"}}
      ]
    }]
  }"

Result on vLLM 0.20.0:

{
  "choices": [{"message": {"content": "!!!!!!!!!!!!!!!!!!!..."}}],
  "usage": {"prompt_tokens": 25, "completion_tokens": 200}
}

prompt_tokens: 25 confirms only the text tokens were processed — the audio data was completely ignored. The model degenerates into repeating ! because it received no audio to transcribe.

Result on vLLM 0.17.x (same model, same flags, same request): Correct transcription with prompt_tokens in the hundreds.

2. Transcription endpoint — also broken

curl -s -X POST http://localhost:8083/v1/audio/transcriptions \
  -H "Authorization: Bearer vllm_token_here" \
  -F file=@test_audio.wav \
  -F model=ibm-granite/granite-speech-4.1-2b

Result: {"text":"!!!!!!!!!!!!!!!!!..."} — same degenerate output.

3. With --chat-template-content-format openai — assertion error

Adding --chat-template-content-format openai to the vLLM serve command causes vLLM to recognize the audio data, but the multimodal processor crashes:

AssertionError: Failed to apply prompt replacement for mm_items['audio'][0]

This happens because the model's built-in chat template ({{ message['content'] }}) doesn't include the <|audio|> placeholder token that vLLM 0.20's multimodal processor expects. Even providing a custom chat template that correctly renders <|audio|> still fails (the transcription endpoint, which bypasses chat templates, also produces garbage).

Root Cause Analysis

• vLLM 0.20.0 defaults --chat-template-content-format to auto, which resolves to string for this model
• In string mode, the input_audio content part is serialized as a Python dict string instead of being processed as multimodal data
• In openai mode, the audio IS extracted, but the multimodal processor's prompt replacement logic fails because the model's simple chat template doesn't generate <|audio|> placeholder tokens
• The /v1/audio/transcriptions endpoint (which has its own prompt generation) also fails, suggesting the audio encoder pipeline itself may be broken
• MediaConnector.fetch_audio() works correctly in isolation — the audio decoding is fine, the issue is in the model inference pipeline

Working Configuration (vLLM 0.17.x)

The identical model, quantization (fp8), and API request work correctly on vLLM 0.17.x without any --chat-template-content-format or --chat-template flags. The 0.17 → 0.20 multimodal pipeline refactor introduced this regression.

Impact

This is a complete blocker for using granite-speech-4.1-2b with vLLM 0.20.0 — no audio processing works through any API path.

python collect_env.py
Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (aarch64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : version 3.28.3
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 05:01:18) [GCC 11.3.0] (64-bit runtime)
Python platform              : Linux-6.17.0-1014-nvidia-aarch64-with-glibc2.39
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA GB10
Nvidia driver version        : 580.142
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  20
On-line CPU(s) list:                     0-19
Vendor ID:                               ARM
Model name:                              Cortex-X925
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      10
Socket(s):                               1
Stepping:                                r0p1
Frequency boost:                         disabled
CPU(s) scaling MHz:                      100%
CPU max MHz:                             3900.0000
CPU min MHz:                             1378.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
Model name:                              Cortex-A725
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      10
Socket(s):                               1
Stepping:                                r0p1
CPU(s) scaling MHz:                      100%
CPU max MHz:                             2808.0000
CPU min MHz:                             338.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
L1d cache:                               1.3 MiB (20 instances)
L1i cache:                               1.3 MiB (20 instances)
L2 cache:                                25 MiB (20 instances)
L3 cache:                                24 MiB (2 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-19
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.2.0
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] onnxruntime==1.24.4
[pip3] pyzmq==27.1.0
[pip3] sentence-transformers==5.3.0
[pip3] torch==2.11.0
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0
[pip3] torchcodec==0.11.0+cu130
[pip3] torchvision==0.26.0
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] flashinfer-python                           0.6.8.post1      pypi_0                pypi
[conda] numpy                                       2.2.0            pypi_0                pypi
[conda] nvidia-cublas                               13.1.0.3         pypi_0                pypi
[conda] nvidia-cuda-cupti                           13.0.85          pypi_0                pypi
[conda] nvidia-cuda-nvrtc                           13.0.88          pypi_0                pypi
[conda] nvidia-cuda-runtime                         13.0.96          pypi_0                pypi
[conda] nvidia-cudnn-cu13                           9.19.0.56        pypi_0                pypi
[conda] nvidia-cudnn-frontend                       1.18.0           pypi_0                pypi
[conda] nvidia-cufft                                12.0.0.61        pypi_0                pypi
[conda] nvidia-cufile                               1.15.1.6         pypi_0                pypi
[conda] nvidia-curand                               10.4.0.35        pypi_0                pypi
[conda] nvidia-cusolver                             12.0.4.66        pypi_0                pypi
[conda] nvidia-cusparse                             12.6.3.3         pypi_0                pypi
[conda] nvidia-cusparselt-cu13                      0.8.0            pypi_0                pypi
[conda] nvidia-cutlass-dsl                          4.4.2            pypi_0                pypi
[conda] nvidia-cutlass-dsl-libs-base                4.4.2            pypi_0                pypi
[conda] nvidia-ml-py                                13.595.45        pypi_0                pypi
[conda] nvidia-nccl-cu13                            2.28.9           pypi_0                pypi
[conda] nvidia-nvjitlink                            13.0.88          pypi_0                pypi
[conda] nvidia-nvshmem-cu13                         3.4.5            pypi_0                pypi
[conda] nvidia-nvtx                                 13.0.85          pypi_0                pypi
[conda] pyzmq                                       27.1.0           pypi_0                pypi
[conda] sentence-transformers                       5.3.0            pypi_0                pypi
[conda] torch                                       2.11.0           pypi_0                pypi
[conda] torch-c-dlpack-ext                          0.1.5            pypi_0                pypi
[conda] torchaudio                                  2.11.0           pypi_0                pypi
[conda] torchcodec                                  0.11.0+cu130     pypi_0                pypi
[conda] torchvision                                 0.26.0           pypi_0                pypi
[conda] transformers                                4.57.6           pypi_0                pypi
[conda] triton                                      3.6.0            pypi_0                pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.20.0
vLLM Build Flags:
  CUDA Archs: 12.0f; ROCm: Disabled; XPU: Disabled
GPU Topology:
  	GPU0	NIC0	NIC1	NIC2	NIC3	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NODE	NODE	NODE	NODE	0-19	0		N/A
NIC0	NODE	 X 	PIX	NODE	NODE				
NIC1	NODE	PIX	 X 	NODE	NODE				
NIC2	NODE	NODE	NODE	 X 	PIX				
NIC3	NODE	NODE	NODE	PIX	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: rocep1s0f0
  NIC1: rocep1s0f1
  NIC2: roceP2p1s0f0
  NIC3: roceP2p1s0f1

==============================
     Environment Variables
==============================
TORCH_CUDA_ARCH_LIST=12.0f
CUDA_VISIBLE_DEVICES=0
CUDA_VISIBLE_DEVICES=0
MAX_JOBS=4
VLLM_TARGET_DEVICE=cuda
LD_LIBRARY_PATH=/usr/local/cuda-13.0/targets/sbsa-linux/lib:/usr/lib/aarch64-linux-gnu:
CUDA_HOME=/usr/local/cuda-13.0
CUDA_HOME=/usr/local/cuda-13.0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_wnm3

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.