vllm - 💡(How to fix) Fix [Bug]: Vision encoder crashes on SM100 (Jetson Thor) — `_vllm_fa2_C` compiled for SM80-only, no override available for vision encoder [5 comments, 4 participants]

Your current environment

Human Context

I'm running vLLM on an NVIDIA AGX Thor Developer Kit (built on the Blackwell architecture, SM110a, and listed as a supported production target in NVIDIA's NGC container release notes.

The issue is narrow and reproducible: we know that language model path works on this hardware because the setting --attention-backend TRITON_ATTN hides the SM80-compiled _vllm_fa2_C library from running. However, the vision encoder path has no equivalent escape, so any multimodal model that uses MMEncoderAttention crashes unconditionally at startup on SM100 hardware during the profiling pass before the server even starts accepting requests.

I've verified this across three different container environments (jetson-containers vLLM 0.19.0, NGC vLLM 26.02, and the NVIDIA NIM container), and the failure is consistent and deterministic. The fix appears conceptually straightforward — extend the attention backend override to MMEncoderAttention the same way it already works for the language model — though I recognize I may be missing architectural constraints that make this harder than it looks.

AI

Environment

• vLLM version: 0.19.0
• Hardware: NVIDIA AGX Thor Developer Kit (SM110a / Blackwell)
• CUDA: 13.2
• OS: Ubuntu 24.04 aarch64 SBSA / L4T r39.0 / JetPack 7.2
• Model: Qwen/Qwen3.5-35B-A3B (multimodal, qwen3_5_moe architecture)
• Container: jetson-containers vLLM build

Bug description

The language model path serves correctly on SM100 using --attention-backend TRITON_ATTN, which bypasses the SM80-compiled _vllm_fa2_C library. However, the vision encoder (MMEncoderAttention) unconditionally calls _vllm_fa2_C.varlen_fwd during the KV cache profiling pass (profile_run → embed_multimodal), with no equivalent backend override available. This causes a hard crash at startup on any SM100 device.

Error

File ".../vllm/model_executor/layers/attention/mm_encoder_attention.py", line 402, in forward_cuda
    return self._forward_fa(query, key, value, cu_seqlens, max_seqlen)
File ".../vllm/v1/attention/ops/vit_attn_wrappers.py", line 51, in flash_attn_maxseqlen_wrapper
    output = flash_attn_varlen_func(...)
File ".../vllm/vllm_flash_attn/flash_attn_interface.py", line 300, in flash_attn_varlen_func
    out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(...)
torch.AcceleratorError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
(cudaErrorUnsupportedPtxVersion)

Root cause

_vllm_fa2_C is a pre-compiled flash attention library targeting SM80. On SM100 it fails with cudaErrorUnsupportedPtxVersion. The language model path avoids this via --attention-backend TRITON_ATTN, which routes through Triton JIT instead. MMEncoderAttention has no equivalent escape hatch — forward_cuda calls _forward_fa unconditionally regardless of the configured attention backend.

Workaround

--language-model-only disables vision entirely and allows the model to serve text successfully. This is not acceptable for multimodal use cases.

Expected behavior

MMEncoderAttention should respect a backend override (e.g. --mm-attention-backend triton or falling back to Triton when _vllm_fa2_C is unavailable for the target architecture), consistent with how the language model attention path handles SM100.

Steps to reproduce

vllm serve Qwen/Qwen3.5-35B-A3B \
  --attention-backend TRITON_ATTN \
  --trust-remote-code \
  --limit-mm-per-prompt '{"image":4}'

Crashes during profile_run before serving begins. Reproducible on every startup on SM110a.

Additional context

• The NGC vLLM 26.02 container (nvcr.io/nvidia/vllm:26.02-py3) does not support qwen3_5_moe architecture (transformers version too old), so switching containers is not a viable workaround.
• The NVIDIA NIM container (nvcr.io/nim/qwen/qwen3.5-35b-a3b:1.7.0-variant) hits a different but related issue: ptxas-blackwell does not recognize sm_110a, suggesting the Thor sm variant is undertested across the ecosystem.
• SGLang's FA4 backend explicitly supports SM100 without this issue and may be the longer-term solution, but vLLM is the stable serving stack for this hardware configuration.

🐛 Describe the bug

Human Context

I'm running vLLM on an NVIDIA AGX Thor Developer Kit (built on the Blackwell architecture, SM110a, and listed as a supported production target in NVIDIA's NGC container release notes.

The issue is narrow and reproducible: we know that language model path works on this hardware because the setting --attention-backend TRITON_ATTN hides the SM80-compiled _vllm_fa2_C library from running. However, the vision encoder path has no equivalent escape, so any multimodal model that uses MMEncoderAttention crashes unconditionally at startup on SM100 hardware during the profiling pass before the server even starts accepting requests.

I've verified this across three different container environments (jetson-containers vLLM 0.19.0, NGC vLLM 26.02, and the NVIDIA NIM container), and the failure is consistent and deterministic. The fix appears conceptually straightforward — extend the attention backend override to MMEncoderAttention the same way it already works for the language model — though I recognize I may be missing architectural constraints that make this harder than it looks.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.