Your current environment

Collecting environment information...

    System Info

============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35

============================== PyTorch Info

PyTorch version : 2.10.0+cu129 Is debug build : False CUDA used to build PyTorch : 12.9 ROCM used to build PyTorch : N/A

============================== Python Environment

Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.8.0-107-generic-x86_64-with-glibc2.35

============================== CUDA / GPU Info

Is CUDA available : True CUDA runtime version : 12.9.86 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090

Nvidia driver version : 580.126.20 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Vendor ID: AuthenticAMD Model name: AMD EPYC 7402P 24-Core Processor CPU family: 23 Model: 49 Thread(s) per core: 1 Core(s) per socket: 48 Socket(s): 1 Stepping: 0 BogoMIPS: 5600.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean flushbyasid pausefilter pfthreshold v_vmsave_vmload vgif umip rdpid overflow_recov succor Virtualization: AMD-V Hypervisor vendor: KVM Virtualization type: full L1d cache: 3 MiB (48 instances) L1i cache: 3 MiB (48 instances) L2 cache: 24 MiB (48 instances) L3 cache: 768 MiB (48 instances) NUMA node(s): 8 NUMA node0 CPU(s): 0-5 NUMA node1 CPU(s): 6-11 NUMA node2 CPU(s): 12-17 NUMA node3 CPU(s): 18-23 NUMA node4 CPU(s): 24-29 NUMA node5 CPU(s): 30-35 NUMA node6 CPU(s): 36-41 NUMA node7 CPU(s): 42-47 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled Vulnerability Spec rstack overflow: Mitigation; SMT disabled Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected

============================== Versions of relevant libraries

[pip3] flashinfer-python==0.6.6 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.9.1.4 [pip3] nvidia-cuda-cupti-cu12==12.9.79 [pip3] nvidia-cuda-nvrtc-cu12==12.9.86 [pip3] nvidia-cuda-runtime-cu12==12.9.79 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.18.0 [pip3] nvidia-cufft-cu12==11.4.1.4 [pip3] nvidia-cufile-cu12==1.14.1.1 [pip3] nvidia-curand-cu12==10.3.10.19 [pip3] nvidia-cusolver-cu12==11.7.5.82 [pip3] nvidia-cusparse-cu12==12.5.10.65 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-cutlass-dsl==4.4.2 [pip3] nvidia-cutlass-dsl-libs-base==4.4.2 [pip3] nvidia-ml-py==13.595.45 [pip3] nvidia-nccl-cu12==2.27.5 [pip3] nvidia-nvjitlink-cu12==12.9.86 [pip3] nvidia-nvshmem-cu12==3.4.5 [pip3] nvidia-nvtx-cu12==12.9.79 [pip3] pyzmq==27.1.0 [pip3] torch==2.10.0+cu129 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.10.0+cu129 [pip3] torchvision==0.25.0+cu129 [pip3] transformers==4.57.6 [pip3] triton==3.6.0 [conda] Could not collect

============================== vLLM Info

ROCM Version : Could not collect vLLM Version : 0.18.1 vLLM Build Flags: CUDA Archs: 7.0 7.5 8.0 8.9 9.0 10.0 12.0; ROCm: Disabled GPU Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV4 0-47 0-7 N/A GPU1 NV4 X 0-47 0-7 N/A

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

============================== Environment Variables

NVIDIA_VISIBLE_DEVICES=0,1 OMP_NUM_THREADS=8 NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=NVL PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=0,1 CUDA_VISIBLE_DEVICES=0,1 MKL_NUM_THREADS=8 NCCL_SHM_DISABLE=0 NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 CUDA_VERSION=12.9.1 LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_ENABLE_CUDA_COMPATIBILITY=0 TORCH_CUDA_ARCH_LIST=7.0 7.5 8.0 8.9 9.0 10.0 12.0 VLLM_USAGE_SOURCE=production-docker-image PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

🐛 Describe the bug

Summary

Serving Qwen/Qwen3-32B-AWQ with tensor_parallel_size=2 on two RTX 3090 GPUs reliably triggers NVIDIA Xid 13: Out Of Range Address (illegal memory access) during CUDA graph replay at batch sizes > 8. The engine dies with CUDA error: an illegal memory access was encountered in _vllm_fa2_C.varlen_fwd or in the NCCL ProcessGroupNCCL::Watchdog. Consistently the fault is reported on PCI:0000:01:00 at TPC 0, SM 0 of varying GPCs (0-4), suggesting the first warps launched per GPC.

Workaround: Narrow CUDA graph capture to small batches only via --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8],"max_cudagraph_capture_size":8}'. Workload stays stable at 32 concurrent streams for extended periods. --enforce-eager also works but cuts aggregate throughput ~4×.

Reproduced on both vllm/vllm-openai:v0.19.0 and vllm/vllm-openai:v0.18.1.

Reproducer

docker compose excerpt (the failing config; swapping image tag between v0.18.1 and v0.19.0 both reproduce):

vllm-27b:
  image: vllm/vllm-openai:v0.19.0
  command:
    - serve
    - Qwen/Qwen3-32B-AWQ
    - --host
    - 0.0.0.0
    - --port
    - "8000"
    - --tensor-parallel-size
    - "2"
    - --max-model-len
    - "40960"
    - --gpu-memory-utilization
    - "0.75"
    - --max-num-seqs
    - "32"
    - --max-num-batched-tokens
    - "8192"
    - --enable-prefix-caching
    - --enable-chunked-prefill
    - --disable-custom-all-reduce
    - --reasoning-parser
    - qwen3
    - --enable-auto-tool-choice
    - --tool-call-parser
    - qwen3_coder
    - --served-model-name
    - qwen3-32b
  environment:
    NVIDIA_VISIBLE_DEVICES: "0,1"
    NCCL_P2P_LEVEL: NVL
    NCCL_IB_DISABLE: "1"
  shm_size: "32gb"

Workload to trigger: a handful of concurrent /v1/chat/completions requests with varying output length. In our stack this comes from Open WebUI plus a few scheduled agents; the crash usually fires within 30 seconds to 5 minutes of real traffic.

A synthetic trigger:

for i in $(seq 1 16); do
  curl -s -X POST http://localhost:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"qwen3-32b","messages":[{"role":"user","content":"Write 150 words about topic '$i'."}],"max_tokens":300}' &
done
wait

Observed failure

Worker traceback (representative; sometimes it surfaces in _vllm_fa2_C.varlen_fwd, sometimes in the NCCL watchdog):

(Worker_TP0 pid=126) ERROR 04-17 06:12:14 [multiproc_executor.py:949] WorkerProc hit an exception.
(Worker_TP0 pid=126) ERROR [multiproc_executor.py:949]     out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
(Worker_TP0 pid=126) ERROR [multiproc_executor.py:949] torch.AcceleratorError: CUDA error: an illegal memory access was encountered
(EngineCore pid=103) ERROR [core.py:1110] RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered'
(APIServer pid=1) ERROR [async_llm.py:707] vllm.v1.engine.exceptions.EngineDeadError

dmesg at the same timestamp:

NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 0): Out Of Range Address
NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 0, SM 0): Multiple Warp Errors
NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 0, SM 0): Out Of Range Address
NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 3, TPC 0, SM 0): Multiple Warp Errors
NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 4, TPC 0, SM 0): Out Of Range Address
NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 4, TPC 0, SM 0): Multiple Warp Errors
NVRM: Xid (PCI:0000:01:00): 43, VLLM::Worker channel 0x0000001e (process killed)

Every fault lands on TPC 0, SM 0 of the affected GPCs, consistent with the first warp launched per GPC, suggesting an indexing/pointer bug in the captured-graph replay rather than hardware.

After the crash, the other TP worker's CUDA kernel remains stuck at 100% SM utilization as a zombie (NCCL all-reduce hanging). The only way to reap it is to stop and remove the container.

Diagnostic isolation

• Hardware is healthy: Qwen/Qwen2.5-7B-Instruct on GPU 0 alone (--tensor-parallel-size 1) ran 5 consecutive generation requests cleanly with no Xid events. The same physical GPU that faults under TP=2 + Qwen3-32B-AWQ is fine under TP=1 + Qwen2.5-7B.
• Plain AWQ (no Marlin) is stable but very slow: --quantization awq runs stable at TP=2 but throughput drops ~4× vs Marlin (~4.8 tok/s per stream vs ~22 tok/s).
• --enforce-eager is stable: disables CUDA graph capture entirely, eliminating Xids. Throughput ~21 tok/s per stream.
• Narrow graph capture is stable and fast: --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8],"max_cudagraph_capture_size":8}' (Marlin kept, graphs captured only for small batches, larger batches fall back to eager via continuous batching) eliminates Xids and keeps ~22 tok/s per stream (~720 tok/s aggregate at 32 concurrent). This is my current production config.
• Wider capture also stable at small scale but regresses latency: [1,2,4,8,16] stayed stable in a 32-parallel smoke test but individual request latency roughly doubled. Did not probe further up the boundary.

Expected vs actual

• Expected: the default cudagraph_capture_sizes list (which includes 16, 24, 32, 40, 48, 56, 64) replays safely under TP=2 with Qwen3-32B-AWQ on Ampere.
• Actual: replay at these larger captured batch sizes produces out-of-range GPU memory accesses and kills the worker.

Notes

• Not Xid 79/48 (bus/VRAM): this is Xid 13, which NVIDIA documents as an application-caused SM warp fault.
• Not hardware: the same physical GPU handles TP=1 + Qwen2.5-7B cleanly with no Xid events.
• Model config: max_position_embeddings=40960, rope_scaling=null, sliding_window=null (as shipped in Qwen/Qwen3-32B-AWQ/config.json).
• Happy to provide VLLM_LOGGING_LEVEL=DEBUG or VLLM_TRACE_FUNCTION=1 traces if it would help pinpoint which op goes out-of-bounds at replay time.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.