vllm - 💡(How to fix) Fix [Bug]: CUDA illegal instruction in Mamba2 mixed prefill/decode path [1 participants]

🐛 Describe the bug

CUDA illegal instruction in Mamba2 mixed prefill/decode path; local fence before decode avoids it

I am seeing an intermittent CUDA illegal instruction when running a NemotronH Nano Omni VLM with vLLM V1 on H100s. The error surfaces in GPUModelRunner._to_list() at self.transfer_event.synchronize(), but the failing kernel appears to be upstream.

A very small local workaround appears to make the issue stable: add torch.cuda.synchronize() in MambaMixer2.conv_ssm_forward() after the prefill SSM-state writeback and before entering the decode branch, only when the batch contains both prefill and decode work.

I pushed an annotated commit showing the exact surfaced-error location and the narrow workaround:

https://github.com/vllm-project/vllm/commit/6bcdc655a0f8090205be1c08d9e4fbcb3d39ae0f

Environment

• GPU: NVIDIA H100 80GB
• vLLM: 0.20.0 / V1 engine
• Model: NemotronH_Nano_VL_V2
• Model type: hybrid Mamba2 + Attention + MoE multimodal VLM
• Workload: NeMo-RL GRPO rollout generation
• Relevant vLLM settings:
  • enforce_eager=True
  • attention_backend='FLASHINFER'
  • mamba_ssm_cache_dtype='float32'
  • max_num_batched_tokens=16384

Error

The failure consistently surfaces here:

File ".../vllm/v1/worker/gpu_model_runner.py", line 4288, in sample_tokens
  ) = self._bookkeeping_sync(...)
File ".../vllm/v1/worker/gpu_model_runner.py", line 3386, in _bookkeeping_sync
  valid_sampled_token_ids = self._to_list(sampled_token_ids)
File ".../vllm/v1/worker/gpu_model_runner.py", line 6919, in _to_list
  self.transfer_event.synchronize()
torch.AcceleratorError: CUDA error: an illegal instruction was encountered
CUDA kernel errors might be asynchronously reported at some other API call,
so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1

Does this look like a missing stream/event dependency around Mamba2 ssm_state writeback and decode selective_state_update()?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.