vllm - 💡(How to fix) Fix [Bug]: MTP speculative decoding crash with illegal memory access on long sequences (Qwen3.6-27B-FP8, v0.19.1) [1 participants]

All worker processes then fail with torch.AcceleratorError: CUDA error: an illegal memory access was encountered in gpu_model_runner.py line 1706 (prev_common_req_indices_tensor = torch.tensor(...)). torch.AcceleratorError: CUDA error: an illegal memory access was encountered

TL;DR

Disabling speculative decoding or reducing the number of speculative tokens may temporarily mitigate the crash issue with MTP speculative decoding in the Qwen3.6-27B-FP8 model.

Guidance

• Investigate the speculative decoding configuration, specifically the num_spec_tokens parameter, to determine if reducing its value can prevent the crash without significantly impacting performance.
• Verify that the issue is not related to GPU memory usage, as the reported usage is low (~5.4% KV cache), but consider monitoring GPU memory allocation during the crash to rule out any memory-related issues.
• Review the gpu_model_runner.py code, particularly line 1706, to understand the context of the torch.AcceleratorError and how it relates to the speculative decoding process.
• Test the model with a smaller input size or a different model to isolate if the issue is specific to the Qwen3.6-27B-FP8 model or the speculative decoding configuration.

Example

No specific code example can be provided without modifying the existing codebase, but reviewing the gpu_model_runner.py file and the speculative decoding configuration may offer insights into the cause of the crash.

Notes

The exact cause of the crash is unclear, but it appears to be related to the speculative decoding process. Disabling speculative decoding or reducing the number of speculative tokens may be a temporary workaround, but a more permanent solution will require further investigation into the gpu_model_runner.py code and the speculative decoding configuration.

Recommendation

Apply workaround: Reduce the num_spec_tokens value to a lower number (e.g., 1 or 2) to mitigate the crash issue while investigating the root cause. This may impact performance but can help prevent the crash and allow for further debugging.