vllm - 💡(How to fix) Fix [Bug]: RuntimeError: flashinfer_fp8_blockscale_gemm fails on H100 NVL MIG 3g.47gb with Qwen3.6-35B-A3B-FP8 [1 participants]

Your current environment

Environment:

• vLLM version: v0.19.1
• GPU: NVIDIA H100 NVL, MIG slice 3g.47gb (47 GB)
• Model: Qwen/Qwen3.6-35B-A3B-FP8
• quantization: fp8, kv-cache-dtype: fp8

Error: RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at "/pytorch/c10/cuda/CUDACachingAllocator.cpp":1154

Stacktrace: torch.ops.vllm.flashinfer_fp8_blockscale_gemm.default(...) → run_deepgemm → torch.empty → CUDA allocator assert

Reproducible: Yes, consistently on MIG slice. enforce-eager does not help.

🐛 Describe the bug

vLLM fails to start with Qwen3.6-35B-A3B-FP8 on H100 NVL MIG slice (3g.47gb, 47 GB) with RuntimeError in flashinfer_fp8_blockscale_gemm during profile_run().

Startup command: python3 -m vllm.entrypoints.openai.api_server
--model /models/qwen36-35b-a3b-fp8
--quantization fp8
--kv-cache-dtype fp8
--max-model-len 65536
--gpu-memory-utilization 0.90
--enforce-eager

Error: RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at "/pytorch/c10/cuda/CUDACachingAllocator.cpp":1154

Full traceback ends at: torch.ops.vllm.flashinfer_fp8_blockscale_gemm.default(buf15, arg12_1, arg13_1, 128, True) → _flashinfer_fp8_blockscale_gemm_impl → torch.cond → run_deepgemm → torch.empty → CUDA assert

Notes:

• --enforce-eager does not resolve the issue
• With --enforce-eager, a secondary error occurs: only 0.14 GiB KV cache available after profiling
• Model loads successfully (34.26 GiB), crash happens during profile_run()
• MIG context may be relevant — CUDA memory allocator behaves differently on MIG slices
• torch_compile cache cleared before testing, same result

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.