Your current environment

• vLLM: 0.18.2rc1.dev25+gef53395e2 (built from main, 2026-04-01, includes PRs #33417 and #29242)
• FlashInfer: 0.6.6
• PyTorch: 2.12.0a0+git6fcbf6d
• CUDA: 13.2
• GPU: NVIDIA GeForce RTX 5080 (SM 12.0, 16 GB VRAM)
• Driver: 595.45.04
• OS: Ubuntu 24.04, Linux 6.17.0-19-generic

Model

nvidia/Nemotron-Cascade-2-30B-A3B-NVFP4 — NemotronH hybrid architecture (Mamba2 + MoE + Attention), NVFP4 quantized via modelopt.

• 52 layers: 6 attention + 23 Mamba2 SSM + 23 MoE (128 routed experts, 6 active, 1 shared)
• ~18.4 GB on disk → requires --cpu-offload-gb 5 to fit on 16 GB VRAM

Describe the bug

All NVFP4 MoE backends produce numerically incorrect (garbage) output on SM120 when CPU offloading is enabled. The model loads successfully, the engine starts, and inference runs without errors — but the output is completely wrong (immediate EOS or random tokens).

Every MoE backend tested produces garbage:

Example outputs:

Suspected root cause: CPU offload + MoE interaction

The community has confirmed working NVFP4 MoE on SM120 without CPU offloading:

• RTX PRO 6000 (96 GB): 39 tok/s native FP4 MoE after CUTLASS fix (NVIDIA/cutlass#3096)
• RTX 5090 (32 GB): ~202 tok/s via llama.cpp GGUF variant

Our RTX 5080 (16 GB) requires 5 GB CPU offloading because the model is 18.4 GB. With --cpu-offload-gb 5, expert weights are split across GPU and CPU. The fused MoE kernel may not correctly handle experts that have been offloaded to CPU memory.

Additionally, both pure-attention NVFP4 models (Qwen3-4B-NVFP4) and hybrid non-MoE models (Qwen3.5-9B-NVFP4 with GDN layers) work correctly on the same GPU with the same vLLM build — even with CPU offloading. Only models with MoE layers produce garbage.

How to reproduce

#!/usr/bin/env python3
"""Minimal reproduction: NVFP4 MoE garbage on SM120 + CPU offload."""
import asyncio
import os

os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"
os.environ["VLLM_ALLOW_INSECURE_SERIALIZATION"] = "1"
os.environ["VLLM_ALLOW_LONG_MAX_MODEL_LEN"] = "1"
os.environ["FLASHINFER_CUDA_ARCH_LIST"] = "12.0f"  # Required for SM120 MoE JIT

async def main():
    from vllm import LLM, SamplingParams

    llm = LLM(
        model="nvidia/Nemotron-Cascade-2-30B-A3B-NVFP4",
        trust_remote_code=True,
        max_model_len=4096,
        gpu_memory_utilization=0.95,
        cpu_offload_gb=5,  # Required: 18.4 GB model > 16 GB VRAM
        enforce_eager=True,
        max_num_batched_tokens=4096,
    )

    outputs = llm.generate(
        ["The capital of France is"],
        SamplingParams(max_tokens=32, temperature=0),
    )
    print(f"Output: {outputs[0].outputs[0].text!r}")
    # Expected: "Paris..." or similar
    # Actual: "." or garbage

if __name__ == "__main__":
    asyncio.run(main())

To reproduce, you need:

1. An SM120 GPU with < 18.4 GB VRAM (RTX 5080 16 GB), forcing CPU offload
2. CUDA 13.2+ (for compute_120f support)
3. FlashInfer 0.6.6+

Additional notes

1. FlashInfer auto-detection bug: flashinfer/compilation_context.py auto-appends "a" suffix for SM major >= 9, producing compute_120a instead of compute_120f. The compute_120a arch does not support TMA WS grouped GEMM tactics needed for NVFP4 MoE, causing "illegal instruction" crashes. Workaround: FLASHINFER_CUDA_ARCH_LIST=12.0f.

2. hf_quant_config.json FP8 KV: The model ships with "kv_cache_quant_algo": "FP8" but provides no FP8 KV scaling factors (vLLM warns "Using KV cache scaling factor 1.0"). Setting this to "none" to use BF16 KV does not affect the MoE garbage — the issue is purely in the MoE path.

3. CPU offload assertion: vLLM's gpu_model_runner.py had an assertion blocking CPU offload with hybrid models (InputBatch reinit). Current main has this as a warning, which is correct — the model loads fine.

Before submitting a new issue...

• I have searched for similar issues and couldn't find any duplicates.
• I have verified that the issue persists on the latest vLLM main branch.
• I have provided all necessary details including reproduction steps.