vllm - 💡(How to fix) Fix [Bug]: NGC vLLM 26.02 rejects Nemotron-3-Super-120B-A12B-NVFP4 — quant_algo MIXED_PRECISION not in whitelist [1 comments, 2 participants]

Your current environment

Hardware: NVIDIA DGX Spark (GB10, 128 GB unified memory, aarch64, SM 12.1)
Container: nvcr.io/nvidia/vllm:26.02-py3
vLLM: 0.15.1+befbc472 (NVIDIA Release 26.02, build 270699521)
CUDA: 13.1 (forward compat mode, driver 590.48.01, kernel driver 580.142)
PyTorch: 2.11.0a0+eb65b36914.nv26.2

collect_env.py was not run because vLLM crashes before initialization completes.

🐛 Describe the bug

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 cannot be served on a DGX Spark using the current NGC vLLM container. The model's config.json (produced by modelopt 0.43.0) sets quant_algo: "MIXED_PRECISION", which vLLM 0.15.1 rejects against a hardcoded whitelist.

The FP8 variant does not fit in 128 GB unified memory (~120 GB weights alone), so the NVFP4 variant is the only viable path for single-Spark deployment.

Reproduction:

docker run --gpus all --rm --ipc=host \
  -v /path/to/hf-cache:/root/.cache/huggingface \
  nvcr.io/nvidia/vllm:26.02-py3 \
  vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
    --dtype auto --kv-cache-dtype fp8 --tensor-parallel-size 1 \
    --attention-backend TRITON_ATTN --gpu-memory-utilization 0.85 \
    --enforce-eager --trust-remote-code

Error:

pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig
  Value error, ModelOpt currently only supports: ['FP8', 'FP8_PER_CHANNEL_PER_TOKEN',
  'FP8_PB_WO', 'NVFP4'] quantizations in vLLM. Please check the `hf_quant_config.json`
  file for your model's quant configuration.

Root cause: The model's quantization_config contains:

{
  "quant_algo": "MIXED_PRECISION",
  "quant_method": "modelopt",
  "quantized_layers": { "...40,961 NVFP4 layers, 139 FP8 layers..." }
}

The 139 FP8 layers are Mamba mixer projections and shared experts. vLLM 0.15.1 validates quant_algo against a whitelist that does not include MIXED_PRECISION.

The model card workaround does not work with the NGC container. The model card instructs users to:

pip install -U vllm --extra-index-url https://wheels.vllm.ai/097eb544e9a22810c9b7a59e586b61627b308362

Inside the NGC container, this replaces the pinned torch (2.11.0a0+nv26.2, CUDA 13.1) with PyPI torch 2.9.1 (CUDA 12.x), crashing with ImportError: libcudart.so.12: cannot open shared object file. Installing with --no-deps keeps NGC's torch but produces ModuleNotFoundError: No module named 'torch._opaque_base' because vLLM 0.16.0+ requires torch APIs absent in the NGC torch build.

Related PRs and issues:

• #35047 — "add mixed precision support for modelopt" (merged 2026-02-26). Shipped after the 26.02 NGC container was cut.
• #36312 — "Fixed modelopt mixed precision quant format loading" (open). Reports that even after #35047, MIXED_PRECISION still fails to parse.
• #35528 — W4A8_MXFP4_FP8 also rejected by the same whitelist.
• #30938 — FP8_PER_CHANNEL_PER_TOKEN / FP8_PB_WO similarly rejected.
• #34356 — Nemotron-3 FP8 MoE backend regression in 0.15.x.
• #35065 — Nemotron-3-Nano NVFP4 fails on Blackwell SM120 (no compatible MoE backend).

Possible interim workaround (untested): Patching the model's config.json to set quant_algo: "NVFP4" instead of "MIXED_PRECISION" may allow vLLM 0.15.1 to accept the model, since 99.7% of layers are NVFP4 and per-layer quant annotations are preserved. We have not yet verified whether the 139 FP8 layers load correctly under this override.

Asks:

1. Merge #36312 (or an equivalent fix) so that MIXED_PRECISION is recognized.
2. Ship an NGC vLLM container that includes the fix.
3. Update the NVFP4 model card to document the NGC container incompatibility until a fixed container is available.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

This issue was drafted by Claude (Opus 4.6) and steered by @algal.