PR #38985: [feat] Support modelopt_mixed for Turing and Ampere via Marlin

• Repository: vllm-project/vllm
• Author: ir1ka
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/38985

Description (problem / solution / changelog)

Leverages Marlin kernels to enable modelopt_mixed quantization support, extending compatibility to NVIDIA Turing and Ampere architectures.

Due to limitations in Marlin, tensor dimensions must be aligned; however, the output dimensions of certain layers in the nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 model were misaligned. Consequently, zero-padding was applied. As a precautionary measure, zero-padding was restricted exclusively to layers utilizing FP8 per-tensor quantization.

@jinzhen-lin Could you please take a look at this PR when you have a moment?

#38776

Purpose

Support model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 for Turing or Ampere.

Test Plan

Test model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4. Validated implementation using nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4, ensuring functional correctness and performance stability on 4x RTX3090.

Command: vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 --served-model-name Nemotron-3-Super --tensor-parallel-size 4 --enable-expert-parallel --trust-remote-code --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser nemotron_v3 --kv-cache-memory-bytes 3G --max-model-len auto --async-scheduling --enable-prefix-caching --enable-chunked-prefill --max-num-seqs 4

Test Result

The vllm serve command launched successfully, and performance tests appear normal.

vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:15:32 [loggers.py:259] Engine 000: Avg prompt throughput: 2.9 tokens/s, Avg generation throughput: 46.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO:     127.0.0.1:35860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:15:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:15:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO:     127.0.0.1:46700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:18:42 [loggers.py:259] Engine 000: Avg prompt throughput: 2.2 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:18:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

Previously, the model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 failed to launch because the modelopt_mixed constraint restricted deployment to SM89 or newer architectures only.

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Changed files

• csrc/quantization/marlin/awq_marlin_repack.cu (modified, +6/-4)
• csrc/quantization/marlin/gptq_marlin_repack.cu (modified, +6/-4)
• vllm/model_executor/kernels/linear/scaled_mm/cutlass.py (modified, +4/-0)
• vllm/model_executor/layers/quantization/modelopt.py (modified, +13/-1)
• vllm/model_executor/layers/quantization/utils/marlin_utils_fp8.py (modified, +66/-11)