pytorch - 💡(How to fix) Fix [vllm] [2.12 regression][Quantization] CPU offload diverges from non-offload for w4a16 (Qwen1.5-MoE-A2.7B) [1 comments, 2 participants]

Summary

Under torch 2.12.0 + triton 3.7.0, vLLM's test_cpu_offload_compressed_tensors fails because results from running with --cpu-offload-gb 1 differ from running without CPU offload (same args otherwise) on a w4a16-quantized Qwen1.5-MoE model:

AssertionError: Results for model='nm-testing/Qwen1.5-MoE-A2.7B-Chat-quantized.w4a16' are not the same.

The test is a parity check between two settings of the same model (one offloading, one not) — passing means CPU offload is functionally transparent. Failure means CPU offload produces different outputs vs. fully on-GPU on torch 2.12. Passes on torch 2.11. Blocking the torch 2.12 upgrade for vLLM (vllm-project/vllm#40077).

Environment

• torch: 2.12.0+cu130 (test channel)
• triton: 3.7.0
• CUDA: 13.0
• Python: 3.12.13
• Model: nm-testing/Qwen1.5-MoE-A2.7B-Chat-quantized.w4a16 (compressed-tensors w4a16)
• Quant method: gptq_marlin

Reproduction

Failing test:

tests/quantization/test_cpu_offload.py::test_cpu_offload_compressed_tensors

Test body:

@pytest.mark.skipif(
    not is_quant_method_supported("gptq_marlin"),
    reason="gptq_marlin is not supported on this GPU type.",
)
def test_cpu_offload_compressed_tensors(monkeypatch):
    monkeypatch.setenv("VLLM_TEST_FORCE_LOAD_FORMAT", "auto")
    compare_two_settings(
        "nm-testing/Qwen1.5-MoE-A2.7B-Chat-quantized.w4a16",
        ["--enforce_eager"],
        ["--enforce_eager", "--cpu-offload-gb", "1"],
        max_wait_seconds=480,
    )

compare_two_settings runs the model under both arg sets and compares output text — they must match. They don't, on torch 2.12.

Reproducibility

• torch-2.12 branch: failed https://buildkite.com/vllm/ci/builds/63095#019dcf15-80ae-4e8a-a68a-b2f75c554b00
• main (torch 2.11) — passes on every recent build:
  • 2026-04-25 daily: https://buildkite.com/vllm/ci/builds/62981
  • 2026-04-26 nightly: https://buildkite.com/vllm/ci/builds/62990
  • 2026-04-26 daily: https://buildkite.com/vllm/ci/builds/63026
  • 2026-04-27 nightly: https://buildkite.com/vllm/ci/builds/63061

Diagnosis request

CPU offload divergence suggests the host-side branch of the quantized matmul (gptq_marlin / compressed-tensors w4a16) on torch 2.12 doesn't bit-identically match the GPU-resident branch — could be a tensor copy/cast/dequant change. Could a maintainer check whether torch 2.12 changes the device-transfer or dequantize numeric behavior for w4a16 weights when half the layers are on CPU?

Links

• vLLM PR: vllm-project/vllm#40077
• Umbrella: pytorch/pytorch#180899

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel