pytorch - 💡(How to fix) Fix [vllm] [2.12 regression] Qwen3-VL vision-connector LoRA generation diverges from golden output [1 comments, 2 participants]

Summary

Under torch 2.12.0 + triton 3.7.0, vLLM's test_qwen3vl_vision_lora (Qwen3-VL with the EpochEcho/qwen3-4b-vl-lora-vision-connector LoRA adapter) generates output that doesn't match the golden expected pattern:

AssertionError: Generated text "A majestic tower stands tall" doesn't match expected pattern "A majestic skyscraper stands tall, partially obscured by a vibrant canopy of cherry blossoms, against a clear blue sky."

Passes on torch 2.11. Newly failing on the 2026-04-27 run; Qwen3-VL is a different model from #181409 (which is Qwen2-VL). Blocking the torch 2.12 upgrade for vLLM (vllm-project/vllm#40077).

Environment

• torch: 2.12.0+cu130 (test channel)
• triton: 3.7.0
• CUDA: 13.0 / Driver: 570.133.20
• Python: 3.12.13
• Base model: Qwen3-VL (QWEN3VL_MODEL_PATH)
• LoRA adapter: EpochEcho/qwen3-4b-vl-lora-vision-connector

Reproduction

Failing test:

tests/lora/test_qwenvl.py::test_qwen3vl_vision_lora

config = TestConfig(
    model_path=QWEN3VL_MODEL_PATH,
    lora_path=qwen3vl_vision_lora_files,
    mm_processor_cache_gb=0,
    enable_tower_connector_lora=True,
)
tester = Qwen2VLTester(config)
for lora_id in [1, 2]:
    tester.run_test(TEST_IMAGES, expected_outputs=EXPECTED_OUTPUTS, lora_id=lora_id)

Assertion (in Qwen2VLTester.run_test): assert expected.startswith(generated).

Observed: "A majestic tower stands tall" Expected: "A majestic skyscraper stands tall, partially obscured by a vibrant canopy of cherry blossoms, against a clear blue sky."

Reproducibility

• torch-2.12 branch: failed https://buildkite.com/vllm/ci/builds/63095#019dcf15-801e-4341-8c4f-f4af1300c1cd
• main (torch 2.11) — passes on every recent build:
  • 2026-04-25 daily: https://buildkite.com/vllm/ci/builds/62981
  • 2026-04-26 nightly: https://buildkite.com/vllm/ci/builds/62990
  • 2026-04-26 daily: https://buildkite.com/vllm/ci/builds/63026
  • 2026-04-27 nightly: https://buildkite.com/vllm/ci/builds/63061

Diagnosis request

Same family as pytorch/pytorch#181409 (Qwen2-VL vision-tower LoRA divergence) and pytorch/pytorch#181168 (Qwen2-VL multi-image divergence). The Qwen3-VL LoRA case is a different model (Qwen3 vs Qwen2) and a different LoRA target (vision connector), but the symptom — small numerical drift on torch 2.12 producing a deterministic divergence in LoRA-adapted multimodal generation — is the same family. Could a maintainer investigate the shared kernel-level cause (matmul / attention / rotary) under LoRA?

Links

• vLLM PR: vllm-project/vllm#40077
• Umbrella: pytorch/pytorch#180899

cc @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @aakhundov @coconutruben @jataylo