Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix [vllm] [2.12 regression] test_lm_eval_accuracy_v1_engine: gsm8k accuracy drops 0.54 -> 0.48 (deterministic, Qwen2-1.5B) [1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#181169•Fetched 2026-04-23 07:22:13
View on GitHub
Comments
0
Participants
1
Timeline
135
Reactions
0
Author
atalmanatalman
Participants
atalmanatalman
Timeline (top)
mentioned ×63subscribed ×63labeled ×8cross-referenced ×1

Under torch 2.12.0 + triton 3.7.0, vLLM's test_lm_eval_accuracy_v1_engine consistently fails because gsm8k accuracy drops from the expected 0.54 (+/-0.03) to 0.4806671721000758:

AssertionError: Expected: 0.54 | Measured: 0.4806671721000758

Passes on torch 2.11. The measured value is bit-identical across three runs over three days, which points at a deterministic numerical regression, not stochastic noise. Blocking the torch 2.12 upgrade for vLLM (vllm-project/vllm#40077).

Root Cause

Under torch 2.12.0 + triton 3.7.0, vLLM's test_lm_eval_accuracy_v1_engine consistently fails because gsm8k accuracy drops from the expected 0.54 (+/-0.03) to 0.4806671721000758:

Code Example

tests/entrypoints/openai/correctness/test_lmeval.py::test_lm_eval_accuracy_v1_engine

---

MODEL_NAME = "Qwen/Qwen2-1.5B-Instruct"
TASK = "gsm8k"
FILTER = "exact_match,strict-match"
RTOL = 0.03
EXPECTED_VALUE = 0.54
RAW_BUFFERClick to expand / collapse

Summary

Under torch 2.12.0 + triton 3.7.0, vLLM's test_lm_eval_accuracy_v1_engine consistently fails because gsm8k accuracy drops from the expected 0.54 (+/-0.03) to 0.4806671721000758:

AssertionError: Expected: 0.54 | Measured: 0.4806671721000758

Passes on torch 2.11. The measured value is bit-identical across three runs over three days, which points at a deterministic numerical regression, not stochastic noise. Blocking the torch 2.12 upgrade for vLLM (vllm-project/vllm#40077).

Environment

  • torch: 2.12.0+cu130 (test channel)
  • triton: 3.7.0
  • CUDA: 13.0 / Driver: 570.133.20
  • Python: 3.12.13
  • GPU: NVIDIA L4 (23 GiB)
  • Model: Qwen/Qwen2-1.5B-Instruct
  • Task: gsm8k, filter exact_match,strict-match, num_concurrent=500, max_model_len=4096

Reproduction

Failing test:

tests/entrypoints/openai/correctness/test_lmeval.py::test_lm_eval_accuracy_v1_engine

Test constants (from tests/entrypoints/openai/correctness/test_lmeval.py):

MODEL_NAME = "Qwen/Qwen2-1.5B-Instruct"
TASK = "gsm8k"
FILTER = "exact_match,strict-match"
RTOL = 0.03
EXPECTED_VALUE = 0.54

Reproducibility on torch 2.12 branch (3/3 days, bit-identical)

Same Measured: 0.4806671721000758 across three runs:

Passes on same-day main builds (torch 2.11):

Note: 62254 (2026-04-21 main nightly) also showed this test failing, but with a different root cause (RuntimeError: Server exited unexpectedly.) — an unrelated server-init crash, not the accuracy drop reported here.

Diagnosis request

A ~6 percentage-point drop on gsm8k is well outside normal run-to-run noise, and the measured value is bit-identical across three torch-2.12 runs — strongly points to a deterministic numerical change introduced by torch 2.12 or triton 3.7 (possibly attention, matmul, or rotary). Could a maintainer investigate whether this is intentional vs. regression?

Links

  • vLLM PR: vllm-project/vllm#40077
  • Umbrella: pytorch/pytorch#180899

cc @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @aakhundov @coconutruben @jataylo @drisspg @liangel-02 @howardzhang-cv

extent analysis

TL;DR

Downgrade to torch 2.11 to potentially resolve the deterministic numerical regression causing the gsm8k accuracy drop.

Guidance

  • Investigate the changes introduced in torch 2.12 and triton 3.7, focusing on attention, matmul, or rotary operations, to determine if they are the cause of the accuracy drop.
  • Verify that the issue is not related to the specific GPU or CUDA version by testing on different hardware configurations.
  • Run the test multiple times on torch 2.11 to confirm that the issue is not present in that version.
  • Consider opening a bug report with the PyTorch or Triton teams to investigate the potential regression.

Example

No code snippet is provided as the issue seems to be related to a version change rather than a code-specific problem.

Notes

The issue seems to be specific to the combination of torch 2.12 and triton 3.7, and downgrading to torch 2.11 may resolve the issue. However, it is essential to investigate the root cause to ensure that the fix is not just a workaround.

Recommendation

Apply the workaround of downgrading to torch 2.11, as it has been confirmed to pass the test, while investigating the root cause of the issue with the torch 2.12 and triton 3.7 combination.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#mixed precision#training loop#device allocation#model download#tokenizer error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

pytorch - 💡(How to fix) Fix [vllm] [2.12 regression] test_lm_eval_accuracy_v1_engine: gsm8k accuracy drops 0.54 -> 0.48 (deterministic, Qwen2-1.5B) [1 participants]