Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix [vllm] [2.12 regression] test_elastic_ep_scaling_uneven: GSM8K accuracy collapses to 0.000 (threshold 0.58) [2 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#181633•Fetched 2026-04-28 06:24:15
View on GitHub
Comments
2
Participants
2
Timeline
36
Reactions
0
Author
atalmanatalman
Participants
atalmanatalmanLucaskabelaLucaskabela
Timeline (top)
mentioned ×15subscribed ×15commented ×2labeled ×2

Under torch 2.12.0 + triton 3.7.0, vLLM's test_elastic_ep_scaling_uneven fails because GSM8K accuracy on the initial 2-GPU configuration drops to 0.000:

AssertionError: [Initial (2 GPUs)] GSM8K accuracy 0.000 is below expected threshold 0.58

The model isn't generating any correct answers — accuracy didn't merely degrade, it collapsed to zero. Passes on torch 2.11. Blocking the torch 2.12 upgrade for vLLM (vllm-project/vllm#40077).

Error Message

A separate RuntimeError: Failed to unpickle serialized exception also appears in the same job, indicating a worker-side crash propagated incompletely. Accuracy collapsing to 0.000 (not just degrading) suggests the model is producing garbage tokens — could indicate weight loading/quant kernel failure, attention scaling regression, or rotary/softmax breaking a generation path entirely. The accompanying "Failed to unpickle serialized exception" hints at a worker-side exception that wasn't surfaced cleanly. Could a maintainer trace whether a worker crashed during EP setup on torch 2.12?

Root Cause

Under torch 2.12.0 + triton 3.7.0, vLLM's test_elastic_ep_scaling_uneven fails because GSM8K accuracy on the initial 2-GPU configuration drops to 0.000:

Code Example

tests/distributed/test_elastic_ep.py::test_elastic_ep_scaling_uneven

---

assert accuracy >= EXPECTED_ACCURACY, (
    f"[{stage}] GSM8K accuracy {accuracy:.3f} is below "
    f"expected threshold {EXPECTED_ACCURACY}"
)
# Observed: 0.000 vs threshold 0.58
RAW_BUFFERClick to expand / collapse

Summary

Under torch 2.12.0 + triton 3.7.0, vLLM's test_elastic_ep_scaling_uneven fails because GSM8K accuracy on the initial 2-GPU configuration drops to 0.000:

AssertionError: [Initial (2 GPUs)] GSM8K accuracy 0.000 is below expected threshold 0.58

The model isn't generating any correct answers — accuracy didn't merely degrade, it collapsed to zero. Passes on torch 2.11. Blocking the torch 2.12 upgrade for vLLM (vllm-project/vllm#40077).

Environment

  • torch: 2.12.0+cu130 (test channel)
  • triton: 3.7.0
  • CUDA: 13.0
  • Python: 3.12.13
  • Setup: 2 GPUs initial (Elastic EP), GSM8K eval

Reproduction

Failing test:

tests/distributed/test_elastic_ep.py::test_elastic_ep_scaling_uneven

Assertion in _run_gsm8k_eval:

assert accuracy >= EXPECTED_ACCURACY, (
    f"[{stage}] GSM8K accuracy {accuracy:.3f} is below "
    f"expected threshold {EXPECTED_ACCURACY}"
)
# Observed: 0.000 vs threshold 0.58

A separate RuntimeError: Failed to unpickle serialized exception also appears in the same job, indicating a worker-side crash propagated incompletely.

Reproducibility

Diagnosis request

Accuracy collapsing to 0.000 (not just degrading) suggests the model is producing garbage tokens — could indicate weight loading/quant kernel failure, attention scaling regression, or rotary/softmax breaking a generation path entirely. The accompanying "Failed to unpickle serialized exception" hints at a worker-side exception that wasn't surfaced cleanly. Could a maintainer trace whether a worker crashed during EP setup on torch 2.12?

Links

  • vLLM PR: vllm-project/vllm#40077
  • Umbrella: pytorch/pytorch#180899

cc @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci @aditvenk @xmfan @weifengpy

extent analysis

TL;DR

Downgrade to torch 2.11 to potentially resolve the accuracy collapse issue in vLLM's test_elastic_ep_scaling_uneven test.

Guidance

  • Verify that the issue is specific to torch 2.12.0 by checking the build history and comparing the results with torch 2.11.
  • Investigate the "Failed to unpickle serialized exception" error to determine if it's related to the accuracy collapse issue.
  • Check the model's weight loading and quantization kernel to ensure they are working correctly with torch 2.12.0.
  • Consider tracing the worker-side exception to determine if a worker crashed during EP setup on torch 2.12.

Notes

The accuracy collapse issue may be specific to the combination of torch 2.12.0 and triton 3.7.0, and downgrading to torch 2.11 may resolve the issue.

Recommendation

Apply workaround: Downgrade to torch 2.11, as the issue is not present in this version and it may provide a temporary solution until the root cause is identified and fixed.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#autograd error#model save/load#optimization#mixed precision#training loop

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

pytorch - 💡(How to fix) Fix [vllm] [2.12 regression] test_elastic_ep_scaling_uneven: GSM8K accuracy collapses to 0.000 (threshold 0.58) [2 comments, 2 participants]