Issue Station
Sign In
Tools hereToolbox

vllm - 💡(How to fix) Fix [CI Failure]: Language Models Test (Extended Generation)

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Root Cause

  • Flaky test
  • Can reproduce locally
  • Caused by external libraries (e.g. bug in transformers)

Code Example

Matched tokens:	[222, 40, 494, 447, 9009, 98, 828, 366]
[2026-05-11T07:29:20Z] hf:	'\n# + [markdown] id="5f7o229v00-L"\n# **1950** - <NAME>'	{59: -4.088154315948486, 58: -4.088154315948486, 56: -4.213154315948486, 55: -4.213154315948486, 54: -4.213154315948486}
[2026-05-11T07:29:20Z] vllm:	'\n# + [markdown] id="68284477"\n# **1950**\n#\n# - '	{59: Logprob(logprob=-4.071040153503418, rank=1, decoded_token='6'), 56: Logprob(logprob=-4.196040153503418, rank=2, decoded_token='3'), 54: Logprob(logprob=-4.196040153503418, rank=3, decoded_token='1'), 57: Logprob(logprob=-4.196040153503418, rank=4, decoded_token='4'), 55: Logprob(logprob=-4.196040153503418, rank=5, decoded_token='2')}
RAW_BUFFERClick to expand / collapse

Name of failing test

tests/models/language/generation/test_common.py::test_models[True-False-5-32-bigcode/starcoder2-3b] and tests/models/language/generation/test_common.py::test_models[False-False-5-32-bigcode/starcoder2-3b]

Basic information

  • Flaky test
  • Can reproduce locally
  • Caused by external libraries (e.g. bug in transformers)

Could not reproduce on 1 x B200. Do not have Hopper on hand atm.

🧪 Describe the failing test

Generation mismatch

Matched tokens:	[222, 40, 494, 447, 9009, 98, 828, 366]
[2026-05-11T07:29:20Z] hf:	'\n# + [markdown] id="5f7o229v00-L"\n# **1950** - <NAME>'	{59: -4.088154315948486, 58: -4.088154315948486, 56: -4.213154315948486, 55: -4.213154315948486, 54: -4.213154315948486}
[2026-05-11T07:29:20Z] vllm:	'\n# + [markdown] id="68284477"\n# **1950**\n#\n# - '	{59: Logprob(logprob=-4.071040153503418, rank=1, decoded_token='6'), 56: Logprob(logprob=-4.196040153503418, rank=2, decoded_token='3'), 54: Logprob(logprob=-4.196040153503418, rank=3, decoded_token='1'), 57: Logprob(logprob=-4.196040153503418, rank=4, decoded_token='4'), 55: Logprob(logprob=-4.196040153503418, rank=5, decoded_token='2')}

📝 History of failing test

Started between commit 08bfedc15 and 2488d1dc (dated ~ April 7)

<img width="753" height="228" alt="Image" src="https://github.com/user-attachments/assets/d4931e45-9301-4888-b18f-54b96e3a344f" />

CC List.

No response

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#retriever error#indexing error#inference speed#output truncation#response parsing

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING