vllm - ✅(Solved) Fix [Bug]: Deepseek R1 produces incorrect output [2 pull requests, 4 comments, 2 participants]

vllm2026-04-03 18:31:26

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

vllm-project/vllm#38931•Fetched 2026-04-08 02:44:51

View on GitHub

Comments

Participants

Timeline

Reactions

Author

wzhao18

Participants

wzhao18

ZJY0516

Timeline (top)

commented ×4cross-referenced ×2added_to_project_v2 ×1labeled ×1

PR fix notes

PR #38989: [Bug] Fix routing bias dtype for trtllm per-block fp8 moe

Repository: vllm-project/vllm
Author: wzhao18
State: closed | merged: True
Link: https://github.com/vllm-project/vllm/pull/38989

Description (problem / solution / changelog)

Purpose

Fix [Bug]: Deepseek R1 produces incorrect output FIX https://github.com/vllm-project/vllm/issues/39179 Flashinfer v0.6.7 requires bf16 routing bias dtype for trtllm MoE. We have done this in https://github.com/vllm-project/vllm/pull/38423 for nvfp4 and fp8 per-tensor, but haven't done for fp8 per-block.

Test Plan

Test Result

main:

vllm serve deepseek-ai/DeepSeek-R1 --trust-remote-code --tensor-parallel-size 8

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.0197|±  |0.0038|
|     |       |strict-match    |     5|exact_match|↑  |0.0000|±  |0.0000|

branch:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9598|±  |0.0054|
|     |       |strict-match    |     5|exact_match|↑  |0.9591|±  |0.0055|

<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

</details>

Changed files

vllm/model_executor/layers/fused_moe/experts/trtllm_fp8_moe.py (modified, +5/-0)

Code Example

Your output of `python collect_env.py` here

---

vllm serve deepseek-ai/DeepSeek-R1 --trust-remote-code --tensor-parallel-size 8

---

import requests
import json

# 1. Update the URL to the chat endpoint
url = "http://localhost:8000/v1/chat/completions"

# 2. Structure the payload with roles
payload = {
    # "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2025?"}
    ],
    "max_tokens": 1000,
    "temperature": 0,
    "stream": True
}

response = requests.post(url, json=payload, stream=True)

---

You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>

RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

Your output of `python collect_env.py` here

</details>

🐛 Describe the bug

vllm serve deepseek-ai/DeepSeek-R1 --trust-remote-code --tensor-parallel-size 8

import requests
import json

# 1. Update the URL to the chat endpoint
url = "http://localhost:8000/v1/chat/completions"

# 2. Structure the payload with roles
payload = {
    # "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2025?"}
    ],
    "max_tokens": 1000,
    "temperature": 0,
    "stream": True
}

response = requests.post(url, json=payload, stream=True)

Output

You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>
You are a helpful assistant. <think>

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The issue is likely caused by the model not being specified in the payload, causing the assistant to repeat its initial message indefinitely.

Guidance

Verify that the model is correctly specified in the payload by uncommenting the "model" line and ensuring it points to a valid model.
Check the documentation for the correct model names and versions that can be used with the vllm service.
Test the request with a different model to see if the issue persists.
Consider adding error handling to the request to catch and handle any potential errors that may be occurring.

Example

payload = {
    "model": "Qwen/Qwen2.5-1.5B-Instruct",  # Uncomment and verify the model name
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2025?"}
    ],
    "max_tokens": 1000,
    "temperature": 0,
    "stream": True
}

Notes

The issue may be specific to the vllm service or the model being used, and further investigation may be required to determine the root cause.

Recommendation

Apply workaround: Specify a valid model in the payload to see if it resolves the issue.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#logging issue #authentication issue #prompt issue #agent setup #task chaining

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - ✅(Solved) Fix [Bug]: Deepseek R1 produces incorrect output [2 pull requests, 4 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

PR fix notes

PR #38989: [Bug] Fix routing bias dtype for trtllm per-block fp8 moe

Description (problem / solution / changelog)

Purpose

Test Plan

Test Result

Changed files

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

TRENDING

vllm - ✅(Solved) Fix [Bug]: Deepseek R1 produces incorrect output [2 pull requests, 4 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

PR fix notes

PR #38989: [Bug] Fix routing bias dtype for trtllm per-block fp8 moe

Description (problem / solution / changelog)

Purpose

Test Plan

Test Result

Changed files

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

RELATED_DISCOVERY

TRENDING