vllm - 💡(How to fix) Fix [Bug]: The premature implementation of structured generation constraints in qwen3 led to a disastrous decline in model capabilities. [4 comments, 3 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#39677Fetched 2026-04-14 05:38:15
View on GitHub
Comments
4
Participants
3
Timeline
5
Reactions
0
Timeline (top)
commented ×4labeled ×1

Error Message

Error reply

Root Cause

output

{
  "model": "qwen3-8B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "N",
        "reasoning_content": "\nOkay, the user is asking if the sky is blue. I need to answer with just Y or N. Let me think. The sky appears blue because of Rayleigh scattering. When sunlight enters the Earth's atmosphere, the shorter wavelengths of light (like blue and violet) are scattered more than the longer wavelengths (like red and yellow). Even though violet has a shorter wavelength than blue, the sky appears blue because our eyes are more sensitive to blue light and some of the violet light is absorbed by the upper atmosphere. So, the answer should be yes. But wait, sometimes the sky can look different, like during sunrise or sunset when it's more red or orange. But the general answer is that the sky is blue. So the answer is Y.\n"
      }
    }
  ]
}

Code Example

{
    "model": "qwen3-8B",
    "messages": [{"role": "user", "content": "Please answer with only Y or N: Is the sky blue?"}],
    "temperature": 0.0,
    "max_tokens": 500,
    "chat_template_kwargs": {"enable_thinking": true},
    "guided_regex": "[YN]"
}

---

{
  "model": "qwen3-8B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "N",
        "reasoning_content": "\nOkay, the user is asking if the sky is blue. I need to answer with just Y or N. Let me think. The sky appears blue because of Rayleigh scattering. When sunlight enters the Earth's atmosphere, the shorter wavelengths of light (like blue and violet) are scattered more than the longer wavelengths (like red and yellow). Even though violet has a shorter wavelength than blue, the sky appears blue because our eyes are more sensitive to blue light and some of the violet light is absorbed by the upper atmosphere. So, the answer should be yes. But wait, sometimes the sky can look different, like during sunrise or sunset when it's more red or orange. But the general answer is that the sky is blue. So the answer is Y.\n"
      }
    }
  ]
}

---

{
    "model": "qwen3-8B",
    "messages": [{"role": "user", "content": "Please answer with only Y or N: Is the sky blue?"}],
    "temperature": 0.0,
    "max_tokens": 500,
    "chat_template_kwargs": {"enable_thinking": true}
}

---

{
  "model": "qwen3-8B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "\n\nY",
        "reasoning_content": "\nOkay, the user is asking if the sky is blue. I need to answer with just Y or N. Let me think. The sky appears blue because of Rayleigh scattering. When sunlight enters the Earth's atmosphere, the shorter wavelengths of light (like blue and violet) are scattered more than the longer wavelengths (like red and yellow). Even though violet has a shorter wavelength than blue, the sky appears blue because our eyes are more sensitive to blue light and some of the violet light is absorbed by the upper atmosphere. So, the answer should be yes. But wait, sometimes the sky can look different, like during sunrise or sunset when it's more red or orange. But the general answer is that the sky is blue. So the answer is Y.\n"
      }
    }
  ]
}
RAW_BUFFERClick to expand / collapse

Your current environment

Query: Please answer with only Y or N: Is the sky blue?

Error reply

request body

{
    "model": "qwen3-8B",
    "messages": [{"role": "user", "content": "Please answer with only Y or N: Is the sky blue?"}],
    "temperature": 0.0,
    "max_tokens": 500,
    "chat_template_kwargs": {"enable_thinking": true},
    "guided_regex": "[YN]"
}

output

{
  "model": "qwen3-8B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "N",
        "reasoning_content": "\nOkay, the user is asking if the sky is blue. I need to answer with just Y or N. Let me think. The sky appears blue because of Rayleigh scattering. When sunlight enters the Earth's atmosphere, the shorter wavelengths of light (like blue and violet) are scattered more than the longer wavelengths (like red and yellow). Even though violet has a shorter wavelength than blue, the sky appears blue because our eyes are more sensitive to blue light and some of the violet light is absorbed by the upper atmosphere. So, the answer should be yes. But wait, sometimes the sky can look different, like during sunrise or sunset when it's more red or orange. But the general answer is that the sky is blue. So the answer is Y.\n"
      }
    }
  ]
}

Unconstrained reply

request body

{
    "model": "qwen3-8B",
    "messages": [{"role": "user", "content": "Please answer with only Y or N: Is the sky blue?"}],
    "temperature": 0.0,
    "max_tokens": 500,
    "chat_template_kwargs": {"enable_thinking": true}
}

output

{
  "model": "qwen3-8B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "\n\nY",
        "reasoning_content": "\nOkay, the user is asking if the sky is blue. I need to answer with just Y or N. Let me think. The sky appears blue because of Rayleigh scattering. When sunlight enters the Earth's atmosphere, the shorter wavelengths of light (like blue and violet) are scattered more than the longer wavelengths (like red and yellow). Even though violet has a shorter wavelength than blue, the sky appears blue because our eyes are more sensitive to blue light and some of the violet light is absorbed by the upper atmosphere. So, the answer should be yes. But wait, sometimes the sky can look different, like during sunrise or sunset when it's more red or orange. But the general answer is that the sky is blue. So the answer is Y.\n"
      }
    }
  ]
}

Although it can be generated normally under unconstrained conditions, it has two extra \n characters.

🐛 Describe the bug

docker

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The issue can be fixed by adjusting the response processing to remove extra newline characters when the model is constrained to provide a specific format of answer.

Guidance

  • The problem seems to stem from the model's response formatting, particularly when it's constrained by a regular expression (guided_regex) to provide a specific answer format.
  • The extra newline characters (\n) in the constrained response might be due to the model's internal processing or the way the response is generated under constrained conditions.
  • To mitigate this, consider preprocessing the model's response to remove any unnecessary newline characters before presenting it as the final answer.
  • Verify that the model's response matches the expected format by checking the output against the specified regular expression ("[YN]").

Example

No specific code example can be provided without knowing the exact implementation details, but a general approach might involve using string manipulation functions to remove leading or trailing newline characters from the model's response.

Notes

The issue seems specific to the interaction between the model's response generation and the constraints applied to it. The solution might require adjustments to how the model's output is processed or how the constraints are applied.

Recommendation

Apply a workaround by preprocessing the model's response to ensure it matches the expected format, specifically removing any extra newline characters that do not comply with the [YN] regular expression constraint.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING