vllm - 💡(How to fix) Fix [Bug]: Thinking token budget not enforced with MTP speculative decoding (works without MTP) [1 participants]

vllm serve 'Qwen/Qwen3.5-35B-A3B-FP8' \
  --host 0.0.0.0 \
  --port 5001 \
  --reasoning-parser qwen3 \
  --reasoning-config '{"reasoning_start_str": "<think>", "reasoning_end_str": " Now formulate the final answer.</think>"}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --max-model-len 131072 \

---

import httpx
from openai import OpenAI

httpx_client = httpx.Client(verify=False, timeout=None)

client = OpenAI(
    base_url=base_url,
    api_key=api_key,
    http_client=httpx_client
)

completion = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B-FP8",
    messages=[
        {"role": "system", "content": "Answer the user query"},
        {"role": "user", "content": "9.11 and 9.8, which is greater?"},
    ],
    extra_body={"thinking_token_budget": 50},
    temperature=0
)

print(completion.choices[0].message.reasoning)

---

Thinking Process:

1. Analyze the Request...
2. Identify the Numbers: Now formulate the final answer.

---

vllm serve 'Qwen/Qwen3.5-35B-A3B-FP8' \
  --host 0.0.0.0 \
  --port 5001 \
  --reasoning-parser qwen3 \
  --reasoning-config '{"reasoning_start_str": "<think>", "reasoning_end_str": " Now formulate the final answer.</think>"}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --max-model-len 131072 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 1 \
  --speculative-config.max_model_len 131072 \

---

import httpx
from openai import OpenAI

httpx_client = httpx.Client(verify=False, timeout=None)

client = OpenAI(
    base_url=base_url,
    api_key=api_key,
    http_client=httpx_client
)

completion = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B-FP8",
    messages=[
        {"role": "system", "content": "Answer the user query"},
        {"role": "user", "content": "9.11 and 9.8, which is greater?"},
    ],
    extra_body={"thinking_token_budget": 50},
    temperature=0
)

print(completion.choices[0].message.reasoning)

---

Thinking Process:

1.  **Analyze the Request:** The user is asking to compare two numbers, 9.11 and 9.8, and determine which one is greater.

2.  **Identify the Numbers:**
    *   Number A: 9.11
    *   Number B: 9.8

3.  **Compare the Numbers:**
    *   Look at the whole number part (left of the decimal point): Both are 9.
    *   Look at the tenths place (first digit right of the decimal point):
        *   9.11 has a 1.
        *   9.8 has an 8.
    *   Compare the digits in the tenths place: 8 is greater than 1.
    *   Therefore, 9.8 is greater than 9.11.

4.  **Common Pitfall:** Sometimes people treat decimals like whole numbers (thinking 11 is greater than 8), but in decimal notation, place value matters. 9.8 is equivalent to 9.80. 9.80 > 9.11.

5.  **Formulate the Answer:** State clearly which number is greater and briefly explain why (comparing place values).

6.  **Final Output Construction:** "9.8 is greater than 9.11." (Maybe add a brief explanation for clarity).

    *   Draft: 9.8 is greater.
    *   Refinement: 9.8 is greater than 9.11. When comparing decimals, look at the tenths place. 8 is larger than 1.

7.  **Final Decision:** Keep it direct and helpful. "9.8 is greater." or "9.8 is greater than 9.11."

    Let's write: "9.8 is greater than 9.11."
    Explanation: 9.8 is the same as 9.80, and 80 hundredths is more than 11 hundredths.

    Wait, keep it simple.
    "9.8 is greater."

    Let's provide the answer clearly.
    "9.8 is greater than 9.11."

    Why?
    9.8 = 9.80
    9.11 = 9.11
    80 > 11.

    Okay, ready to generate response