PR #40738: [Bugfix] Fix GDN conv + SSM state corruption with ngram spec decode

• Repository: vllm-project/vllm
• Author: tdoublep
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/40738

Description (problem / solution / changelog)

Summary

Fix output corruption when using ngram speculative decoding with hybrid GDN models (e.g., Qwen3.5) in mamba_cache_mode="none".

After a spec decode step accepts N tokens, the next non-spec decode step must read SSM state from block N-1 and conv state from an offset position. Two bugs prevented this:

1. num_accepted_tokens was not passed to SSM metadata builders on non-spec steps
2. causal_conv1d_fn had no mechanism to offset conv state reads based on accepted tokens

Changes

• gdn_attn.py: Compute spec_decode_src_indices for SSM state correction; pad num_accepted_tokens with 1s for prefill sequences in mixed batches
• gdn_linear_attn.py: Pre-copy SSM state from accepted block to block 0; pass num_accepted_tokens to conv kernels gated on whether correction is needed
• causal_conv1d.py: Add IS_SPEC_DECODING path to _causal_conv1d_fwd_kernel that offsets conv state reads/writes by num_accepted_tokens - 1
• gpu_model_runner.py: Pass num_accepted_tokens to GDN/Mamba2 builders on non-spec steps

Test plan

• Single-prompt: baseline vs ngram match token-for-token (Qwen3.5-0.8B)
• Mixed-batch: short prompt matches baseline; long prompt generates coherent output
• Kernel fix verified necessary: disabling conv offset causes regression
• Existing GDN + spec decode CI tests

Single-prompt:

from vllm import LLM, SamplingParams
MODEL, PROMPT = "Qwen/Qwen3.5-0.8B", "<code>\nclass Calculator:\n    def add(self, a, b):\n        return a + b\n</code>\n<update>\nAdd subtract and multiply methods\n</update>"
ARGS = dict(model=MODEL, trust_remote_code=True, enforce_eager=True, enable_chunked_prefill=True, max_model_len=4096)
SPEC = {"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 10, "prompt_lookup_min": 2}
S = SamplingParams(max_tokens=200, temperature=0)
b = list(LLM(**ARGS).generate([PROMPT], S)[0].outputs[0].token_ids)
n = list(LLM(**ARGS, speculative_config=SPEC).generate([PROMPT], S)[0].outputs[0].token_ids)
print("PASS" if b == n else "FAIL")

Mixed-batch (short + long prompt, max_num_batched_tokens=64): see reproduce_gdn_ngram_mixed.py in the branch.

Fixes #39273

AI-assisted: Yes (Claude). Not duplicating any existing PR.

Changed files

• vllm/model_executor/layers/mamba/gdn_linear_attn.py (modified, +27/-0)
• vllm/model_executor/layers/mamba/ops/causal_conv1d.py (modified, +20/-2)
• vllm/v1/attention/backends/gdn_attn.py (modified, +32/-2)
• vllm/v1/worker/gpu_model_runner.py (modified, +11/-0)

PR #36138: [Bugfix] Grammar was ignored when reasoning ended within speculated tokens

• Repository: vllm-project/vllm
• Author: sfbemerk
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/36138

Description (problem / solution / changelog)

Purpose

This PR attempts to fix a bug (#31858, #34650) when Speculative Decoding (such as MTP), Reasoning, and Structured Output / Grammar are used in combination: typically, grammar is not enabled during reasoning but only for the final answer. However, when the reasoning end token is generated, any subsequent draft tokens are not validated against the grammar, leading to an invalid final answer.

Test Plan

In general, the bug seems to be independent of the specific SpecDecode method; originally I had observed it with DeepSeek models and MTP, but for testing I recommend a smaller model like Qwen3-8B and using the same model as draft model. This way, we have high acceptance rates for our tests and a high likelihood that the original bug appears.

vllm serve "Qwen/Qwen3-8B" \
  --max-model-len 40960 \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"draft_model","model":"Qwen/Qwen3-8B","num_speculative_tokens":5}'

The test request should have response_format=json_schema and a prompt that lurkes the model into generating not pure json, e.g.

The original bug can also be reproduced for Model Runner V2, this bugfix works there as well. For testing, you should choose a different speculative method (since draft_model is not supported yet):

VLLM_USE_V2_MODEL_RUNNER=1 vllm serve "Qwen/Qwen3-8B" \
  --max-model-len 40960 \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"eagle3","model":"RedHatAI/Qwen3-8B-speculator.eagle3","num_speculative_tokens":5}'

The original bug is still present in vllm v0.17.1.

Test Result

without bugfix, the content field contains invalid json, e.g. because of markdown fences

"content": "```json\n{\n\n\"name\": \"Eldrin the Flameheart\",\n\"age\": 32,\n\"role\": \"warrior\"\n}```"

with the bugfix, the content field contains valid json that satisfies the requested grammar

"content": "{\n\n\"name\": \"Eldrin the Flameheart\",\n\"age\": 32,\n\"role\": \"warrior\"\n}"

I am happy to receive feedback and suggestions on how to improve the PR: the interplay of spec decode, grammar, reasoning, and async scheduling seems to be quite complex.

Related

There had been several attempts to fix this bug before: my first attempt in #34241 would reject all speculated tokens in the step where reasoning_end was detected, which was working fine, but was suboptimal. #34978 started with a better approach that would validate all speculative tokens following reasoning_end, but contained some bugs in the end and was discontinued.

Changed files

• tests/v1/structured_output/test_reasoning_structured_output.py (modified, +243/-39)
• vllm/v1/core/sched/scheduler.py (modified, +39/-22)
• vllm/v1/structured_output/__init__.py (modified, +121/-34)

PR #40783: [Bugfix] Fix Qwen3 reasoning parser: raw text tags, transition loss, end detection, token counting, withhold recovery

• Repository: vllm-project/vllm
• Author: ExtReMLapin
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/40783

Description (problem / solution / changelog)

To be used with https://github.com/vllm-project/vllm/pull/40861

Purpose

This PR fixes several critical edge cases in the Qwen3 reasoning parser where reasoning text was lost or incorrectly classified.

A key issue addressed is that Qwen3, while following system prompt instructions during reasoning, often outputs tool call tags as raw text fragments (e.g., < + tool + _call + >) instead of a single special token. Since these fragments arrive as multiple regular text tokens across different streaming deltas, the previous ID-based detection would fail or corrupt the output.

Key Changes

• Raw Text & Fragmented Tag Support: The parser now tracks the literal string <tool_call> across deltas. It correctly identifies the reasoning-to-content transition even when the tag is "built" over several steps from regular text tokens, preventing the parser from missing the start of a tool call.
• Transition Data Integrity: Fixed a bug in vllm/parser/abstract_parser.py where the final fragment of reasoning was silently overwritten by tool call content when both occurred in the same delta.
• Speculative Decoding Fix: Added is_reasoning_end_streaming, a delta-only variant used by parse_delta instead of is_reasoning_end. The paired-token guard in is_reasoning_end is intentionally preserved (it prevents system-prompt tool-call examples from triggering an early reasoning end); only its comment was clarified. The new streaming check inspects delta_ids without the guard, so a complete <tool_call>…</tool_call> delivered in a single delta (MTP / speculative decoding) correctly signals the end of reasoning.
• Preserve Multiple Tool Calls: Fixed extract_content_ids to search for the first occurrence of a tool call instead of the last, ensuring no tool calls are dropped when reasoning ends implicitly.
• Reasoning/Content Boundary: Ensured that any content emitted after an explicit </think> tag is never swallowed back into the reasoning field, even if a tool call follows shortly after.

Additional fixes (commit d3be2250f)

• count_reasoning_tokens override for the Qwen3.5+ template: the inherited depth-counter starts at depth=0 and is never incremented (because <think> lives in the prompt, not the model output), so it returned 0 reasoning tokens for every Qwen3.5+ generation. This silently broke usage.completion_tokens_details.reasoning_tokens reporting in API responses. The override treats the start of the output as the reasoning region and stops at the first </think> (or, for the Qwen3.5 implicit-end case, the first <tool_call>).

• Recover withheld <tool prefix when the continuation is a false positive: the partial-overlap guard withholds the trailing bytes of current_text when they look like the start of <tool_call>. Previously, if the model emitted an unrelated continuation (e.g. <tool_use>, or just <tool_belt), those bytes were silently dropped. The parser now re-emits them as reasoning.

Test Result

All new tests pass, and existing reasoning tests remain green.

Changed files

• tests/reasoning/test_qwen3_reasoning_parser.py (modified, +557/-0)
• vllm/parser/abstract_parser.py (modified, +25/-2)
• vllm/reasoning/qwen3_reasoning_parser.py (modified, +120/-32)

PR #39055: Fix Qwen3 reasoning tool calls embedded inside think

• Repository: vllm-project/vllm
• Author: ZenoAFfectionate
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/39055

Description (problem / solution / changelog)

Summary

This PR fixes a Qwen3/Qwen3.5 non-streaming compatibility issue when using:

• --reasoning-parser qwen3
• --tool-call-parser qwen3_coder

Qwen models can emit XML tool calls inside <think> ... </think>. The current non-streaming pipeline extracts reasoning first and only parses tool calls from content, so valid XML tool calls embedded in reasoning are lost.

This patch updates qwen3_reasoning_parser to promote valid XML tool-call blocks out of reasoning into content, allowing the existing qwen3_coder tool parser to recover them without changing the generic serving stack.

Why this scope

This PR fixes parser recovery, not model generation behavior. It does not try to prevent Qwen3.5 from emitting tool calls inside <think>; it makes vLLM robust when that output pattern appears.

Tests

Added tests cover:

• unchanged behavior for normal reasoning extraction
• embedded tool call promotion from reasoning to content
• successful parsing by qwen3_coder
• truncated reasoning recovery without </think>
• preservation of post-</think> content

Limitation

This change fixes the non-streaming path. Streaming recovery would require additional serving-layer changes and is intentionally left out of this minimal patch.

Changed files

• docs/design/qwen3_reasoning_tool_call_recovery.md (added, +88/-0)
• tests/reasoning/test_qwen3_reasoning_parser.py (modified, +114/-0)
• vllm/reasoning/qwen3_reasoning_parser.py (modified, +49/-2)

PR #40768: [Bugfix] Fix CUDA crash caused by stale async placeholder tokens in speculative decoding

• Repository: vllm-project/vllm
• Author: z1ying
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/40768

Description (problem / solution / changelog)

Fixes #37159

Summary

This PR fixes a CUDA device-side assert (vectorized_gather_kernel: ind >= 0) triggered by speculative decoding with async scheduling.

The root cause is that -1 placeholder tokens could leak into input_ids, leading to invalid GPU embedding lookups.

Reproduced on Tesla V100-PCIE-16GB with zai-org/GLM-OCR + MTP speculative decoding under concurrent multimodal requests (both vLLM v0.17.1 and vLLM v0.19.1).

Background

Async speculative decoding uses -1 as a placeholder token to indicate:

"This token will be filled on GPU later."

However, under certain scheduling conditions, these placeholders were not properly overwritten, and eventually reached the embedding kernel.

Failure scenario

This occurs when:

• A request carries async placeholder tokens (-1)
• It was not present in the previous worker batch (e.g., newly scheduled, or re-added after preemption)

In this case, worker-side overwrite logic is skipped:

# gpu_model_runner.py: _prepare_input_ids()
for cur_index in range(num_reqs):
    prev_index = prev_positions[cur_index]
    if prev_index < 0:
        continue  # overwrite skipped, -1 survives

As a result, -1 propagates through the pipeline:

token_ids_cpu
  → input_ids.copy_to_gpu()
    → embedding lookup
      → vectorized_gather_kernel
        → CUDA assert (index -1 out of bounds)

Why V100 is particularly affected

V100 (16 GB) forces aggressive chunked prefill, which creates mixed batches (prefill + decode) more frequently than larger-VRAM GPUs. Mixed batches increase the probability that a decode request with async placeholder intent has no previous worker slot, triggering the stale -1 path.

Root Cause

The core issue is that spec_token_ids was overloaded with two different meanings:

1. Real speculative draft tokens — valid token IDs from the proposer
2. Async placeholder markers — -1 sentinels awaiting GPU-side overwrite

Because of this coupling:

• Scheduler sometimes emits placeholders as if they were real tokens
• Worker-side overwrite is not guaranteed for requests absent from the previous batch
• -1 can reach the GPU execution path

Fix

Decouple token semantics from placeholder state.

Key point: -1 must never reach GPU unless overwrite is guaranteed.

1. Separate real tokens from placeholder state

# request.py
self.spec_token_ids: list[int] = []           # ONLY real draft tokens
self.num_pending_async_spec_placeholders = 0  # placeholder count only

2. Stop injecting -1 into spec_token_ids

Before (old code):

request.spec_token_ids = self._spec_token_placeholders  # writes [-1, -1, ...] directly

After (new code):

request.num_pending_async_spec_placeholders = self.num_spec_tokens  # just a count

spec_token_ids is no longer touched here; it stays clean for real draft tokens only.

3. Scheduler-level safe materialization

New method _consume_spec_decode_tokens_for_step() replaces the inline logic in schedule() with three clear branches:

if request.spec_token_ids:
    # Real draft tokens from proposer — use directly, no gating needed.
    spec_token_ids = request.spec_token_ids
elif (
    self.scheduler_config.async_scheduling
    and pending_placeholders > 0
    and request.request_id in self.prev_step_scheduled_req_ids
):
    # Request was in previous worker batch → GPU overwrite is guaranteed.
    spec_token_ids = [-1] * pending_placeholders
else:
    # No previous slot → drop placeholders. -1 never reaches worker.
    return None

Guarantee:

• If GPU overwrite will NOT happen → never emit -1
• If GPU overwrite WILL happen → -1 is safe

4. Reset pending state on lifecycle transitions

Clear num_pending_async_spec_placeholders = 0 on:

• _preempt_request() — request evicted from running batch
• update_draft_token_ids() prefill chunk branch — draft tokens ignored during chunked prefill
• update_draft_token_ids() real draft branch — real draft tokens replace any pending placeholder intent

This prevents stale placeholder state from leaking across steps.

Why this is sufficient

Now align scheduler and worker invariants:

These conditions match exactly, so -1 is always safe by construction. The worker-side code requires no changes.

Files changed

Test Plan

Unit tests

Four regression tests covering the full state space of the fix:

pytest tests/v1/core/sched/test_async_scheduler.py -v -k "test_consume_async_spec"

End-to-end validation

Reproduced the original crash on Tesla V100-PCIE-16GB (vLLM v0.19.1) with zai-org/GLM-OCR + MTP speculative decoding under concurrent multimodal requests. After applying this fix, the same stress test (blast.py, 12 concurrent image OCR requests × 4 rounds) runs to completion with all requests returning 200.

Server launch:

CUDA_LAUNCH_BLOCKING=1 TORCH_USE_CUDA_DSA=1 \
vllm serve zai-org/GLM-OCR \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
  --served-model-name glm-ocr \
  --trust-remote-code \
  --gpu-memory-utilization 0.90 \
  --max-model-len 131072 \
  --limit-mm-per-prompt '{"image": 1}' \
  --tensor-parallel-size 1

Test (blast.py):

import asyncio, aiohttp, base64, random, os

IMAGE_DIR = "/workspace/test"
URL = "http://localhost:8000/v1/chat/completions"
CONCURRENCY = 12
ROUNDS = 4

images = [f for f in os.listdir(IMAGE_DIR)
          if f.endswith(('.jpg', '.png', '.jpeg'))]

def get_payload():
    img_path = os.path.join(IMAGE_DIR, random.choice(images))
    with open(img_path, "rb") as f:
        img_b64 = base64.b64encode(f.read()).decode()
    return {
        "model": "glm-ocr",
        "messages": [{"role": "user", "content": [
            {"type": "image_url",
             "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
            {"type": "text", "text": "请识别图片中的所有文字"}
        ]}],
        "max_tokens": 1024
    }

async def send(session, req_id):
    try:
        async with session.post(URL, json=get_payload(),
                timeout=aiohttp.ClientTimeout(total=3600)) as r:
            body = await r.json()
            if r.status != 200:
                print(f"[{req_id}] ERROR {r.status}: {body}")
            else:
                text = body["choices"][0]["message"]["content"]
                print(f"[{req_id}] OK: {text[:60]}...")
    except Exception as e:
        print(f"[{req_id}] EXCEPTION: {type(e).__name__}: {e}")

async def main():
    connector = aiohttp.TCPConnector(limit=CONCURRENCY)
    async with aiohttp.ClientSession(connector=connector) as session:
        for r in range(ROUNDS):
            print(f"\n=== Round {r+1}/{ROUNDS} ===")
            await asyncio.gather(
                *[send(session, f"{r}-{i}") for i in range(CONCURRENCY)])
            await asyncio.sleep(0.2)
    print("\n=== Done. Server survived all rounds. ===")

asyncio.run(main())

Test Results

Unit tests — all 13 tests in test_async_scheduler.py pass (including 4 new regression tests):

End-to-end — blast.py stress test on V100, all requests return 200 OK with no crash:

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Changed files

• tests/v1/core/test_async_scheduler.py (modified, +105/-1)
• tests/v1/core/utils.py (modified, +13/-3)
• vllm/v1/core/sched/async_scheduler.py (modified, +9/-8)
• vllm/v1/core/sched/scheduler.py (modified, +66/-11)
• vllm/v1/request.py (modified, +9/-1)