Issue Station
Sign In
Tools hereToolbox

vllm - 💡(How to fix) Fix [Bug]: Based on Qwen3.5-35B-A3B, why does enabling MTP speculative decoding actually reduce the prefix cache hit rate? [3 comments, 3 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#38182•Fetched 2026-04-08 01:31:48
View on GitHub
Comments
3
Participants
3
Timeline
4
Reactions
0
Author
uOnePieceuOnePiece
Participants
AngazennAngazennFED4FED4uOnePieceuOnePiece
Timeline (top)
commented ×3labeled ×1
RAW_BUFFERClick to expand / collapse

Your current environment

My ENV： +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA H20-3e On | 00000000:08:00.0 Off | 0 | | N/A 41C P0 121W / 500W | 131191MiB / 143771MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 99363 C VLLM::EngineCore 13118... | +-----------------------------------------------------------------------------------------+

🐛 Describe the bug

Based on Qwen3.5-35B-A3B, why does enabling MTP speculative decoding actually reduce the prefix cache hit rate?

Benchmarks details: Input: 5500 fixed prefix+200 random input Output: 10

Scenario 1: Command: vllm serve/mnt/workspace/qwen35_35B_A3B --enable-prefix-caching Prefix cache hit rate: around 71%

Scenario 2: Command: vllm serve /mnt/workspace/qwen35_35B_A3B --enable-prefix-caching --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' Prefix cache hit rate: around 92%

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

To address the issue of reduced prefix cache hit rate when enabling MTP speculative decoding, we need to adjust the speculative decoding configuration.

Step-by-Step Solution:

  1. Review Speculative Decoding Configuration: Check the current speculative decoding configuration to understand how it's set up.
  2. Adjust num_speculative_tokens: Try reducing the num_speculative_tokens value to see if it improves the prefix cache hit rate.
  3. Implement Custom Cache Logic: If adjusting num_speculative_tokens doesn't help, consider implementing custom cache logic to handle speculative decoding.

Example Code Snippet:

# Adjust speculative decoding configuration
speculative_config = {
    "method": "mtp",
    "num_speculative_tokens": 0.5  # Adjust this value based on your needs
}

# Implement custom cache logic
def custom_cache_logic(input_data, cache):
    # Custom logic to handle speculative decoding and caching
    if speculative_decoding_enabled:
        # Apply custom caching rules
        cache_hit_rate = calculate_cache_hit_rate(input_data, cache)
        return cache_hit_rate
    else:
        # Default caching behavior
        return default_cache_behavior(input_data, cache)

Verification

To verify the fix, run the benchmarks again with the adjusted speculative decoding configuration and custom cache logic. Compare the prefix cache hit rates to ensure they have improved.

Extra Tips

  • Monitor the system's GPU memory usage and adjust the speculative decoding configuration accordingly to avoid memory bottlenecks.
  • Consider implementing additional logging to track cache hit rates and speculative decoding performance.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#retriever error#indexing error#inference speed#output truncation#response parsing

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING