Your current environment

<details> See below </details>

🐛 Describe the bug

Problem

Speculative decoding acceptance length (AL) smoothly and monotonically regresses over time when serving models with Eagle/Eagle3 draft models in vLLM 0.17.1. AL never recovers — only a full service restart resets it to the initial high value. The degradation correlates with traffic volume per prompt type — only high-volume prompt types are affected.

Setup

• vLLM 0.17.1, v1 engine, async scheduling enabled
• 4x H200 GPUs (564GB total), gpu_memory_utilization=0.9 (all models executed with TP=4)
• Model ~250GB, leaving ~257GB for KV cache (~5M tokens capacity, ~312K blocks at block_size=16)
• KV cache dtype: GPT OSS 120B - BF16 (auto), ~50KB per token, other models uses FP8
• Block size: 16
• Prefix caching enabled
• Traffic: ~10-20 RPS
• ~60 different prompt types, average ~30K tokens each
• Per type: ~27K tokens cacheable (unique to that type), ~3K tokens unique per request
• First 4K tokens shared across all types, remaining 23K unique per type
• Traffic distribution: 5 types occupy ~50% of traffic, other 55 types share the rest
• Average OSL: 40 tokens

Observations

1. GPT OSS 120B BF16 (BF16 weights and KV, Eagle2, 4 draft tokens, batch_size=2): AL starts at ~3.07, smoothly regresses to ~2.8 over hours. Only 2-3 highest-volume types affected.
2. GPT OSS 120B (BF16 weights and KV, Eagle3, 4 draft tokens, batch_size=2): AL starts at ~3.07, smoothly regresses to ~2.8 over hours. Only 2-3 highest-volume types affected.
3. Qwen3 235B Instruct (FP8 weights&activation&KV cache, Eagle3, 5 draft tokens, batch_size=4): AL starts at ~4.5, smoothly regresses to ~2.8. All high-volume types affected, low-volume types unimpacted.
4. Qwen3 235B Instruct (FP8 weights&activation&KV cache, Eagle3, 5 draft tokens, batch_size=2): AL starts at ~4.2, regresses to ~3.8. Reducing speculation length to lower values does NOT change the regression behavior.
5. Nova (FP8 weights&activation&KV cache, Eagle3, 5 draft tokens, batch_size=2) with vLLM: AL starts at ~3.9, regresses to ~3.5 vs with TRT-LLM: AL is stable and does not regress.
6. Restart resets AL immediately to the initial high value, regardless of current traffic level.
7. TRT-LLM: slight AL regression at peak traffic but recovers during low traffic. vLLM only regresses, never recovers.
8. Both Eagle2 and Eagle3 shows regression of AL over time.
9. TTFT and prefix cache hit rate appear stable.
10. Running Qwen3 235B Instruct with newer vLLM version 0.19.1 does not fix the problem, regression is still observed.

Relevant GitHub Issues/PRs

• Issue #14649: "EAGLE / MTP Doesn't Overwrite Approximated Hidden States / KV Cache" — describes the fundamental problem. Unfixed in v0, but v1's first pass mechanism largely addresses it (small residual gap).
• PR #14990: "Enhance EAGLE Architecture with Proper RMS Norms" — merged Mar 2025, adds proper norm handling to Eagle model.
• PR #16370: "[V1][Spec Decode] KV cache slots for eagle heads" — merged, shows v1 gap is small in offline benchmarks (2.43 vs 2.48 for 5 spec tokens on Llama 3.1 8B).
• PR #14464: "EAGLE output norm bug" — merged, fixes return x + residual, None vs return x, residual in Eagle model.
• PR #29845: "Simplified alternative padded-speculation acceptance rate fix" — merged, adds num_rejected_tokens_gpu adjustment to seq_lens before sequential drafting loop.

Experiments we are running now (will be updated with new observations)

• Disabling prefix caching
• Disabling async scheduling
• Run with batch size=1

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.