vllm - 💡(How to fix) Fix [Performance]: Significant TTFT Regression with Speculative Decoding (EAGLE3) [1 participants]

Proposal to improve performance

When enabling speculative decoding (tested with both EAGLE3 and DFlash methods), I observe a substantial increase in Time-To-First-Token (TTFT), particularly at the P99 percentile. While the expected improvement in Time-Per-Output-Token (TPOT) is confirmed, the TTFT degradation appears to be an inherent overhead of the speculative decoding process that might be more pronounced than previously understood.

Report of performance regression

I started a vLLM server with a target model and enable a draft model via --speculative-config. Example for EAGLE3 on GPU:

vllm serve /path/to/Qwen3-8B \
    --served-model-name qwen3-8b \
    --port 8188 \
    --enforce-eager \
    --speculative-config '{"method": "eagle3", "model": "/path/to/Qwen3-8B-speculator.eagle3", "num_speculative_tokens": 15}'

Then I ran a benchmark using the vllm bench serve command with a custom dataset (MATH-500):

vllm bench serve \
    --model /path/to/Qwen3-8B \
    --base-url http://127.0.0.1:8188 \
    --served-model-name qwen3-8b \
    --dataset-name custom \
    --dataset-path /path/to/dataset.jsonl \
    --num-prompts 50 \
    --max-concurrency 1 \
    --temperature 0

Baseline (No Speculative Decoding):

• Mean TPOT: ~28.20 ms
• Mean TTFT: ~73.97 ms
• P99 TTFT: ~249.90 ms

With Speculative Decoding (EAGLE3):

• Mean TPOT: ~17.11 ms (a 39.3% improvement, which is great)
• Mean TTFT: ~158.44 ms (a 114% increase)
• P99 TTFT: ~1078.03 ms (a 331% increase).

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

My env is: NVIDIA V100 vllm 0.17.0 Model: Qwen3-8B(Target) & corresponding EAGLE-3 draft model

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.