vllm - 💡(How to fix) Fix [Performance]: KV Cache FP8 unexpectedly slow on batched serving [1 comments, 2 participants]

Proposal to improve performance

No response

Report of performance regression

Summary

When benchmarking FP8 vs BF16 on gpt-oss-20b with vLLM on a single H100, FP8 performs as expected (equal or slightly better) for single-concurrency workloads, but is ~10% slower than BF16 under a concurrent mixed-length workload (concurrency 8, 150 requests, ~20K input tokens).

Environment

• Model: openai/gpt-oss-20b
• vLLM version: 0.19.0
• GPU: 1x H100

FP8:

python3 -m vllm.entrypoints.openai.api_server \
    --model openai/gpt-oss-20b \
    --kv-cache-dtype "fp8" \
    --no-enable-prefix-caching

BF16:

python3 -m vllm.entrypoints.openai.api_server \
    --model openai/gpt-oss-20b \
    --no-enable-prefix-caching

Benchmark Setup

Short context (concurrency 1, 10 requests, 512 in / 512 out):

vllm bench serve --dataset-name sonnet --sonnet-input-len 512 --sonnet-output-len 512 --max-concurrency 1 --num-prompts 10

Long context (concurrency 1, 10 requests, 130K in / 512 out):

vllm bench serve --dataset-name sonnet --sonnet-input-len 130000 --sonnet-output-len 512 --max-concurrency 1 --num-prompts 10

Concurrent mixed (concurrency 8, 150 requests, ~20K in / ~2K out, ±15% range):

vllm bench serve --dataset-name random --random-input-len 20000 --random-output-len 2000 --random-range-ratio 0.15 --max-concurrency 8 --num-prompts 150

Results

Observations

• Short context: FP8 has slight quantization overheads that aren't amortized (expected)
• Long context: FP8 ~9% faster on TTFT, ~5% faster on decode (expected)
• Concurrent mixed: BF16 is ~10% higher throughput, ~34% lower TTFT, ~8% lower TPOT — unexpected

I am wondering what could be going wrong here?

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.