PR #39752: add warning when FP8 KV cache misses prefill query quantization

• Repository: vllm-project/vllm
• Author: qiching
• State: closed | merged: True
• Link: https://github.com/vllm-project/vllm/pull/39752

Description (problem / solution / changelog)

Purpose

add a startup warning log when FP8 KV cache is enabled (--kv-cache-dtype fp8) but use_prefill_query_quantization is not set, so users are aware of the FP8 prefill attention option.

Resolves #39751

Background: Per PR #31195 by @pavanimajety, FP8 prefill query quantization was intentionally gated behind use_prefill_query_quantization=true because it regresses short-sequence performance (~20% at ISL=1024). However, for long-context workloads (ISL >= 4K), the FP8 FMHA kernel is significantly faster (up to 1.4x at 128K). Currently, users with --kv-cache-dtype fp8 silently get BF16 prefill attention with no indication that a faster path exists.

This PR adds a warning_once log in determine_prefill_query_data_type() for the case where FP8 KV cache is active and the backend supports FP8 prefill, but the user has not enabled it. The warning includes the exact --attention-config flag.

no default behavior is changed.

Test Plan

1. Start vLLM with FP8 KV cache but without the flag:

vllm serve <model> --kv-cache-dtype fp8

Verify the warning appears in startup logs:

WARNING: FP8 KV cache is enabled but prefill queries are not quantized to FP8. 
For long-context workloads (ISL >= 4K), enabling FP8 prefill attention can 
significantly improve prefill latency. To enable, add: 
--attention-config '{"use_prefill_query_quantization": true}'

1. Start with the flag enabled — verify no warning, and the existing info log appears instead:

vllm serve <model> --kv-cache-dtype fp8 \
  --attention-config '{"use_prefill_query_quantization": true}'

Expected: INFO: FP8 prefill attention enabled: query data type is FP8

1. Start without FP8 KV cache — verify no warning:

vllm serve <model>

Test Result

Profiling results on GB300 (DeepSeek-R1-FP4, DP=4, ISL=128K) showing the speedup users miss without this flag:

Short-Seq Benchmark

To verify whether the cast overhead has improved since the original analysis (Dec 2025), we ran end-to-end benchmarks on GB300 with the following setup:

• Model: DeepSeek-R1-0528-FP4
• Parallelism: DP=4 (matching Pavani's PR #31195 config, but we used max-model-len=34816 to also test ISL=4096)
• KV cache: fp8
• Platform: GB300

Findings:

• ISL=1024: FP8 prefill still regresses ~20% in throughput, confirming the current default is correct.
• ISL=4096: Crossover point — FP8 is ~3.6% faster.

Changed files

• vllm/model_executor/layers/attention/mla_attention.py (modified, +13/-0)