vllm - 💡(How to fix) Fix [Performance]: Massive increase (~10x) in KV Cache capacity for Gemma 4 in v0.21.0

Proposal to improve performance

No response

Report of performance regression

No response

Misc discussion on performance

Obersvation While testing Gemma 4 (specifically using cyankiwi/gemma-4-31B-it-AWQ-4bit) on an A100 80GB, I observed a massive increase in available KV cache capacity between vLLM v0.20.0 and v0.21.0.

Comparison Data Using the same args and --gpu-memory-utilization 0.9:

Total available KV cache memory remained constant (~50 GB), indicating the efficiency gain is per token.

Technical Analysis Gemma 4 utilizes several advanced memory-saving features:

1. Alternating Attention: Mix of local sliding-window layers (1024 tokens) and global full-context layers.
2. Shared KV Cache: The last $N$ layers reuse KV states from earlier layers.
3. Dual RoPE: Standard RoPE for sliding layers and pruned RoPE for global layers. (Source: Hugging Face Blog)

It appears that in v0.20.0, the memory profiler may have been treating all layers as global and non-shared, leading to massive over-allocation. In v0.21.0, vLLM seems to correctly recognize the sliding-window budget and shared KV states, unlocking the actual capacity of the architecture.

Request This is a massive win for performance and deployment density. I suggest documenting this improvement in the release notes or the model support page so users know to upgrade for Gemma 4.

Suggested wording

Users upgrading from vLLM v0.20.0 to v0.21.0 will notice a significant increase in available KV cache capacity for Gemma 4 (roughly 9.5x). This is due to optimized support for Gemma 4's unique architecture, including alternating sliding-window attention and shared KV states, which drastically reduces the memory cost per token.

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.