vllm - 💡(How to fix) Fix [Proposal] Topology-Aware KV Cache Compression for Memory-Efficient Inference [3 comments, 2 participants]

Proposal to improve performance

[Proposal] Topology-Aware KV Cache Compression for Memory-Efficient Inference

Is your feature request related to a problem? Please describe.

Long-context LLM inference (e.g., 128K tokens with DeepSeek-V3, Llama 3.1, etc.) faces significant KV cache memory pressure, limiting batch size, throughput, and real-time service stability. Current KV cache compression methods (e.g., SnapKV, StreamingLLM, H2O) don't fully leverage directional topology information or adapt to phase-dependent attention patterns.

Describe the solution you'd like

This proposal introduces a topology-aware KV cache compression framework with:

• Selective directional pruning: Retains top-k dense directions based on sparsity thresholds, reducing quantization noise in low-bit regimes
• Four-phase lifecycle compression: Adaptive compression rates across Initial → Mid → Recent → Terminal stages, mirroring natural attention evolution
• NPU-optimized matrix operations: Exponential-free computation through matrix-only operations, enabling efficient NPU acceleration

Expected Impact

• 70-85% KV cache memory reduction for long-context workloads
• 15-20% FLOPs margin improvement, enhancing real-time service stability
• Particularly effective for Mixture-of-Experts (MoE) architectures like DeepSeek-V3, Qwen2.5-MoE, etc.

Relevant Context

This approach extends attention head importance scoring principles to KV cache directional sparsity. The framework is designed to integrate with vLLM's PagedAttention architecture and is compatible with:

• DeepSeek-V3's MLA (Multi-head Latent Attention)
• MoE routing patterns for dynamic compression
• NPU backends (Huawei Ascend, etc.) via matrix-optimized operations

Additional Context

• Full technical proposal (12-page detailed specification) available upon request
• Already shared with DeepSeek team via official channels
• Seeking community feedback and potential collaboration on vLLM integration

Would you be willing to contribute?

Yes, willing to collaborate on implementation, testing, and benchmarking within vLLM's existing attention backend architecture.

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.