vllm - 💡(How to fix) Fix [RFC]: O(1) KV Cache for vLLM: 4.8x Speedup & 22x More Accurate than TurboQuant on Qwen2.5-7B [1 comments, 2 participants]

🚀 The feature, motivation and pitch

Hi vLLM team and community,

First, thank you for building and maintaining PagedAttention—it’s the absolute gold standard for LLM serving.

I’m opening this RFC to discuss a potential architectural pathway for extreme long-context serving (128k+). Currently, as context length $T$ grows, the memory bandwidth required during the generation phase scales at $O(T)$. While quantization (FP8, AWQ, or methods like TurboQuant/QJL) reduces the footprint, it often introduces significant variance explosion (degrading model quality) and still fundamentally scales linearly.

Our research team (Singularity Principle) has developed a different mathematical approach based on trace-class admissibility, resulting in the Nuclear ZFC (NZFC) architecture. By maintaining a bounded prototype memory and using an online mass-bias direct readout, we can execute attention without fully materializing the past KV cache, freezing the query readout complexity at strictly $O(1)$.

We recently completed a PyTorch-level PoC on Qwen2.5-7B (GQA, 1024 ctx). The benchmark directly compares our scatter_add_ based FixC engine against the TurboQuant baseline.

Hard Metrics:

• Query Latency: Flat at 1.04ms ($O(1)$), compared to the baseline scaling to 16.22ms at 1K tokens.
• End-to-End Speedup: 3.66ms vs 17.65ms (4.8x faster).
• Quality (rel-L2 Error): 0.020, which is 22x more accurate than the 1-bit QJL baseline (0.437).

<img width="1589" height="495" alt="Image" src="https://github.com/user-attachments/assets/2cb1fa94-a8db-4977-9835-716e0fbde6e3" /> <img width="1390" height="397" alt="Image" src="https://github.com/user-attachments/assets/1dd95c60-afc3-46a2-90ea-bfade753204b" />

You can reproduce these results using our Colab PoC here: 👉 https://colab.research.google.com/drive/1tISt1MWcti8oubURkDhTlwS7rf_BG4wB?usp=sharing (Note: Core IP is patent-pending (PCT/KR2026/002215) but fully open for open-source evaluation and academic review).

Integration Question for vLLM Maintainers: Our next milestone is writing custom Triton/CUDA kernels for the FixC engine (scatter_add_ updates and mass-log bias readout).

Given vLLM's PagedAttention architecture, what would be the most non-intrusive way to introduce a custom $O(1)$ attention backend? Specifically, since NZFC modifies both the cache update (in-place accumulation) and the readout mechanics, should we look into adding a new AttentionBackend subclass, or would this require deeper modifications to the Block Manager?

We would deeply appreciate any architectural feedback, code roasting, or pointers on how to best integrate this with vLLM's ecosystem.

Thanks!

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.