vllm - 💡(How to fix) Fix RFC: Memory-backend interface for KV cache hot paths and attention decode [1 participants]

Motivation.

Hi vLLM team,

I am experimenting with a TurboQuant-compatible memory backend evaluation path, focused on KV-cache residency, kv_dot, and attention decode hot-path pressure.

The goal is not to replace vLLM’s scheduler or execution model, but to explore whether selected KV-cache / attention-decode operations can be routed through an external memory-centric backend while keeping fallback-safe behavior.

Current evaluation repository:

https://github.com/ixu2486/tq_compat_eval

The current direction includes:

• KV-cache hot-path analysis
• TurboQuant-compatible evaluation layout
• memory-backend routing experiments
• PIM-compatible execution path exploration
• fallback-safe CPU/backend comparison
• future compatibility with larger-context inference and speculative decoding pressure points

If the vLLM team or maintainers are interested in testing this direction, please reply here. I can add the relevant personnel to this project repository for evaluation access.

I would also appreciate guidance on whether this kind of work would fit better as:

• an external backend experiment
• a plugin-style integration
• a future RFC
• or a benchmark/evaluation-only companion project

Thanks.

Proposed Change.

Introduce an optional external memory-backend interface for selected KV-cache hot paths, especially kv_dot and attention decode.

The proposed change is not to replace vLLM’s scheduler, executor, or existing attention backends. Instead, it would allow experimental memory-centric backends to evaluate selected KV-cache / attention-decode operations through a fallback-safe interface.

The initial scope could be limited to an external evaluation path or plugin-style backend, with CPU/reference fallback and benchmark-only validation before any deeper integration.

This would make it possible to test TurboQuant-compatible KV residency layouts, memory-backend routing, and PIM-compatible execution paths without requiring disruptive changes to vLLM core.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.