vllm - 💡(How to fix) Fix [RFC]: [Roadmap] Mooncake Store Connector Feature Enrichment Roadmap

Summary

The Mooncake Store Connector currently provides basic KV cache prefill-offloading via prefix caching. This RFC tracks extending it into a full-featured disaggregated KV cache layer — covering tiered storage, scheduler-level coordination, advanced model architectures (HMA / GQA), and transfer-compute overlap — to meet the demands of a wider range of models and serving scenarios in production.

Category 1: Infrastructure — Store Backend Capabilities

• Standalone Store Architecture — Support Mooncake store's standalone mode, where a dedicated process allocates and manages Mooncake store memory segments. This reduces memory allocation fragmentation and minimizes the metadata size maintained in the Mooncake master. (Done in #42689)

• SSD Offload — Support SSD as a tiered storage backend. KV cache blocks are first buffered in CPU memory, with support for asynchronous flush to SSD or write-on-eviction from CPU memory. On read, SSD data is loaded into a staging buffer before being served to GPU, dramatically increasing effective KV cache capacity. (Done in #42689)

• Memory Semantics — Unify the address space of HBM and DRAM within a supernode, leveraging the hyperplane to enable direct transfer between HBM and DRAM.

Category 2: Scheduler Integration — Coordination with Upper-Layer Scheduling

• KV Cache Event Propagation — Emit KVCacheEvent objects (e.g., BlockStored, BlockRemoved) from the store connector so that external orchestrators (global schedulers, routing layers) can observe cache state changes in real time, enabling cache-aware request routing across engines.

• Recompute on Get Failure — When a load operation fails (e.g., remote block evicted between lookup and transfer), report the affected block IDs back to the scheduler so it can trigger recomputation of those tokens instead of serving stale/missing KV data. (Ref: vllm-ascend#9701)

Category 3: Model Adaptation — Supporting Diverse Model Architectures

• Decode-Phase KV Cache Put with GQA TP Asymmetry — Support saving KV cache generated during the decode phase for disaggregated P/D architectures. The main challenge is that under GQA with asymmetric TP, the KV cache produced during decode must be transformed before it can be reused by prefill.

• Hybrid Attention — Support models with heterogeneous attention types across layers (e.g., DeepSeek-V4, Qwen3.5). (Done in #42828)

Category 4: Transfer Optimization — Improving Transfer Efficiency

• Layer-wise KV Transfer — Support per-layer granularity for KV cache put/get operations. This enables overlapping KV transfer with model computation.

CC List.

@ivanium @stmatengss @Pz1116