Motivation

In production LLM serving, many requests share common prompt prefixes — system prompts, few-shot examples, RAG context, multi-turn conversation history, etc. vLLM's local prefix caching effectively reuses KV cache within a single instance, but it cannot help in the following scenarios:

• Cross-instance reuse: Multiple vLLM instances serving similar traffic recompute the same KV blocks independently.
• Post-eviction recomputation: After KV cache eviction due to memory pressure, the same blocks must be recomputed from scratch.
• Cold-start warm-up: A newly launched instance has an empty KV cache and must compute all blocks, even those that have been computed elsewhere.

A Mooncake Store Connector addresses these gaps by using Mooncake's distributed store as a shared KV cache pool. Computed KV cache blocks are stored with content-addressable keys (block hashes). Before computing a prefill, the engine queries the store — if the blocks already exist, they are loaded directly, skipping redundant prefill computation.

This capability is orthogonal to PD disaggregation and benefits any vLLM deployment topology.

Current Implementation in vllm-ascend

We have implemented and validated this approach in the vllm-ascend project (the Ascend NPU hardware plugin for vLLM). The implementation is located at vllm_ascend/distributed/kv_transfer/kv_pool/ascend_store/. Below we describe its architecture.

Overall Architecture

┌──────────────────────────────────────────────────────────────────────┐
│                     AscendStoreConnector                             │
│                   (KVConnectorBase_V1)                               │
│                                                                      │
│   Scheduler Process                    Worker Process(es)            │
│  ┌────────────────────┐              ┌──────────────────────────┐    │
│  │  KVPoolScheduler   │    ZMQ RPC   │     KVPoolWorker         │    │
│  │                    │◄────────────►│                          │    │
│  │  - Lookup cache hit│              │  - Register KV caches    │    │
│  │  - Build metadata  │              │  - LookupKeyServer       │    │
│  │  - Track requests  │              │    (rank 0 only)         │    │
│  └────────────────────┘              │  - Transfer threads      │    │
│                                      └────────────┬─────────────┘    │
│                                                   │                  │
│                                      ┌────────────▼─────────────┐    │
│                                      │   SendingThread          │    │
│                                      │   RecvingThread          │    │
│                                      │   (+ layerwise variants) │    │
│                                      └────────────┬─────────────┘    │
│                                                   │                  │
│                                      ┌────────────▼─────────────┐    │
│                                      │  MooncakeDistributedStore │    │
│                                      │  + TransferEngine         │    │
│                                      │  (zero-copy RDMA/DMA)    │    │
│                                      └──────────────────────────┘    │
└──────────────────────────────────────────────────────────────────────┘

Component Details

1. Connector Entry Point

AscendStoreConnector implements KVConnectorBase_V1 and splits behavior by role:

• KVConnectorRole.SCHEDULER: Creates KVPoolScheduler to handle cache hit queries and metadata construction.
• KVConnectorRole.WORKER: Creates KVPoolWorker to handle KV cache registration, backend interaction, and background transfer threads. Worker at rank 0 additionally starts a LookupKeyServer — a ZMQ REP server that handles cache existence queries from the scheduler.

Key configuration options:

• use_layerwise: Enable per-layer transfer (pipelined with forward pass) vs. whole-request transfer.
• consumer_is_to_put: Allow consumer-role instances to also write to the store.
• load_async: Enable async loading between scheduler steps.

2. Scheduler Side — KVPoolScheduler

The scheduler orchestrates KV cache reuse decisions without direct access to the distributed store.

Cache Hit Query (get_num_new_matched_tokens):

The scheduler sends the request's block hashes and token length to worker rank 0 via ZMQ (LookupKeyClient → LookupKeyServer). The worker queries the Mooncake Store for block existence and returns the number of prefix tokens that are cached. The scheduler then tells vLLM to allocate blocks for these tokens without scheduling them for compute.

Metadata Construction (build_connector_meta):

For each scheduled request, the scheduler produces a ReqMeta object containing:

• load_spec: How many tokens to load from the store (if cache hit).
• can_save: Whether to store the computed blocks after prefill.
• block_ids: Local block IDs for memory address calculation.
• block_hashes: Content hashes for key generation.

These are bundled into AscendConnectorMetadata and passed to workers via SchedulerOutput.

Request Lifecycle Tracking:

The scheduler maintains RequestTracker objects that track per-request state: how many tokens have been saved so far, allocated block IDs, and chunk boundaries. This enables correct handling of chunked prefill, preemption, and resumed requests.

3. Worker Side — KVPoolWorker

The worker manages the actual data movement between local KV cache and the distributed store.

KV Cache Registration (register_kv_caches):

When vLLM allocates KV cache tensors, the worker:

1. Computes base_addr and block_len for each layer's KV cache tensor.
2. Registers all memory regions with the Mooncake Transfer Engine (register_buffer), making them accessible for zero-copy RDMA transfer.
3. Stores the address mapping in ChunkedTokenDatabase for later address calculation.

Backend Selection:

The worker dynamically loads the storage backend based on configuration (backend field in kv_connector_extra_config). The vllm-ascend implementation supports mooncake, memcache, and yuanrong backends.

Transfer Thread Management:

Based on the role and configuration, the worker spawns daemon threads:

• Producer role (kv_producer / kv_both): starts a SendingThread.
• Consumer role with async load: starts a RecvingThread.
• Layerwise mode: uses LayerSendingThread / LayerRecvingThread instead.

4. Content-Addressable Key Design

KV cache blocks are addressed by a composite key derived from vLLM's native BlockHash:

{model_name}@pcp{pcp_rank}@dcp{dcp_rank}@head_or_tp_rank:{rank}@pp_rank:{rank}@{block_hash_hex}

• block_hash_hex: vLLM's BlockHash converted to hex string. This is a content hash of the token sequence within the block, ensuring identical prompt prefixes produce identical keys regardless of which instance computed them.
• Parallelism-aware encoding: TP/PP/PCP/DCP rank fields ensure each parallel rank stores and retrieves its own KV head partition correctly.
• Layerwise key variant: For per-layer transfer, a @{layer_id} suffix is appended.

ChunkedTokenDatabase.process_tokens() iterates over a request's block hashes, producing (start_idx, end_idx, PoolKey) tuples. prepare_value() then maps each block to physical memory addresses:

addr = kv_cache_base_addr + block_id * block_len
size = block_len / block_size * (end - start)

This enables the store to read from / write to exact memory locations in vLLM's paged KV cache, achieving zero-copy transfer.

5. Async Transfer Threads

Transfer threads run as daemon threads, consuming requests from a queue:

SendingThread (KVCacheStoreSendingThread):

1. Dequeue a ReqMeta from the request queue.
2. Generate keys via process_tokens().
3. Call exists() to skip already-stored blocks (deduplication).
4. Compute memory addresses via prepare_value().
5. Wait for GPU/NPU compute event synchronization (ensuring the KV data is ready).
6. Call put(keys, addrs, sizes) to write blocks to the store.
7. Optionally generate BlockStored KV cache events for external event consumers.

RecvingThread (KVCacheStoreRecvingThread):

1. Dequeue a ReqMeta with load_spec.
2. Generate keys and compute target memory addresses.
3. Call get(keys, addrs, sizes) to read blocks directly into local KV cache memory.
4. Mark the request as finished.

Layerwise Variants:

LayerSendingThread and LayerRecvingThread operate per-layer, enabling pipelined transfer during the forward pass. This is compatible with vLLM's save_kv_layer() / wait_for_layer_load() interfaces.

6. Mooncake Store Integration

The Mooncake Store backend uses two Mooncake components:

• MooncakeDistributedStore: A distributed key-value store providing batch_put_from_multi_buffers, batch_get_into_multi_buffers, and batch_is_exist APIs.
• TransferEngine: Manages RDMA connections and memory registration for zero-copy transfer.

Initialization flow:

1. Load config from JSON file (MOONCAKE_CONFIG_PATH): metadata server address, segment sizes, protocol, device name.
2. Create TransferEngine (global singleton with double-checked locking) and initialize RDMA connections.
3. Call MooncakeDistributedStore.setup() with the transfer engine, connecting to the metadata server.
4. During register_buffer(), register KV cache memory regions with the transfer engine.

Data transfer operations (put / get) pass memory addresses and sizes directly to the Mooncake store, which handles RDMA-based zero-copy transfer between devices through the distributed store.

End-to-End Data Flow

Request Arrives
      │
      ▼
┌─────────────────────────────────────────────────────────────┐
│ Scheduler: get_num_new_matched_tokens()                     │
│   1. Send block_hashes to LookupKeyServer via ZMQ           │
│   2. Worker queries Mooncake Store: exists(keys)             │
│   3. Return: N tokens are cached externally                  │
│   4. Scheduler allocates blocks, marks N tokens as "load"    │
└─────────────────────────────┬───────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Scheduler: build_connector_meta()                           │
│   - For cached tokens: ReqMeta(load_spec=LoadSpec(...))     │
│   - For new tokens: ReqMeta(can_save=True)                  │
│   → Passed to Worker via SchedulerOutput                    │
└─────────────────────────────┬───────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Worker: start_load_kv()                                     │
│   - For requests with load_spec:                            │
│     → RecvingThread.get(keys, addrs, sizes)                 │
│     → Blocks loaded from Mooncake Store into local KV cache │
└─────────────────────────────┬───────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ vLLM Forward Pass                                           │
│   - Loaded blocks: KV cache is already populated            │
│   - New blocks: Computed normally by the model              │
└─────────────────────────────┬───────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Worker: wait_for_save()                                     │
│   - For newly computed blocks:                              │
│     → Record GPU/NPU event                                  │
│     → SendingThread.put(keys, addrs, sizes)                 │
│     → Blocks stored to Mooncake Store for future reuse      │
└─────────────────────────────────────────────────────────────┘

Proposed Change for vLLM Implementation

Based on the validated architecture in vllm-ascend, we propose adding a MooncakeStoreConnector to vLLM. Below is an initial design suggestion. We welcome community discussion on the design details.

Suggested File Structure

vllm/distributed/kv_transfer/kv_connector/v1/mooncake_store/
├── __init__.py
├── mooncake_store_connector.py   # MooncakeStoreConnector(KVConnectorBase_V1)
├── store_scheduler.py             # Scheduler-side: lookup + metadata
├── store_worker.py                # Worker-side: threads + store interaction
├── kv_transfer.py                 # Background send/recv threads
└── config_data.py                 # Key generation, metadata structures

Key Design Points

1. Connector Interface

MooncakeStoreConnector implements KVConnectorBase_V1 with the standard scheduler/worker role split. Scheduler-side handles cache hit lookup and metadata construction; worker-side handles KV cache registration, transfer thread management, and Mooncake Store interaction.

2. Content-Addressable Keys

Reuse vLLM's native BlockHash as the core of the cache key. The key format encodes model name, parallelism ranks (TP/PP), and block hash to ensure correctness under distributed parallelism.

3. Zero-Copy Transfer

Use Mooncake's TransferEngine for RDMA memory registration and MooncakeDistributedStore for distributed put/get operations. Memory addresses point directly into vLLM's paged KV cache tensors, avoiding intermediate copies.

4. Async Transfer

Background daemon threads handle put/get operations asynchronously, decoupled from the forward pass. Layerwise variants enable pipelined transfer during model execution. GPU event synchronization ensures data consistency.

5. Deduplication

Before storing, the sending thread calls exists() to check which blocks are already in the store, skipping redundant writes. This is a natural consequence of content-addressable keys.

6. Configuration

# Example configuration
serve_args = {
    "kv_connector": "MooncakeStoreConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
        "use_layerwise": False,
        "load_async": True,
    }
}

# Mooncake Store configuration
export MOONCAKE_CONFIG_PATH=/path/to/config.json

Open Discussion

We'd like to discuss the following aspects with the community:

• Naming and placement: Is mooncake_store/ the right location? Should this be a submodule of the existing mooncake/ directory?
• Generalization: Should the store interaction be abstracted to support other distributed stores in the future, or should we keep it Mooncake-specific for now?
• Integration with existing prefix caching: How should the store connector interact with vLLM's local prefix caching (KVCacheManager)? Should there be a unified hierarchy (local cache → shared store)?
• Eviction policy: Should the store connector participate in cache eviction decisions, or purely act as a passive store?

References

• vllm-ascend implementation: vllm_ascend/distributed/kv_transfer/kv_pool/ascend_store/
• Mooncake project: https://github.com/kvcache-ai/Mooncake
• vLLM V1 KV Connector interface: vllm/distributed/kv_transfer/kv_connector/v1/base.py

CC List

@ivanium @stmatengss @dtcccc @Pz1116