Motivation.

Implement a "selective offloading" directive in llm-d/vLLM offload path, allowing CPU and FS offloading connectors (and possibly other connectors), to only offload parts of the prompt outside of GPU Memory.

At the moment, there is no control available to control KV Cache offloading. It's all (all generated KVCache is offloaded during the forward pass) or nothing (when the connector is disabled).

Public traces like Mooncake or Alibaba show that 40-60% of the KV Cache is stored but never reused. Creating a mechanism to only store some tokens while discarding others can allow vLLM orchestrators (e.g. llm-d) to implement policies that can improve performance or reduce cost in certain scenarios. A few non comprehensive example given:

• Long one-off requests in between more common prompts;
• KV Cache stored on some remote server with limited bandwidth.

See also the llm-d Context-Aware KVCache retention.

Proposed Change.

I have implemented a draft in this fork, adding an API to the prompt that allow storing only some configurable percentage of the prompt, starting from the beginning and stop when prompt percentage is reached. This is not the final design, and it was implemented only to test the hypothesis (results coming soon).

Implementation discussion.

As of right now, the ^^^aforementioned fork adds a standalone directive in the call to instruct the offloadConnector. Another option would be to extend the APIs discussed in RFC #37003

Feedback Period.

1-2 Weeks.

Feedback welcome, especially regarding:

• API shape and abstractions;
• Implementation logic and location, e.g. kv_connector vs OffloadConnector;
• Alignment with the vLLM and llm-d project goals.

CC List.

@vMaroon @animeshtrivedi

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.