vllm - 💡(How to fix) Fix [RFC]: Fast Runtime Resize of a Pre-Built EP Topology

Motivation.

The current ElasticEP workflow in vLLM performs runtime EP scaling by reconstructing communication and runtime resources whenever the topology changes, resulting in scaling latencies of tens of seconds. In many deployments, however, the maximum resource budget is fixed and known in advance. This creates an opportunity to pre-initialize the maximum topology once at startup and reuse it across scaling events. The goal of this proposal is to perform EP scaling by adjusting only the active subset of that topology, reducing scaling latency to a few seconds while maintaining correctness.

Proposed Change.

Scale down

• Remap experts onto the active ranks.
• Offload weights and KV cache from inactive ranks.
• Shrink the active communication group.
• Remove inactive ranks from communication and scheduling.

Scale up

• Restore weights and KV cache for reactivated ranks.
• Expand the active communication group.
• Restore expert placement across the enlarged rank set.
• Re-enable communication and request routing for the reactivated ranks.

<div align="center"> <img src="https://github.com/user-attachments/assets/401be053-95c1-46a8-981d-b9fd524e5053" alt="Architecture" /> </div>

Feedback Period.

No response

CC List.

No response

Any Other Things.

We submitted #44994 to demonstrate the effectiveness of this approach.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.