Summary

Dynamic MoE expert weight offloading for vLLM. Expert weights live in CPU pinned memory; a fixed-size GPU cache holds the hottest experts; LRU eviction and cross-layer prediction minimize cache misses. Models that exceed GPU VRAM can run on smaller hardware.

PR 1 is open: #37190 (~600 LOC Python, passing CI).

This RFC covers the full 3-PR architecture and provides production data from tinyserve, an independent implementation of the same techniques (30 tok/s decode on 8 GB GPU, 325 tests).

Motivation

Large MoE models (DeepSeek-V3 671B, Qwen3.5-122B) don't fit in a single GPU. Only a small subset of experts activates per token (e.g., 8 of 256), so most expert weights sit idle. Moving cold experts to CPU and caching hot ones on GPU lets these models run on hardware that would otherwise OOM.

Prior art in vLLM

Key design principle

The cache is a weight provider, not a special forward path. The kernel does not know or care where weights came from. No bypass of the runner pipeline.

Production data (tinyserve)

RTX PRO 2000 8 GB, GPT-OSS-20B MXFP4, 238 cache slots, single-stream decode:

Caveat: These numbers are single-stream on a laptop GPU. Multi-user batched inference on H100 will have different bottlenecks (higher H2D bandwidth, but batch diversity may reduce hit rates). In-tree benchmarks will accompany each PR.

Architecture

ExpertWeightProvider (ABC)
├── FullGPUProvider        -- zero-cost passthrough (default, no overhead)
└── CachedWeightProvider   -- GPU LRU cache + CPU backing store
      ├── GPUSlotManager   -- fixed-address GPU buffers
      ├── LRUEviction      -- pure-Python eviction (collections.OrderedDict)
      └── CPUBackingStore   -- pinned DRAM, all local experts

Integration point

At FusedMoEModularMethod.apply() — replace direct layer.w13_weight access with provider.prepare(topk_ids). No bypass of the runner. All paths go through runner.forward() -> quant_method.apply(), preserving EP dispatch, DP chunking, and shared-expert overlap.

prepare() contract

class ExpertWeightProvider(ABC):
    @abstractmethod
    def prepare(self, topk_ids: torch.Tensor) -> ExpertWeightResult:
        """Ensure requested experts are GPU-resident. Returns GPU tensors."""
        ...

@dataclass
class ExpertWeightResult:
    w1: torch.Tensor       # [capacity, ...] GPU-resident, fixed address
    w2: torch.Tensor       # [capacity, ...] GPU-resident, fixed address
    topk_ids: torch.Tensor # always remapped to slot indices
    w1_scale: Optional[torch.Tensor] = None
    w2_scale: Optional[torch.Tensor] = None

Design choices driven by torch.compile compatibility (per @zou3519's review of #29941):

• topk_ids is always remapped — no boolean flag that changes tensor interpretation
• Scales are fixed attributes, not a dynamic dict — avoids graph breaks
• GPU buffers allocated once at init (fixed addresses for CUDA graph capture)
• prepare() is inherently dynamic Python (eviction, H2D copies) and must run outside any torch.compile boundary

Quant-agnostic tensor registration

Each quant method declares what tensors to cache. The cache stores them as opaque blobs by name — adding a new quant format requires zero cache code changes.

EP compatibility

The provider composes expert_map (global->local) with cache mapping (local->slot) into a single GPU mapping tensor. The kernel's existing -1 skip logic works unchanged.

Batched prefill

prepare() accepts batch topk_ids, deduplicates to unique expert IDs, loads each once. At 3K context with top_k=4 and 32 experts: 32 loads vs 12K sequential — 375x reduction. Without this, prefill is the #1 bottleneck.

CUDA graph compatibility (PR 2)

GPU buffers are fixed-address. A persistent int32 mapping tensor [num_experts] -> slot_index is updated by prepare() on CPU before graph replay. Inside the graph, slot lookup is pure indexing (slot_ids = mapping[topk_ids]) — no Python, no control flow. Same pattern as KV cache block tables.

What prepare() does NOT do

• No CPU fallback computation. If more experts are needed than cache capacity, prepare() raises a clear error with guidance to increase --moe-expert-cache-size. Hand-rolled CPU MoE with Python loops is a correctness trap.
• No silent config downgrade. If the user requests offloading and it's incompatible, that's an error, not a silent no-op.

Phased PR plan

PR 1 scope (what ships)

• ExpertWeightProvider ABC + FullGPUProvider (zero-cost passthrough)
• CachedWeightProvider with LRU eviction via collections.OrderedDict (no external deps)
• Integration into FusedMoEModularMethod.apply() — the bypass path (_forward_with_expert_cache) is deleted
• Synchronous H2D copies (no streams, no prefetch)
• --enforce-eager required (documented)
• BF16 + FP8 per-tensor scale support
• Tests: unit tests for cache logic + integration test through the full runner path
• Zero overhead on default path (no cache): one if provider is not None check per layer

PR 1 limitations (stated explicitly)

• Not for latency-sensitive production serving. Synchronous H2D + no CUDA graphs = batch inference and evaluation only.
• Single-GPU only. TP stall behavior from cache misses is not characterized. Use on multi-GPU TP setups is unsupported until timing analysis is done.
• LRU only. Other eviction policies (LFU, SLRU, ARC) ship in follow-ups if workload data justifies them.

Default-path overhead

When moe_expert_cache_size == 0 (99%+ of users): one if check per layer per forward pass. No allocations, no imports, no code paths touched. FullGPUProvider is a passthrough that returns the existing weight tensors unchanged.

Open questions

1. Integration depth in PR 1: Should ExpertWeightProvider.prepare() be called inside FusedMoEModularMethod.apply(), or should it be lifted to the model runner level (outside compile boundary) from the start? The former is simpler for PR 1; the latter is required for PR 2. Preference?

2. Memory accounting: GPU slot buffers are capacity * expert_size (e.g., 32 slots x 67MB = ~2.1 GB for DeepSeek-V3 BF16). Should these be allocated during vLLM's memory profiling phase so they're visible to the memory profiler, or is a separate accounting path acceptable?

3. Observability timeline: Production deployers want per-layer hit/miss metrics exposed to Prometheus from day one. Should basic counters ship in PR 1 (adds ~30 LOC) or is PR 3 acceptable?

4. Policy ABC surface: For future pluggable eviction (ARC, LIRS), the policy interface needs to receive hit/miss/recency signals, not just eviction requests. Should PR 1's LRU implementation use a policy ABC that's ARC-ready, or is a simple OrderedDict wrapper sufficient?

CC

@mgoin @zou3519 @pavanimajety

References

• RFC #33869 — prior MoE offload RFC
• PR #37190 — PR 1 implementation (open)
• PR #34535 — selective CPU offload (merged, static)
• PR #29941 — async prefetch handler (merged, pattern for PR 2)
• tinyserve — independent validation, 325 tests
• FATE — cross-layer temporal prediction (83% cosine similarity, 97-99% prediction accuracy)

Architecture validated in tinyserve. AI-assisted drafting (Claude Code).