Issue Station
Sign In
Tools hereToolbox

vllm - ✅(Solved) Fix [Bug]: EP support for Fused MoE LoRA is not implemented yet. [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#38005•Fetched 2026-04-08 01:21:59
View on GitHub
Comments
0
Participants
1
Timeline
3
Reactions
0
Author
Nero10578Nero10578
Participants
Nero10578Nero10578
Timeline (top)
cross-referenced ×1labeled ×1referenced ×1

Fix Action

Fixed

PR fix notes

PR #38021: lora: add EP support for FusedMoEWithLoRA

Description (problem / solution / changelog)

Summary

Fixes #38005

Expert Parallelism (EP) was unconditionally blocked when using LoRA with Fused MoE layers via a hard assert not self.base_layer.use_ep. This prevented running LoRA on models like GLM that require EP for TP>2.

Root cause

When EP is active, each rank only owns a subset of the global experts (local_num_experts < global_num_experts). The LoRA stacked weight buffers are correctly sized for local_num_experts, but the incoming tensors in set_lora() have shape (global_num_experts, ...) — so a scatter step was missing.

Changes

  • Remove the assertion blocking EP in FusedMoEWithLoRA.__init__
  • Add _filter_experts_for_ep() — called at the top of set_lora() before any TP slicing. Uses base_layer._expert_map (global→local index, -1 for non-local) to scatter only locally-owned expert rows into the right local positions. Handles the ROCm AITER edge case where _expert_map is extended with shared-expert entries.

The forward-pass machinery (weight allocation, kernel dispatch, expert_map threading through the decorators) already operated on local_num_experts correctly — the only missing piece was the weight-loading path.

Testing

Existing non-EP LoRA MoE paths are unaffected: when _expert_map is None (ep_size==1), _filter_experts_for_ep is a no-op.

Changed files

  • vllm/lora/layers/fused_moe.py (modified, +48/-6)
RAW_BUFFERClick to expand / collapse

Your current environment

Does not concern the question.

🐛 Describe the bug

What is the limitation preventing EP from being used with Fused MoE LoRA? It prevents running LoRA at all on older models like GLM which requires EP for TP>2.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

To enable EP with Fused MoE LoRA for older models like GLM, we need to modify the model configuration and the LoRA implementation.

Steps to Fix

  • Update the model configuration to support EP with Fused MoE LoRA:
    • Set use_ep to True in the model config.
    • Set tp to the desired value (e.g., 4).
  • Modify the LoRA implementation to support EP:
    • Update the forward method to handle EP correctly.
    • Add support for EP in the LoRA class.

Example Code

# Update model config
model_config = {
    'use_ep': True,
    'tp': 4
}

# Modify LoRA implementation
class LoRA(nn.Module):
    def __init__(self, config):
        super(LoRA, self).__init__()
        self.use_ep = config['use_ep']
        self.tp = config['tp']

    def forward(self, x):
        if self.use_ep:
            # Handle EP correctly
            x = x.view(-1, self.tp, x.size(1) // self.tp)
            x = x.permute(0, 2, 1).contiguous()
        # Rest of the forward method implementation
        return x

Verification

To verify the fix, run the model with EP enabled and check that it works correctly for older models like GLM with tp greater than 2.

Extra Tips

  • Make sure to update the documentation to reflect the changes to the model configuration and LoRA implementation.
  • Test the model thoroughly to ensure that it works correctly with EP enabled.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#memory management#API rate limit#retriever error#indexing error#inference speed

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

vllm - ✅(Solved) Fix [Bug]: EP support for Fused MoE LoRA is not implemented yet. [1 pull requests, 1 participants]