Title

[Bug]: DPEngineCoreProc may re-arm DP wave while paused (START_DP_WAVE ignores pause state), causing collective timeout after pause_generation + collective_rpc

Your current environment

OS                           : Ubuntu 24.04.3 LTS (x86_64)
PyTorch version              : 2.10.0a0+gitb0eb5f7
CUDA used to build PyTorch   : 12.9
Python version               : 3.12
Is CUDA available            : True
CUDA runtime version         : 12.9.86
GPU models and configuration :
GPU 0-7: NVIDIA H20
Nvidia driver version        : 570.124.06
vLLM Version                 : v0.16.1rc0

Model

MoE + DPEP setup (data_parallel_size > 1) using coordinator wave handling (internal DP load balancing / DPLBAsyncMPClient).

Bug description

With DPEP, calling:

1. pause_generation(mode="abort")
2. then collective_rpc(...) for online weight update

can intermittently fail with NCCL ALLREDUCE timeout (600s).

Observed behavior: one engine enters the utility collective_rpc, while a peer engine can still be in dummy-batch ALLREDUCE, leading to collective mismatch and timeout.

Root cause

In DPEngineCoreProc._handle_client_request, START_DP_WAVE can set engines_running = True even when scheduler is paused:

# vllm/v1/engine/core.py — DPEngineCoreProc._handle_client_request
if request_type == EngineCoreRequestType.START_DP_WAVE:
    new_wave, exclude_eng_index = request
    if exclude_eng_index != self.engine_index and new_wave >= self.current_wave:
        self.current_wave = new_wave
        if not self.engines_running:          # no pause-state check
            self.engines_running = True       # re-arms dummy-batch loop

This races with pause completion:

• pause_scheduler() may return immediately when engine appears idle (has_work()==False), i.e. no Future wait path.
• A late START_DP_WAVE from coordinator can still arrive after pause returns.
• Peer engine re-enters running loop and executes dummy-batch ALLREDUCE.
• Next collective_rpc is issued; ranks are in different collectives -> timeout.

Race timeline

Engine 0 (idle)          Coordinator            Engine 1 (idle)
     |                       |                       |
 late req -> FIRST_REQ ----->|---- START_DP_WAVE -->| (queued)
     |                       |                       |
 pause_scheduler ------------|---------------------> pause_scheduler
 returns (idle fast path)                            returns (idle fast path)
     |                       |                       |
 pause_generation returns    |                 handles START_DP_WAVE
     |                       |                 engines_running=True
     |                       |                 -> dummy ALLREDUCE
     |                       |                       |
 collective_rpc ---------------------------------> rank mismatch
                                              -> NCCL timeout

Suggested fix

In DPEngineCoreProc._handle_client_request, gate START_DP_WAVE re-arm while paused:

if not self.engines_running and not self.is_scheduler_paused():
    logger.debug("EngineCore starting idle loop for wave %d.", new_wave)
    self.engines_running = True

Why this should be safe:

1. Normal unpaused behavior is unchanged.
2. During pause, wave index (current_wave) is still updated, but loop is not re-armed.
3. On resume, resume_scheduler() already sends start_wave when unfinished requests exist.

Note: this fix targets pause/collective race. Request-admission semantics for late requests during pause are a separate concern.

How to reproduce

1. Start DPEP MoE serving with data_parallel_size=2 and internal DP LB (DPLBAsyncMPClient).
2. Keep inference traffic running.
3. Trigger pause_generation(mode="abort") near wave boundary / in-flight transitions.
4. Immediately trigger all-worker utility collective_rpc (e.g., online weight receive/update).
5. Intermittently see NCCL ALLREDUCE timeout (Timeout(ms)=600000).

Race is timing dependent: it requires a late coordinator START_DP_WAVE arriving after pause has returned but before utility collective is issued.

Relevant logs

[rank7]:[E ProcessGroupNCCL.cpp:N] [Rank 7] Watchdog caught collective operation timeout:
WorkNCCL(SeqNum=N, OpType=ALLREDUCE, ..., Timeout(ms)=600000) ran for 600025 milliseconds before timing out.

Before submitting a new issue...

• I have searched existing issues and confirmed this is not a duplicate.
• I have verified the bug exists on the latest vLLM version.