vllm - 💡(How to fix) Fix [RFC]: Async parallel startup for EngineCore processes in DP/TP scenarios [2 comments, 2 participants]

Motivation.

In multi-GPU deployments using Data Parallelism (DP) or Tensor Parallelism (TP), vLLM starts subprocess workers sequentially. Each proc.start() / WorkerProc.make_worker_process() call takes approximately 12 seconds on H100 (NCCL init + device setup), so total startup time grows linearly with the number of processes:

This O(N) startup latency is a significant barrier for large-scale production deployments, especially in elastic scaling scenarios where new DP shards need to come online quickly.

Correctness bug on non-CUDA platforms. The current implementation temporarily patches CUDA_VISIBLE_DEVICES (or the platform equivalent) in the parent process via patch.dict(os.environ, ...) before calling proc.start(). When multiple EngineCore processes are started concurrently (or even sequentially with tight timing), this creates a race condition: a child process may inherit the environment variable intended for a different child. This affects Ascend NPU (ASCEND_RT_VISIBLE_DEVICES), ROCm (HIP_VISIBLE_DEVICES), and Ray-launched deployments.

Proposed Change.

We propose two independent async startup optimizations, each targeting a different parallelism dimension, plus a correctness fix for the environment variable race condition.

1. EngineCore parallel startup (DP dimension)

File: vllm/v1/engine/utils.py

In CoreEngineProcManager.__init__, we replace the sequential for proc in processes: proc.start() loop with concurrent startup via asyncio.gather + asyncio.to_thread when local_engine_count > 1.

Key design decisions:

• Each proc.start() is wrapped in a _start_with_numa closure and offloaded to a thread-pool thread via asyncio.to_thread, so that per-process NUMA binding context managers (numa_utils.configure_subprocess) can be applied on the same thread without blocking the event loop.
• When the caller is already inside a running event loop (e.g., AsyncLLM), we spin up a dedicated background thread with its own event loop via concurrent.futures.ThreadPoolExecutor, avoiding nested asyncio.run() calls.
• The serial path is preserved for single-engine deployments (local_engine_count == 1) where async overhead is unnecessary.

New functions:

2. Worker parallel startup (TP dimension)

File: vllm/v1/executor/multiproc_executor.py

In MultiprocExecutor._init_executor, we replace the sequential worker creation loop with concurrent startup via asyncio.gather + asyncio.to_thread when all of the following conditions are met:

• Multiprocessing context is spawn (not fork)
• Platform is not CPU
• local_world_size > 1

Why these constraints:

New functions:

3. Race condition fix (_enginecore_bootstrap)

We replace the parent-side patch.dict(os.environ, ...) + set_device_control_env_var() context manager with a child-side bootstrap shim _enginecore_bootstrap that:

1. Receives the environment variable name (evar) and value (value) as arguments, pre-computed in the parent before any concurrency begins
2. Sets os.environ[evar] = value inside the child process before invoking the real engine entry point
3. Eliminates the race condition entirely because each child sets its own environment independently

def _enginecore_bootstrap(
    *,
    evar: str,
    value: str,
    target_fn: Callable[..., Any],
    target_kwargs: dict[str, Any],
) -> None:
    os.environ[evar] = value      # set inside child, not in parent — no race
    target_fn(**target_kwargs)

This shim is only used when need_env_control is True (non-CUDA platforms or Ray launcher). CUDA platforms without Ray use torch.cuda.set_device() inside the worker and are completely unaffected.

data_parallel = vllm_config.parallel_config.data_parallel_size > 1
need_env_control = data_parallel and (
    not current_platform.is_cuda_alike()
    or vllm_config.parallel_config.use_ray
)

Architecture diagram

                        ┌─────────────────────────────────────────┐
                        │         AsyncLLM / LLMEngine            │
                        └──────────────┬──────────────────────────┘
                                       │
                        ┌──────────────▼──────────────────────────┐
                        │     CoreEngineProcManager.__init__()     │
                        │                                          │
                        │  local_engine_count > 1?                 │
                        │  ├─ YES → _run_async_startup()           │
                        │  │         └─ _start_processes_async()   │
                        │  │            └─ asyncio.gather(          │
                        │  │                 proc0.start(),         │
                        │  │                 proc1.start(), ...)    │
                        │  └─ NO  → serial proc.start()            │
                        └──────────────┬──────────────────────────┘
                                       │ (each EngineCore process)
                        ┌──────────────▼──────────────────────────┐
                        │   MultiprocExecutor._init_executor()     │
                        │                                          │
                        │  spawn + GPU/NPU + local_world_size > 1? │
                        │  ├─ YES → _run_async_workers_startup()   │
                        │  │         └─ _start_workers_async()     │
                        │  │            └─ asyncio.gather(          │
                        │  │                 worker0, worker1, ...) │
                        │  └─ NO  → serial make_worker_process()   │
                        └─────────────────────────────────────────┘

Feedback Period.

2 weeks

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.