vllm - 💡(How to fix) Fix [Usage]: How to do offline inference on one rank in a distributed environment? [1 participants]

vllm2026-03-26 16:43:18

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

vllm-project/vllm#38258•Fetched 2026-04-08 01:36:55

View on GitHub

Comments

Participants

Timeline

Reactions

Author

DragonAura

Participants

DragonAura

Timeline (top)

labeled ×1

Code Example

==============================
        System Info
==============================
OS                           : Debian GNU/Linux 12 (bookworm) (x86_64)
GCC version                  : (Debian 12.2.0-14+deb12u1) 12.2.0
Clang version                : 14.0.6
CMake version                : version 3.31.6
Libc version                 : glibc-2.36

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0
Is debug build               : False
CUDA used to build PyTorch   : 13.1
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] (64-bit runtime)
Python platform              : Linux-5.15.120.bsk.2-amd64-x86_64-with-glibc2.36

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.1.115
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version        : 535.161.08
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.17.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================

---

dist.init_process_group(backend="nccl")
print(dist.is_initialized())
if not dist.is_initialized() or dist.get_rank() != 0:
    pass
    # do anything else
else:
    llm = LLM(
        distributed_executor_backend="mp",

RAW_BUFFERClick to expand / collapse

Your current environment

==============================
        System Info
==============================
OS                           : Debian GNU/Linux 12 (bookworm) (x86_64)
GCC version                  : (Debian 12.2.0-14+deb12u1) 12.2.0
Clang version                : 14.0.6
CMake version                : version 3.31.6
Libc version                 : glibc-2.36

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0
Is debug build               : False
CUDA used to build PyTorch   : 13.1
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] (64-bit runtime)
Python platform              : Linux-5.15.120.bsk.2-amd64-x86_64-with-glibc2.36

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.1.115
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version        : 535.161.08
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.17.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================

How would you like to use vllm

For example, for the following code debug.py:

dist.init_process_group(backend="nccl")
print(dist.is_initialized())
if not dist.is_initialized() or dist.get_rank() != 0:
    pass
    # do anything else
else:
    llm = LLM(
        distributed_executor_backend="mp",

python3 debug.py works fine, while torchrun --nproc_per_node="8" --nnodes="1" --node_rank="0" --master_addr="127.0.0.1" --master_port=12345 debug.py hangs during initialization.

To some reason, I need a) One LLM running on rank 0 and uses all GPUs or b) Each rank runs an LLM without TP or DP, but only uses its own GPU.

It seems that both situations meet some issue...

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

To resolve the issue with torchrun hanging during initialization, we need to adjust the way we initialize the distributed process group and the LLM.

Here are the steps to achieve the desired behavior:

For a) One LLM running on rank 0 and using all GPUs:
1. Initialize the distributed process group with dist.init_process_group(backend="nccl").
2. Check if the current rank is 0, and if so, initialize the LLM with all GPUs.
3. Use torch.cuda.set_device to set the device for each process.
For b) Each rank runs an LLM without TP or DP, but only uses its own GPU:
1. Initialize the distributed process group with dist.init_process_group(backend="nccl").
2. Set the device for each process using torch.cuda.set_device.
3. Initialize the LLM for each rank.

Example code for a):

import torch
import torch.distributed as dist

dist.init_process_group(backend="nccl")
if dist.get_rank() == 0:
    # Initialize LLM with all GPUs
    llm = LLM(distributed_executor_backend="mp")
else:
    # Do not initialize LLM for other ranks
    pass

Example code for b):

import torch
import torch.distributed as dist

dist.init_process_group(backend="nccl")
device = torch.device(f"cuda:{dist.get_rank()}")
torch.cuda.set_device(device)
llm = LLM(distributed_executor_backend="mp")

Verification

To verify that the fix worked, run the script with torchrun and check that the LLM is initialized correctly for each rank. You can add print statements or use a debugger to verify the behavior.

Extra Tips

Make sure to set the CUDA_VISIBLE_DEVICES environment variable to control which GPUs are visible to each process.
Use torch.cuda.device_count() to get the number of GPUs available on each node.
Consider using torch.distributed.launch instead of torchrun for more flexibility and control over the distributed training process.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#serialization error #model compatibility #GPU setup #container setup #orchestration issue

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - 💡(How to fix) Fix [Usage]: How to do offline inference on one rank in a distributed environment? [1 participants]

Recommended Tools

GitHub issue graph ai analysis

Code Example

Your current environment

How would you like to use vllm

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

TRENDING

vllm - 💡(How to fix) Fix [Usage]: How to do offline inference on one rank in a distributed environment? [1 participants]

Recommended Tools

GitHub issue graph ai analysis

Code Example

Your current environment

How would you like to use vllm

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING