vllm - 💡(How to fix) Fix [Bug]: RCCL RDNA3 gfx1100 Tp2 ROCM at startup [3 comments, 2 participants]

vllm2026-03-30 21:04:51

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

vllm-project/vllm#38587•Fetched 2026-04-08 01:53:15

View on GitHub

Comments

Participants

Timeline

Reactions

Author

JartX

Participants

github-actions[bot]

JartX

Timeline (top)

subscribed ×6mentioned ×4commented ×3cross-referenced ×2

Error Message

(Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] WorkerProc failed to start. (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] Traceback (most recent call last): (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 826, in worker_main (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] worker = WorkerProc(*args, **kwargs) (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] return func(*args, **kwargs) (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 605, in init (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.worker.init_device() (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 312, in init_device (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.worker.init_device() # type: ignore (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] return func(*args, **kwargs) (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 263, in init_device (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] init_worker_distributed_environment( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 1043, in init_worker_distributed_environment (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ensure_model_parallel_initialized( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1748, in ensure_model_parallel_initialized (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] initialize_model_parallel( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1575, in initialize_model_parallel (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] _TP = init_model_parallel_group( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1157, in init_model_parallel_group (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] return GroupCoordinator( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 376, in init (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.device_communicator = device_comm_cls(

Root Cause

vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 4096 -tp 2 (APIServer pid=59967) INFO 03-30 16:10:38 [utils.py:299] (APIServer pid=59967) INFO 03-30 16:10:38 [utils.py:299] █ █ █▄ ▄█ (APIServer pid=59967) INFO 03-30 16:10:38 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.18.1rc1.dev344+gbf0e29b75 (APIServer pid=59967) INFO 03-30 16:10:38 [utils.py:299] █▄█▀ █ █ █ █ model meta-llama/Llama-3.1-8B-Instruct (APIServer pid=59967) INFO 03-30 16:10:38 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=59967) INFO 03-30 16:10:38 [utils.py:299] (APIServer pid=59967) INFO 03-30 16:10:38 [utils.py:233] non-default args: {'model_tag': 'meta-llama/Llama-3.1-8B-Instruct', 'model': 'meta-llama/Llama-3.1-8B-Instruct', 'max_model_len': 4096, 'tensor_parallel_size': 2} (APIServer pid=59967) WARNING 03-30 16:10:38 [envs.py:1733] Unknown vLLM environment variable detected: VLLM_USE_V1 config.json: 100%|████████████████████████████████| 855/855 [00:00<00:00, 7.80MB/s] (APIServer pid=59967) INFO 03-30 16:10:48 [model.py:549] Resolved architecture: LlamaForCausalLM (APIServer pid=59967) INFO 03-30 16:10:48 [model.py:1679] Using max model len 4096 (APIServer pid=59967) INFO 03-30 16:10:48 [vllm.py:786] Asynchronous scheduling is enabled. tokenizer_config.json: 55.4kB [00:00, 101MB/s] tokenizer.json: 9.09MB [00:00, 20.3MB/s] special_tokens_map.json: 100%|████████████████████| 296/296 [00:00<00:00, 3.32MB/s] generation_config.json: 100%|█████████████████████| 184/184 [00:00<00:00, 2.14MB/s] (EngineCore pid=60177) INFO 03-30 16:10:59 [core.py:105] Initializing a V1 LLM engine (v0.18.1rc1.dev344+gbf0e29b75) with config: model='meta-llama/Llama-3.1-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Llama-3.1-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=meta-llama/Llama-3.1-8B-Instruct, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (EngineCore pid=60177) WARNING 03-30 16:10:59 [multiproc_executor.py:1014] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. (EngineCore pid=60177) INFO 03-30 16:10:59 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.19.0.2 (local), world_size=2, local_world_size=2 (Worker pid=60280) INFO 03-30 16:11:04 [parallel_state.py:1400] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:46155 backend=nccl (Worker pid=60281) INFO 03-30 16:11:05 [parallel_state.py:1400] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:46155 backend=nccl (Worker pid=60280) INFO 03-30 16:11:05 [pynccl.py:111] vLLM is using nccl==2.27.7 (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] WorkerProc failed to start. (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] Traceback (most recent call last): (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 826, in worker_main (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] worker = WorkerProc(*args, **kwargs) (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] return func(*args, **kwargs) (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 605, in __init__ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.worker.init_device() (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 312, in init_device (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.worker.init_device() # type: ignore (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] return func(*args, **kwargs) (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 263, in init_device (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] init_worker_distributed_environment( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 1043, in init_worker_distributed_environment (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ensure_model_parallel_initialized( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1748, in ensure_model_parallel_initialized (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] initialize_model_parallel( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1575, in initialize_model_parallel (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] _TP = init_model_parallel_group( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1157, in init_model_parallel_group (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] return GroupCoordinator( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 376, in __init__ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.device_communicator = device_comm_cls( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 75, in __init__ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.pynccl_comm = PyNcclCommunicator( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 144, in __init__ (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.all_reduce(data) (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 177, in all_reduce (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.nccl.ncclAllReduce( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 429, in ncclAllReduce (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] self.NCCL_CHECK( (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 373, in NCCL_CHECK (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] raise RuntimeError(f"NCCL error: {error_str}") (Worker pid=60281) ERROR 03-30 16:11:06 [multiproc_executor.py:857] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details) 508e2035e749:60281:60281 [1] [FATAL ERROR]: HIP failure: 'the operation cannot be performed in the present state' (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] EngineCore failed to start. (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] Traceback (most recent call last): (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in __init__ (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] super().__init__( (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 114, in __init__ (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] self.model_executor = executor_class(vllm_config) (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 101, in __init__ (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] super().__init__(vllm_config) (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 103, in __init__ (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] self._init_executor() (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 190, in _init_executor (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] self.workers = WorkerProc.wait_for_ready(unready_workers) (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 736, in wait_for_ready (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] raise e from None (EngineCore pid=60177) ERROR 03-30 16:11:14 [core.py:1108] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. (EngineCore pid=60177) Process EngineCore: (EngineCore pid=60177) Traceback (most recent call last): (EngineCore pid=60177) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap (EngineCore pid=60177) self.run() (EngineCore pid=60177) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run (EngineCore pid=60177) self._target(*self._args, **self._kwargs) (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1112, in run_engine_core (EngineCore pid=60177) raise e (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core (EngineCore pid=60177) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=60177) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=60177) return func(*args, **kwargs) (EngineCore pid=60177) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in __init__ (EngineCore pid=60177) super().__init__( (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 114, in __init__ (EngineCore pid=60177) self.model_executor = executor_class(vllm_config) (EngineCore pid=60177) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 101, in __init__ (EngineCore pid=60177) super().__init__(vllm_config) (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=60177) return func(*args, **kwargs) (EngineCore pid=60177) ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 103, in __init__ (EngineCore pid=60177) self._init_executor() (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 190, in _init_executor (EngineCore pid=60177) self.workers = WorkerProc.wait_for_ready(unready_workers) (EngineCore pid=60177) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=60177) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 736, in wait_for_ready (EngineCore pid=60177) raise e from None (EngineCore pid=60177) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. (APIServer pid=59967) Traceback (most recent call last): (APIServer pid=59967) File "/usr/local/bin/vllm", line 10, in <module> (APIServer pid=59967) sys.exit(main()) (APIServer pid=59967) ^^^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main (APIServer pid=59967) args.dispatch_function(args) (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd (APIServer pid=59967) uvloop.run(run_server(args)) (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run (APIServer pid=59967) return __asyncio.run( (APIServer pid=59967) ^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run (APIServer pid=59967) return runner.run(main) (APIServer pid=59967) ^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run (APIServer pid=59967) return self._loop.run_until_complete(task) (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper (APIServer pid=59967) return await main (APIServer pid=59967) ^^^^^^^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 670, in run_server (APIServer pid=59967) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 684, in run_server_worker (APIServer pid=59967) async with build_async_engine_client( (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ (APIServer pid=59967) return await anext(self.gen) (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client (APIServer pid=59967) async with build_async_engine_client_from_engine_args( (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ (APIServer pid=59967) return await anext(self.gen) (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args (APIServer pid=59967) async_llm = AsyncLLM.from_vllm_config( (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config (APIServer pid=59967) return cls( (APIServer pid=59967) ^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 154, in __init__ (APIServer pid=59967) self.engine_core = EngineCoreClient.make_async_mp_client( (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=59967) return func(*args, **kwargs) (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client (APIServer pid=59967) return AsyncMPClient(*client_args) (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=59967) return func(*args, **kwargs) (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 887, in __init__ (APIServer pid=59967) super().__init__( (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 535, in __init__ (APIServer pid=59967) with launch_core_engines( (APIServer pid=59967) ^^^^^^^^^^^^^^^^^^^^ (APIServer pid=59967) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ (APIServer pid=59967) next(self.gen) (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 998, in launch_core_engines (APIServer pid=59967) wait_for_engine_startup( (APIServer pid=59967) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1057, in wait_for_engine_startup (APIServer pid=59967) raise RuntimeError( (APIServer pid=59967) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} root@508e2035e749:/app#

Fix Action

Fix / Workaround

RAW_BUFFERClick to expand / collapse

Your current environment

`==============================` System Info

OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : 22.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-7.2.1 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e) CMake version : version 3.31.10 Libc version : glibc-2.35

============================== PyTorch Info

PyTorch version : 2.10.0+git8514f05 Is debug build : False CUDA used to build PyTorch : N/A ROCM used to build PyTorch : 7.2.53211

============================== Python Environment

Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.12.67-x86_64-with-glibc2.35

============================== CUDA / GPU Info

Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : (gfx1100) Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : 7.2.53211 MIOpen runtime version : 3.5.1 Is XNNPACK available : True

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation Model name: 13th Gen Intel(R) Core(TM) i9-13900K BIOS Model name: 13th Gen Intel(R) Core(TM) i9-13900K CPU family: 6 Model: 183 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 1 Stepping: 1 CPU max MHz: 5800.0000 CPU min MHz: 800.0000 BogoMIPS: 5990.40 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 896 KiB (24 instances) L1i cache: 1.3 MiB (24 instances) L2 cache: 32 MiB (12 instances) L3 cache: 36 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-31 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Vulnerable Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers Vulnerability Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable; BHI: Vulnerable Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Vulnerable

============================== Versions of relevant libraries

[pip3] conch-triton-kernels==1.2.1 [pip3] numpy==2.1.3 [pip3] onnx==1.19.0 [pip3] onnx-ir==0.2.0 [pip3] onnxscript==0.6.2 [pip3] onnxslim==0.1.90 [pip3] pyzmq==27.1.0 [pip3] torch==2.10.0+git8514f05 [pip3] torchaudio==2.9.0+eaa9e4e [pip3] torchvision==0.24.1+d801a34 [pip3] transformers==4.57.6 [pip3] triton==3.6.0 [pip3] triton_kernels==1.0.0 [conda] Could not collect

============================== vLLM Info

ROCM Version : 7.2.53211-06e7d5b vLLM Version : 0.18.1rc1.dev344+gbf0e29b75 (git sha: bf0e29b75) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: ============================ ROCm System Management Interface ============================ ================================ Weight between two GPUs ================================= GPU0 GPU1
GPU0 0 40
GPU1 40 0

================================= Hops between two GPUs ================================== GPU0 GPU1
GPU0 0 2
GPU1 2 0

=============================== Link Type between two GPUs =============================== GPU0 GPU1
GPU0 0 PCIE
GPU1 PCIE 0

======================================= Numa Nodes ======================================= GPU[0] : (Topology) Numa Node: 0 GPU[0] : (Topology) Numa Affinity: -1 GPU[1] : (Topology) Numa Node: 0 GPU[1] : (Topology) Numa Affinity: -1 ================================== End of ROCm SMI Log ===================================

============================== Environment Variables

TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 VLLM_ENABLE_V1_MULTIPROCESSING=1 PYTORCH_ROCM_ARCH=gfx1100 LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib: PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

root@508e2035e749:/app#

🐛 Describe the bug

RCCL 2.27.7 build 81 FAIL RCCL 2.27.7 build 38 works

If I use the one from rocm 7.1.1 on the 7.2.1 build, it works

https://github.com/ROCm/ROCm/issues/6074 Same bug

RUN wget -q " https://repo.radeon.com/rocm/apt/7.1.1/pool/main/r/rccl/rccl_2.27.7.70101-38~22.04_amd64.deb " -O /tmp/rccl.deb
&& mkdir /tmp/rccl_fix && dpkg -x /tmp/rccl.deb /tmp/rccl_fix
&& rm -f /opt/rocm/lib/librccl.so.1.0.70201 /opt/rocm/lib/librccl.so.1 /opt/rocm/lib/librccl.so
&& cp /tmp/rccl_fix/opt/rocm-7.1.1/lib/librccl.so.1.0.70101 /opt/rocm/lib/
&& ln -s librccl.so.1.0.70101 /opt/rocm/lib/librccl.so.1
&& ln -s librccl.so.1 /opt/rocm/lib/librccl.so
&& ln -s librocm_smi64.so.1 /opt/rocm/lib/librocm_smi64.so.7 2>/dev/null || true
&& ldconfig
&& rm -rf /tmp/rccl.deb /tmp/rccl_fix

To see if downgrading rccl works wget -q " https://repo.radeon.com/rocm/apt/7.1.1/pool/main/r/rccl/rccl_2.27.7.70101-38~22.04_amd64.deb " -O /tmp/rccl.deb && sudo mkdir -p /tmp/rccl_fix && sudo dpkg -x /tmp/rccl.deb /tmp/rccl_fix && sudo rm -f /opt/rocm/lib/librccl.so.1.0.70201 /opt/rocm/lib/librccl.so.1 /opt/rocm/lib/librccl.so && sudo cp /tmp/rccl_fix/opt/rocm-7.1.1/lib/librccl.so.1.0.70101 /opt/rocm/lib/ && sudo ln -sf librccl.so.1.0.70101 /opt/rocm/lib/librccl.so.1 && sudo ln -sf librccl.so.1 /opt/rocm/lib/librccl.so && sudo ln -sf librocm_smi64.so.1 /opt/rocm/lib/librocm_smi64.so.7 2>/dev/null || true && sudo ldconfig && sudo rm -rf /tmp/rccl.deb /tmp/rccl_fix

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

The issue seems to be related to the RCCL (Radeon Open Compute Common Layer) version. Downgrading RCCL to version 2.27.7 build 38 from version 2.27.7 build 81 appears to resolve the issue.

To fix the issue, you can try downgrading RCCL using the following steps:

Download the RCCL deb package for version 2.27.7 build 38:

wget -q "https://repo.radeon.com/rocm/apt/7.1.1/pool/main/r/rccl/rccl_2.27.7.70101-38~22.04_amd64.deb" -O /tmp/rccl.deb

Extract the deb package:

mkdir /tmp/rccl_fix && dpkg -x /tmp/rccl.deb /tmp/rccl_fix

Remove the existing RCCL libraries:

rm -f /opt/rocm/lib/librccl.so.1.0.70201 /opt/rocm/lib/librccl.so.1 /opt/rocm/lib/librccl.so

Copy the downgraded RCCL library:

cp /tmp/rccl_fix/opt/rocm-7.1.1/lib/librccl.so.1.0.70101 /opt/rocm/lib/

Create symbolic links for the downgraded RCCL library:

ln -s librccl.so.1.0.70101 /opt/rocm/lib/librccl.so.1
ln -s librccl.so.1 /opt/rocm/lib/librccl.so

Update the dynamic linker runtime bindings:

ldconfig

Remove the temporary files:

rm -rf /tmp/rccl.deb /tmp/rccl_fix

Verification

After downgrading RCCL, you can verify that the issue is resolved by running the vllm serve command again:

vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 4096 -tp 2

If the issue is resolved, the command should run without errors.

Extra Tips

Make sure to check the RCCL version after downgrading to ensure that the correct version is being used:

ldconfig -p | grep librccl

This should display the downgraded RCCL version. If the issue persists, you may need to further investigate the compatibility of the downgraded RCCL version with your system configuration.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #generation error #database connection #vector store #environment variable

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - 💡(How to fix) Fix [Bug]: RCCL RDNA3 gfx1100 Tp2 ROCM at startup [3 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

Your current environment

`==============================` System Info

============================== PyTorch Info

============================== Python Environment

============================== CUDA / GPU Info

============================== CPU Info

============================== Versions of relevant libraries

============================== vLLM Info

============================== Environment Variables

🐛 Describe the bug

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

TRENDING

vllm - 💡(How to fix) Fix [Bug]: RCCL RDNA3 gfx1100 Tp2 ROCM at startup [3 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

Your current environment

============================== System Info

============================== PyTorch Info

============================== Python Environment

============================== CUDA / GPU Info

============================== CPU Info

============================== Versions of relevant libraries

============================== vLLM Info

============================== Environment Variables

🐛 Describe the bug

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING

`==============================` System Info