Issue Station
Sign In
Tools hereToolbox

pytorch - ✅(Solved) Fix Wrong `WARNING: destroy_process_group() was not called before program exit` [1 pull requests, 6 comments, 3 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#178758•Fetched 2026-04-08 01:52:24
View on GitHub
Comments
6
Participants
3
Timeline
46
Reactions
0
Author
FlamefireFlamefire
Participants
anshuS1310anshuS1310FlamefireFlamefireokiemute04okiemute04
Timeline (top)
mentioned ×16subscribed ×16commented ×6referenced ×4

Error Message

import os os.environ["TORCH_DISTRIBUTED_DEBUG"] = "DETAIL"

import torch import torch.distributed as dist import torch.multiprocessing as mp

WORLD_SIZE = 2

def init_distributed(port, rank, world_size, init_file): try: os.environ["MASTER_ADDR"] = "127.0.0.1" os.environ["MASTER_PORT"] = port os.environ["LOCAL_RANK"] = str(rank) os.environ["RANK"] = str(rank) os.environ["LOCAL_SIZE"] = str(world_size) os.environ["WORLD_SIZE"] = str(world_size)

    dist.init_process_group(
        backend="nccl",
        init_method=f"file://{init_file}",
        rank=rank,
        world_size=world_size,
        device_id=torch.device('cuda', rank)
    )
    dist.barrier()
    print(f"[Rank {rank}] ready")
    dist.destroy_process_group()
except Exception as e:
    print(rank, "ERROR", e)

def test_main(): mp.set_start_method("forkserver", force=True) pool = mp.Pool(processes=WORLD_SIZE) results = pool.starmap_async( init_distributed, [('51200', rank, WORLD_SIZE, '/tmp/ptinit.file') for rank in range(WORLD_SIZE)], ) results.wait() pool.close() pool.join() print("Finished")

if name == "main": test_main()

Fix Action

Fix / Workaround

CPU: Architektur: x86_64 CPU Operationsmodus: 32-bit, 64-bit Adressgrößen: 43 bits physical, 48 bits virtual Byte-Reihenfolge: Little Endian CPU(s): 96 Liste der Online-CPU(s): 0-95 Anbieterkennung: AuthenticAMD Modellname: AMD EPYC 7352 24-Core Processor Prozessorfamilie: 23 Modell: 49 Thread(s) pro Kern: 2 Kern(e) pro Sockel: 24 Sockel: 2 Stepping: 0 Übertaktung: aktiviert CPU(s) scaling MHz: 99% Maximale Taktfrequenz der CPU: 2300,0000 Minimale Taktfrequenz der CPU: 1500,0000 BogoMIPS: 4600.42 Markierungen: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es Virtualisierung: AMD-V L1d Cache: 1,5 MiB (48 Instanzen) L1i Cache: 1,5 MiB (48 Instanzen) L2 Cache: 24 MiB (48 Instanzen) L3 Cache: 256 MiB (16 Instanzen) NUMA-Knoten: 4 NUMA-Knoten0 CPU(s): 0-11,48-59 NUMA-Knoten1 CPU(s): 12-23,60-71 NUMA-Knoten2 CPU(s): 24-35,72-83 NUMA-Knoten3 CPU(s): 36-47,84-95 Schwachstelle Gather data sampling: Not affected Schwachstelle Indirect target selection: Not affected Schwachstelle Itlb multihit: Not affected Schwachstelle L1tf: Not affected Schwachstelle Mds: Not affected Schwachstelle Meltdown: Not affected Schwachstelle Mmio stale data: Not affected Schwachstelle Reg file data sampling: Not affected Schwachstelle Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Schwachstelle Spec rstack overflow: Mitigation; Safe RET Schwachstelle Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Schwachstelle Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Schwachstelle Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Schwachstelle Srbds: Not affected Schwachstelle Tsx async abort: Not affected

PR fix notes

PR #178779: Implement missing methods in ProcessGroupWrapper

Description (problem / solution / changelog)

Most importantly shutdown is missing which in the case of the NCCL process group may lead to hangs on termination.

See #178758

<details><summary>Example reproducer:</summary> 
import os
os.environ["TORCH_DISTRIBUTED_DEBUG"] = "DETAIL"

import torch
import torch.distributed as dist
import torch.multiprocessing as mp

WORLD_SIZE = 2

def init_distributed(port, rank, world_size, init_file):
    try:
        os.environ["MASTER_ADDR"] = "127.0.0.1"
        os.environ["MASTER_PORT"] = port
        os.environ["LOCAL_RANK"] = str(rank)
        os.environ["RANK"] = str(rank)
        os.environ["LOCAL_SIZE"] = str(world_size)
        os.environ["WORLD_SIZE"] = str(world_size)

        dist.init_process_group(
            backend="nccl",
            init_method=f"file://{init_file}",
            rank=rank,
            world_size=world_size,
            device_id=torch.device('cuda', rank)
        )
        dist.barrier()
        print(f"[Rank {rank}] ready")
        dist.destroy_process_group()
    except Exception as e:
        print(rank, "ERROR", e)


def test_main():
    mp.set_start_method("forkserver", force=True)
    pool = mp.Pool(processes=WORLD_SIZE)
    results = pool.starmap_async(
        init_distributed,
        [('51200', rank, WORLD_SIZE, '/tmp/ptinit.file') for rank in range(WORLD_SIZE)],
    )
    results.wait()
    pool.close()
    pool.join()
    print("Finished")


if __name__ == "__main__":
    test_main()
</details>

Note that this shows a related issue: The device passed to dist.init_process_group is not passed from ProcessGroupWrapper to the underlying process group. Hence it will try guessing and warn about it:

Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()

I don't see an obvious solution to that so left it for now.

Changed files

  • torch/csrc/distributed/c10d/ProcessGroupWrapper.cpp (modified, +78/-0)
  • torch/csrc/distributed/c10d/ProcessGroupWrapper.hpp (modified, +47/-13)

Code Example

import os
os.environ["TORCH_DISTRIBUTED_DEBUG"] = "DETAIL"

import torch
import torch.distributed as dist
import torch.multiprocessing as mp

WORLD_SIZE = 2

def init_distributed(port, rank, world_size, init_file):
    try:
        os.environ["MASTER_ADDR"] = "127.0.0.1"
        os.environ["MASTER_PORT"] = port
        os.environ["LOCAL_RANK"] = str(rank)
        os.environ["RANK"] = str(rank)
        os.environ["LOCAL_SIZE"] = str(world_size)
        os.environ["WORLD_SIZE"] = str(world_size)

        dist.init_process_group(
            backend="nccl",
            init_method=f"file://{init_file}",
            rank=rank,
            world_size=world_size,
            device_id=torch.device('cuda', rank)
        )
        dist.barrier()
        print(f"[Rank {rank}] ready")
        dist.destroy_process_group()
    except Exception as e:
        print(rank, "ERROR", e)


def test_main():
    mp.set_start_method("forkserver", force=True)
    pool = mp.Pool(processes=WORLD_SIZE)
    results = pool.starmap_async(
        init_distributed,
        [('51200', rank, WORLD_SIZE, '/tmp/ptinit.file') for rank in range(WORLD_SIZE)],
    )
    results.wait()
    pool.close()
    pool.join()
    print("Finished")


if __name__ == "__main__":
    test_main()
RAW_BUFFERClick to expand / collapse

🐛 Describe the bug

I'm getting a warning which seems wrong when using the following, heavily reduced code:

import os
os.environ["TORCH_DISTRIBUTED_DEBUG"] = "DETAIL"

import torch
import torch.distributed as dist
import torch.multiprocessing as mp

WORLD_SIZE = 2

def init_distributed(port, rank, world_size, init_file):
    try:
        os.environ["MASTER_ADDR"] = "127.0.0.1"
        os.environ["MASTER_PORT"] = port
        os.environ["LOCAL_RANK"] = str(rank)
        os.environ["RANK"] = str(rank)
        os.environ["LOCAL_SIZE"] = str(world_size)
        os.environ["WORLD_SIZE"] = str(world_size)

        dist.init_process_group(
            backend="nccl",
            init_method=f"file://{init_file}",
            rank=rank,
            world_size=world_size,
            device_id=torch.device('cuda', rank)
        )
        dist.barrier()
        print(f"[Rank {rank}] ready")
        dist.destroy_process_group()
    except Exception as e:
        print(rank, "ERROR", e)


def test_main():
    mp.set_start_method("forkserver", force=True)
    pool = mp.Pool(processes=WORLD_SIZE)
    results = pool.starmap_async(
        init_distributed,
        [('51200', rank, WORLD_SIZE, '/tmp/ptinit.file') for rank in range(WORLD_SIZE)],
    )
    results.wait()
    pool.close()
    pool.join()
    print("Finished")


if __name__ == "__main__":
    test_main()

Output:

[Rank 1] ready [Rank 0] ready [W330 11:18:35.950408493 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) Finished

Versions

PyTorch version: 2.11.0+cu126

<details><summary>Details</summary> <p> Collecting environment information... PyTorch version: 2.11.0+cu126 Is debug build: False CUDA used to build PyTorch: 12.6 ROCM used to build PyTorch: N/A

OS: Rocky Linux 9.6 (Blue Onyx) (x86_64) GCC version: (GCC) 13.3.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.34

Python version: 3.12.3 (main, Aug 28 2024, 18:10:15) [GCC 13.3.0] (64-bit runtime) Python platform: Linux-5.14.0-570.49.1.el9_6.x86_64-x86_64-with-glibc2.34 Is CUDA available: True CUDA runtime version: 12.6.20 CUDA_MODULE_LOADING set to: GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB GPU 1: NVIDIA A100-SXM4-40GB GPU 2: NVIDIA A100-SXM4-40GB GPU 3: NVIDIA A100-SXM4-40GB GPU 4: NVIDIA A100-SXM4-40GB GPU 5: NVIDIA A100-SXM4-40GB GPU 6: NVIDIA A100-SXM4-40GB GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 580.65.06 cuDNN version: Could not collect Is XPU available: False HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True Caching allocator config: N/A

CPU: Architektur: x86_64 CPU Operationsmodus: 32-bit, 64-bit Adressgrößen: 43 bits physical, 48 bits virtual Byte-Reihenfolge: Little Endian CPU(s): 96 Liste der Online-CPU(s): 0-95 Anbieterkennung: AuthenticAMD Modellname: AMD EPYC 7352 24-Core Processor Prozessorfamilie: 23 Modell: 49 Thread(s) pro Kern: 2 Kern(e) pro Sockel: 24 Sockel: 2 Stepping: 0 Übertaktung: aktiviert CPU(s) scaling MHz: 99% Maximale Taktfrequenz der CPU: 2300,0000 Minimale Taktfrequenz der CPU: 1500,0000 BogoMIPS: 4600.42 Markierungen: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es Virtualisierung: AMD-V L1d Cache: 1,5 MiB (48 Instanzen) L1i Cache: 1,5 MiB (48 Instanzen) L2 Cache: 24 MiB (48 Instanzen) L3 Cache: 256 MiB (16 Instanzen) NUMA-Knoten: 4 NUMA-Knoten0 CPU(s): 0-11,48-59 NUMA-Knoten1 CPU(s): 12-23,60-71 NUMA-Knoten2 CPU(s): 24-35,72-83 NUMA-Knoten3 CPU(s): 36-47,84-95 Schwachstelle Gather data sampling: Not affected Schwachstelle Indirect target selection: Not affected Schwachstelle Itlb multihit: Not affected Schwachstelle L1tf: Not affected Schwachstelle Mds: Not affected Schwachstelle Meltdown: Not affected Schwachstelle Mmio stale data: Not affected Schwachstelle Reg file data sampling: Not affected Schwachstelle Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Schwachstelle Spec rstack overflow: Mitigation; Safe RET Schwachstelle Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Schwachstelle Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Schwachstelle Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Schwachstelle Srbds: Not affected Schwachstelle Tsx async abort: Not affected

Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.6.4.1 [pip3] nvidia-cuda-cupti-cu12==12.6.80 [pip3] nvidia-cuda-nvrtc-cu12==12.6.85 [pip3] nvidia-cuda-runtime-cu12==12.6.77 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cufft-cu12==11.3.0.4 [pip3] nvidia-curand-cu12==10.3.7.77 [pip3] nvidia-cusolver-cu12==11.7.1.2 [pip3] nvidia-cusparse-cu12==12.5.4.2 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-nccl-cu12==2.28.9 [pip3] nvidia-nvjitlink-cu12==12.6.85 [pip3] nvidia-nvtx-cu12==12.6.77 [pip3] torch==2.11.0+cu126 [pip3] triton==3.6.0 [conda] Could not collect

</p> </details>

cc @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci @aditvenk @xmfan

extent analysis

Fix Plan

The warning is due to the process group not being properly destroyed before the program exits.

To fix this issue, we need to ensure that dist.destroy_process_group() is called after all processes have finished their work.

Here are the steps to fix the issue:

  • Modify the init_distributed function to return a value indicating whether the process group was successfully destroyed.
  • In the test_main function, use pool.apply_async instead of pool.starmap_async to get the return values from each process.
  • Check the return values to ensure that all process groups were successfully destroyed before exiting the program.

Here is an example of the modified code:

import os
os.environ["TORCH_DISTRIBUTED_DEBUG"] = "DETAIL"

import torch
import torch.distributed as dist
import torch.multiprocessing as mp

WORLD_SIZE = 2

def init_distributed(port, rank, world_size, init_file):
    try:
        os.environ["MASTER_ADDR"] = "127.0.0.1"
        os.environ["MASTER_PORT"] = port
        os.environ["LOCAL_RANK"] = str(rank)
        os.environ["RANK"] = str(rank)
        os.environ["LOCAL_SIZE"] = str(world_size)
        os.environ["WORLD_SIZE"] = str(world_size)

        dist.init_process_group(
            backend="nccl",
            init_method=f"file://{init_file}",
            rank=rank,
            world_size=world_size,
            device_id=torch.device('cuda', rank)
        )
        dist.barrier()
        print(f"[Rank {rank}] ready")
        dist.destroy_process_group()
        return True
    except Exception as e:
        print(rank, "ERROR", e)
        return False


def test_main():
    mp.set_start_method("forkserver", force=True)
    pool = mp.Pool(processes=WORLD_SIZE)
    results = []
    for rank in range(WORLD_SIZE):
        result = pool.apply_async(init_distributed, ('51200', rank, WORLD_SIZE, '/tmp/ptinit.file'))
        results.append(result)
    for result in results:
        if not result.get():
            print("Error destroying process group")
    pool.close()
    pool.join()
    print("Finished")


if __name__ == "__main__":
    test_main()

Verification

To verify that the fix worked, run the modified program and check that the warning message is no longer printed.

Extra Tips

  • Make sure to properly clean up resources after use to avoid memory leaks and other issues.
  • Use dist.destroy_process_group() to destroy the process group after all processes have finished their work.
  • Use pool.apply_async instead of pool.starmap_async to get the return values from each process.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#orchestration issue#cache issue#memory leak#API versioning

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

pytorch - ✅(Solved) Fix Wrong `WARNING: destroy_process_group() was not called before program exit` [1 pull requests, 6 comments, 3 participants]