pytorch - 💡(How to fix) Fix `torch.compile` + `torch.distributed` communication can cause random segmentation fault at process exit [3 comments, 3 participants]

SIGSEGV(11), PID: 3163834, Thread 3163834: frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8f (0x7fa99c6d5a9f in /opt/conda/envs/torch-base/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: <unknown function> + 0x12990 (0x7fa9e0cf1990 in /usr/lib64/libpthread.so.0) frame #2: waitpid + 0x52 (0x7fa9e0cf1312 in /usr/lib64/libpthread.so.0) frame #3: /opt/conda/envs/torch-base/bin/python3() [0x6455e7] frame #4: _PyEval_EvalFrameDefault + 0x15ed (0x521f7d in /opt/conda/envs/torch-base/bin/python3) frame #5: /opt/conda/envs/torch-base/bin/python3() [0x617f90] frame #6: Py_FinalizeEx + 0x77 (0x5fed57 in /opt/conda/envs/torch-base/bin/python3) frame #7: Py_Exit + 0x8 (0x619728 in /opt/conda/envs/torch-base/bin/python3) frame #8: /opt/conda/envs/torch-base/bin/python3() [0x616eeb] frame #9: /opt/conda/envs/torch-base/bin/python3() [0x616b41] frame #10: PyRun_SimpleStringFlags + 0x5c (0x60fc7c in /opt/conda/envs/torch-base/bin/python3) frame #11: Py_RunMain + 0x4e1 (0x60d5f1 in /opt/conda/envs/torch-base/bin/python3) frame #12: Py_BytesMain + 0x39 (0x5c4809 in /opt/conda/envs/torch-base/bin/python3) frame #13: __libc_start_main + 0xe5 (0x7fa9e01d77e5 in /usr/lib64/libc.so.6) frame #14: /opt/conda/envs/torch-base/bin/python3() [0x5c4635]

SIGSEGV(11), PID: 3163834, Thread 3164280: frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8f (0x7fa99c6d5a9f in /opt/conda/envs/torch-base/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: <unknown function> + 0x12990 (0x7fa9e0cf1990 in /usr/lib64/libpthread.so.0) frame #2: pthread_cond_wait + 0x161 (0x7fa9e0ced371 in /usr/lib64/libpthread.so.0) frame #3: <unknown function> + 0x3506fb (0x7fa8ee2f66fb in /opt/conda/envs/torch-base/lib/python3.12/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so) frame #4: <unknown function> + 0x81ca (0x7fa9e0ce71ca in /usr/lib64/libpthread.so.0) frame #5: clone + 0x43 (0x7fa9e01d68d3 in /usr/lib64/libc.so.6)

TL;DR

The most likely fix for the intermittent segmentation fault when using torch.compile(create_block_mask) with distributed communication is to avoid using torch.compile with functions that are not designed to be compiled, or to update PyTorch to a version where this issue is fixed, if available.

Guidance

• Verify that the segmentation fault is reproducible with the provided minimal repro code to ensure it's not an environment-specific issue.
• Check the PyTorch documentation for any known issues or limitations with using torch.compile with create_block_mask and distributed communication.
• Consider removing the torch.compile call for the create_block_mask function to see if the issue persists, as this could be a workaround if the compilation is not necessary for the specific use case.
• Review the stacktrace to identify any other potential issues or libraries that might be contributing to the segmentation fault, such as compatibility problems between different library versions.

Example

No specific code example is provided as the issue seems to be related to the interaction between PyTorch's compilation and distributed communication features, and any changes would require a deeper understanding of the specific use case and requirements.

Notes

The provided information suggests a potential issue with the interaction between torch.compile and distributed communication in PyTorch. However, without more context or details about the specific requirements and constraints of the project, it's challenging to provide a definitive solution. The suggestions provided are based on the information given and aim to help troubleshoot and potentially mitigate the issue.

Recommendation

Apply workaround: Remove the torch.compile call for the create_block_mask function to see if the issue persists, as this could be a temporary solution until a more permanent fix is available or the root cause is fully understood.