pytorch - 💡(How to fix) Fix [cudagraphs] torch.cond does not work with cudagraphs when nccl collectives are used [8 comments, 3 participants]

TL;DR

• The issue may be resolved by modifying the cond_cg_funcol_repro.py script to properly handle cudagraphs capture with nccl collectives inside torch.cond.

Guidance

• Review the cond_cg_funcol_repro.py script to ensure correct usage of torch.cond and nccl collectives.
• Verify that cudagraphs capture is enabled and configured correctly.
• Check the PyTorch documentation for any known limitations or issues with using nccl collectives inside torch.cond.
• Test the script with a different PyTorch version or configuration to isolate the issue.

Example

• No code snippet can be provided without modifying the original script, but reviewing the repro script at https://gist.github.com/oulgen/1ea85a9ec628d92dce04b787a80bc24a may help identify the issue.

Notes

• The issue may be specific to the master branch or the current PyTorch version.
• Without more information, it's difficult to provide a definitive solution.

Recommendation

• Apply workaround: Modify the cond_cg_funcol_repro.py script to handle cudagraphs capture correctly, as the issue may be related to the script's implementation rather than a PyTorch version issue.