pytorch - 💡(How to fix) Fix Feedback about Reproducibility doc page [1 comments, 2 participants]

Fix Plan

To address the issue, we will add a section on the reproducibility of SDPBackend attention backends in the PyTorch documentation.

Steps

• Add a new section to the randomness notes page: https://docs.pytorch.org/docs/stable/notes/randomness.html
• Describe the different SDPBackend attention backends (MATH, EFFICIENT, etc.) and their effects on reproducibility
• Discuss how these backends interact with low-precision computing
• Provide examples of when to expect deterministic results

Example Code

import torch

# Set the SDPBackend to MATH for deterministic results
torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.deterministic = True

# Set the SDPBackend to EFFICIENT for faster computation
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.deterministic = False

Verification

To verify the fix, check the updated documentation for clarity and accuracy. Test the code snippets to ensure they produce the expected results.

Extra Tips

• When using low-precision computing, be aware of potential trade-offs between speed and accuracy.
• Always set the torch.backends.cudnn.deterministic flag to True when requiring deterministic results.