pytorch - 💡(How to fix) Fix DISABLED test_memory_snapshot (__main__.TestCudaAllocator) [1 comments, 1 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/test_cuda.py", line 4233, in test_memory_snapshot torch.cuda.memory._save_segment_usage(f.name) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/memory.py", line 1166, in _save_segment_usage f.write(_segments(snapshot)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/_memory_viz.py", line 161, in segments return format_flamegraph(f.getvalue()) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/_memory_viz.py", line 100, in format_flamegraph with tempfile.NamedTemporaryFile(mode="wb", suffix=".pl") as f: File "/opt/conda/envs/py_3.10/lib/python3.10/tempfile.py", line 518, in exit self.close() File "/opt/conda/envs/py_3.10/lib/python3.10/tempfile.py", line 525, in close self._closer.close() File "/opt/conda/envs/py_3.10/lib/python3.10/tempfile.py", line 462, in close unlink(self.name) FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpaqz3rvfr.pl'

TL;DR

The most likely fix involves addressing the FileNotFoundError exception when trying to close a temporary file in the tempfile module.

Guidance

• Investigate the tempfile module usage in the torch/cuda/_memory_viz.py file to ensure proper handling of temporary files.
• Verify that the /tmp directory is writable and not being cleaned up prematurely, causing the file to be deleted before it can be closed.
• Check the test_memory_snapshot function in test_cuda.py to see if there are any issues with the way temporary files are being used or closed.
• Review the CI workflow logs to ensure that the test environment is properly configured and that there are no issues with the test runner or the Python environment.

Example

No specific code snippet can be provided without modifying the existing codebase, but ensuring that temporary files are properly closed and that the /tmp directory is not being cleaned up prematurely can be done by adding error handling and logging to the tempfile usage.

Notes

The issue seems to be related to the usage of temporary files in the tempfile module, and addressing this issue may require changes to the torch/cuda/_memory_viz.py file or the test_cuda.py file.

Recommendation

Apply a workaround to ensure that temporary files are properly handled and closed, such as adding try-except blocks to handle the FileNotFoundError exception or modifying the tempfile usage to avoid deleting files prematurely.