pytorch - 💡(How to fix) Fix DISABLED test_cudagraph_indexing_ops_as_strided_scatter_cuda_float32 (__main__.TestCudagraphIndexingOpsCUDA) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3443, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3443, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3442, in wrapper with policy(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2819, in exit raise RuntimeError(msg) RuntimeError: CUDA driver API confirmed a leak in main.TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_as_strided_scatter_cuda_float32! Caching allocator allocated memory was 0 and is now reported as 1024 on device 0. CUDA driver allocated memory was 281018368 and is now 285212672.

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/inductor/test_cudagraph_trees.py TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_as_strided_scatter_cuda_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix for the flaky test is to investigate and address the CUDA memory leak reported in the test failure.

Guidance

• Investigate the CUDA memory leak by analyzing the workflow logs and grep for test_cudagraph_indexing_ops_as_strided_scatter_cuda_float32 to identify patterns or clues that may indicate the cause of the leak.
• Run the test with PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 to reproduce the failure and gather more information about the leak.
• Review the test code in inductor/test_cudagraph_trees.py to ensure proper memory management and deallocation of CUDA resources.
• Consider setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 to suppress the error message and focus on debugging the underlying issue.

Example

No code snippet is provided as the issue does not contain sufficient information to create a specific example.

Notes

The test failure is intermittent, and the CUDA memory leak may be caused by various factors, including incorrect memory management or issues with the CUDA driver.

Recommendation

Apply workaround: Investigate and address the CUDA memory leak to stabilize the test. This approach is recommended as it targets the root cause of the issue, rather than just suppressing the error message.