pytorch - 💡(How to fix) Fix DISABLED test_cudagraph_indexing_ops_scatter_add_cuda_float32 (__main__.TestCudagraphIndexingOpsCUDA) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3443, in wrapper with policy(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2820, in exit raise RuntimeError(msg) RuntimeError: CUDA driver API confirmed a leak in main.TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_add_cuda_float32! Caching allocator allocated memory was 2048 and is now reported as 4096 on device 0. CUDA driver allocated memory was 373489664 and is now 375586816.

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/inductor/test_cudagraph_trees.py TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_add_cuda_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix for the failing test is to investigate and address the CUDA memory leak reported in the test error message.

Guidance

• Investigate the CUDA memory leak by analyzing the workflow logs and grep for test_cudagraph_indexing_ops_scatter_add_cuda_float32 to identify patterns or clues that may indicate the cause of the leak.
• Run the test with the provided command PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/inductor/test_cudagraph_trees.py TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_add_cuda_float32 to reproduce the error and gather more information.
• Review the test file inductor/test_cudagraph_trees.py to ensure that CUDA resources are properly released and that there are no obvious memory leaks in the test code.
• Consider setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 to suppress the error message and simplify the debugging process.

Notes

The provided error message suggests a CUDA memory leak, but the root cause is unclear. Further investigation is needed to determine the source of the leak and develop a fix.

Recommendation

Apply workaround: Investigate and address the CUDA memory leak, as it is the most likely cause of the test failure. This will require careful analysis of the test code and workflow logs to identify the source of the leak.