pytorch - 💡(How to fix) Fix DISABLED test_cudagraph_indexing_ops_scatter_reduce_mean_cuda_float32 (__main__.TestCudagraphIndexingOpsCUDA) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3447, in wrapper with policy(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2824, in exit raise RuntimeError(msg) RuntimeError: CUDA driver API confirmed a leak in main.TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_reduce_mean_cuda_float32! Caching allocator allocated memory was 2048 and is now reported as 4096 on device 0. CUDA driver allocated memory was 373489664 and is now 375586816.

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/inductor/test_cudagraph_trees.py TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_reduce_mean_cuda_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix involves addressing the CUDA memory leak in the test_cudagraph_indexing_ops_scatter_reduce_mean_cuda_float32 test.

Guidance

• Investigate the CUDA memory allocation and deallocation in the test_cudagraph_indexing_ops_scatter_reduce_mean_cuda_float32 test to identify the source of the leak.
• Run the test with the PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 and PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 environment variables to gather more information about the memory leak.
• Review the workflow logs and grep for the test name to study the logs and identify any patterns or clues that may indicate the cause of the memory leak.
• Consider modifying the test to properly release CUDA resources or to use a more efficient memory allocation strategy.

Example

No specific code snippet can be provided without more information about the test implementation, but the test file path inductor/test_cudagraph_trees.py can be used as a starting point for investigation.

Notes

The provided error message indicates a CUDA memory leak, but the root cause of the leak is not immediately clear. Further investigation and debugging are necessary to identify the source of the leak and develop a fix.

Recommendation

Apply a workaround by modifying the test to properly release CUDA resources or to use a more efficient memory allocation strategy, as the root cause of the leak is not immediately clear and may require further investigation.