pytorch - 💡(How to fix) Fix DISABLED test_cudagraph_indexing_ops_scatter_reduce_amin_cuda_float32 (__main__.TestCudagraphIndexingOpsCUDA) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3448, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3447, in wrapper with policy(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2824, in exit raise RuntimeError(msg) RuntimeError: CUDA driver API confirmed a leak in main.TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_reduce_amin_cuda_float32! Caching allocator allocated memory was 2048 and is now reported as 4096 on device 0. CUDA driver allocated memory was 373489664 and is now 375586816.

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/inductor/test_cudagraph_trees.py TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_reduce_amin_cuda_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix for the flaky test is to investigate and address the CUDA memory leak reported in the test failure.

Guidance

• Review the workflow logs linked in the issue to identify patterns or commonalities among the failed test runs.
• Use the provided debugging instructions to grep for the specific test name and analyze the logs for clues about the memory leak.
• Run the test locally with the suggested environment variables (PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 and PYTORCH_TEST_WITH_SLOW_GRADCHECK=1) to reproduce and debug the issue.
• Investigate potential causes of the CUDA memory leak, such as incorrect tensor allocation or deallocation in the test_cudagraph_indexing_ops_scatter_reduce_amin_cuda_float32 test.

Example

No specific code snippet is provided, but the test file path inductor/test_cudagraph_trees.py and the test name test_cudagraph_indexing_ops_scatter_reduce_amin_cuda_float32 can be used to locate and inspect the relevant code.

Notes

The issue is specific to the Linux platform and is related to a flaky test, which may indicate intermittent or environment-dependent behavior. The provided error message suggests a CUDA memory leak, but the root cause is unclear and requires further investigation.

Recommendation

Apply workaround: Investigate and address the CUDA memory leak, and consider re-enabling the test with additional logging or debugging measures to monitor its stability.