pytorch - 💡(How to fix) Fix DISABLED test_cudagraph_indexing_ops_scatter_reduce_amax_cuda_float32 (__main__.TestCudagraphIndexingOpsCUDA) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3443, in wrapper with policy(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2820, in exit raise RuntimeError(msg) RuntimeError: CUDA driver API confirmed a leak in main.TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_reduce_amax_cuda_float32! Caching allocator allocated memory was 2048 and is now reported as 4096 on device 0. CUDA driver allocated memory was 373489664 and is now 375586816.

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/inductor/test_cudagraph_trees.py TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_reduce_amax_cuda_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix for the failing test is to investigate and address the reported CUDA memory leak in the test_cudagraph_indexing_ops_scatter_reduce_amax_cuda_float32 test.

Guidance

• Investigate the CUDA memory leak by running the test with PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 and PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 to gather more information about the leak.
• Review the test code in inductor/test_cudagraph_trees.py to identify potential causes of the memory leak, such as unclosed CUDA resources or incorrect memory allocation.
• Use the provided debugging instructions to find relevant log snippets and study the logs to understand the failure pattern.
• Consider setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 to suppress the error message and focus on debugging the underlying issue.

Example

No code snippet is provided as the issue does not contain sufficient information to create a minimal example.

Notes

The issue is specific to the Linux platform and is related to a flaky test. The provided error message indicates a CUDA memory leak, which needs to be investigated and addressed.

Recommendation

Apply workaround: Investigate and address the CUDA memory leak to stabilize the test. This is recommended as the issue is specific to a single test and investigating the memory leak will help to identify the root cause of the failure.