pytorch - 💡(How to fix) Fix DISABLED test_cudagraph_indexing_ops_masked_scatter_cuda_float32 (__main__.TestCudagraphIndexingOpsCUDA) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3443, in wrapper with policy(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2820, in exit raise RuntimeError(msg) RuntimeError: CUDA driver API confirmed a leak in main.TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_masked_scatter_cuda_float32! Caching allocator allocated memory was 2048 and is now reported as 4096 on device 0. CUDA driver allocated memory was 373489664 and is now 375586816.

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/inductor/test_cudagraph_trees.py TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_masked_scatter_cuda_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix for the flaky test test_cudagraph_indexing_ops_masked_scatter_cuda_float32 is to investigate and address the reported CUDA memory leak.

Guidance

• Review the workflow logs and grep for test_cudagraph_indexing_ops_masked_scatter_cuda_float32 to study the logs and identify patterns or clues related to the memory leak.
• Run the test with PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 and PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 to reproduce the issue and gather more information about the memory leak.
• Investigate the CUDA driver API and caching allocator to understand why the allocated memory is increasing.
• Consider setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 to suppress the error message and focus on debugging the underlying issue.

Example

No code snippet is provided as the issue is related to a specific test and CUDA memory leak, and modifying code without understanding the root cause may not be effective.

Notes

The issue is specific to the test_cudagraph_indexing_ops_masked_scatter_cuda_float32 test and may be related to the CUDA driver API or caching allocator. The provided error message and debugging instructions should be used to investigate and address the memory leak.

Recommendation

Apply workaround: Investigate and address the reported CUDA memory leak by following the provided debugging instructions and reviewing the workflow logs. This approach allows for a targeted investigation and potential fix for the flaky test.