pytorch - 💡(How to fix) Fix DISABLED test_cudagraph_indexing_ops_scatter_cuda_float32 (__main__.TestCudagraphIndexingOpsCUDA) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3443, in wrapper with policy(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2820, in exit raise RuntimeError(msg) RuntimeError: CUDA driver API confirmed a leak in main.TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_cuda_float32! Caching allocator allocated memory was 2048 and is now reported as 4096 on device 0. CUDA driver allocated memory was 373489664 and is now 375586816.

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/inductor/test_cudagraph_trees.py TestCudagraphIndexingOpsCUDA.test_cudagraph_indexing_ops_scatter_cuda_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix involves addressing the CUDA memory leak in the test_cudagraph_indexing_ops_scatter_cuda_float32 test.

Guidance

• Investigate the CUDA memory allocation and deallocation in the test_cudagraph_indexing_ops_scatter_cuda_float32 test to identify the source of the leak.
• Run the test with the PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 and PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 environment variables to enable memory leak checking and slow gradient checking.
• Study the logs from the workflow runs to understand the pattern of the memory leak and identify potential fixes.
• Consider setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 to suppress the error message and focus on debugging the underlying issue.

Example

No specific code snippet is provided, but the test file path inductor/test_cudagraph_trees.py and the test class TestCudagraphIndexingOpsCUDA can be used as a starting point for debugging.

Notes

The issue is specific to the Linux platform and is related to CUDA memory management. The provided error message and debugging instructions suggest that the issue is a memory leak, but the root cause is not immediately clear.

Recommendation

Apply a workaround by re-enabling the test and running it with the suggested environment variables to gather more information about the memory leak. This will allow for further debugging and identification of the root cause.