pytorch - 💡(How to fix) Fix DISABLED test_export_with_parameters_cuda (__main__.ExportTestsDeviceCUDA) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3514, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3514, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3513, in wrapper with policy(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2887, in exit raise RuntimeError(msg) RuntimeError: CUDA driver API confirmed a leak in main.ExportTestsDeviceCUDA.test_export_with_parameters_cuda! Caching allocator allocated memory was 0 and is now reported as 7680 on device 0. CUDA driver allocated memory was 293797888 and is now 295895040.

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/dynamo/test_export.py ExportTestsDeviceCUDA.test_export_with_parameters_cuda

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_export_with_parameters_cuda is failing due to a CUDA memory leak, and the issue can be investigated by analyzing the workflow logs and running the test with specific environment variables.

Guidance

• To debug the issue, follow the provided debugging instructions to find relevant log snippets in the workflow logs.
• Run the test with the environment variables PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 and PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 to reproduce the error and gather more information.
• Investigate the CUDA memory allocation and deallocation in the test to identify the source of the leak.
• Check the test file dynamo/test_export.py and the ExportTestsDeviceCUDA class to understand the test's behavior and potential memory management issues.

Example

No code snippet is provided as the issue is related to a specific test and environment, and modifying the code without further investigation may not be effective.

Notes

The issue is specific to the Linux platform and the CUDA device, so any solution or workaround may need to be tailored to this environment. The test has been disabled due to its flaky nature, and resolving the memory leak issue may require careful analysis of the test's behavior and the CUDA driver's memory management.

Recommendation

Apply workaround: Run the test with the specified environment variables to gather more information and investigate the memory leak issue, as this will provide more insight into the problem without modifying the code or environment.