pytorch - 💡(How to fix) Fix DISABLED test_max_split_expandable (__main__.TestCudaAllocator) [1 comments, 1 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/test_cuda.py", line 4832, in test_max_split_expandable with self.assertRaises(torch.OutOfMemoryError): File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 226, in exit self._raiseFailure("{} not raised".format(exc_name)) File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure raise self.test_case.failureException(msg) AssertionError: OutOfMemoryError not raised

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/test_cuda.py TestCudaAllocator.test_max_split_expandable

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_max_split_expandable is failing intermittently, and the likely fix involves investigating memory allocation issues related to CUDA.

Guidance

• Investigate the workflow logs for patterns or commonalities among the 34 failures to understand the conditions under which the test fails.
• Run the test locally with the provided command PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/test_cuda.py TestCudaAllocator.test_max_split_expandable to reproduce the issue and gather more detailed logs.
• Check for potential memory leaks or allocation issues in the CUDA code, as the test is expecting an OutOfMemoryError but not receiving one.
• Review the test case test_max_split_expandable in test_cuda.py to ensure it is correctly testing the expected behavior and handling edge cases.

Example

No specific code snippet can be provided without further investigation, but the test case in question is located in test_cuda.py and can be examined for potential issues.

Notes

The intermittent nature of the failure and the fact that CI is green despite the test being flaky suggest that the issue may be related to resource allocation or timing. Further investigation is needed to determine the root cause.

Recommendation

Apply workaround: Investigate and address potential memory allocation issues in the CUDA code, as this seems to be the most likely cause of the test failures.