pytorch - 💡(How to fix) Fix DISABLED test_max_cached_size (__main__.TestCachingHostAllocatorConfig) [2 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 59, in testPartExecutor yield File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3528, in wrapper method(*args, **kwargs) File "/var/lib/jenkins/workspace/test/test_cuda.py", line 6298, in test_max_cached_size self.assertEqual(stats["allocations.current"], 0) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4525, in assertEqual raise error_metas.pop()[0].to_error( # type: ignore[index] AssertionError: Scalars are not equal!

Expected 0 but got 1. Absolute difference: 1 Relative difference: inf

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/test_cuda.py TestCachingHostAllocatorConfig.test_max_cached_size

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix involves investigating and addressing the inconsistency in the test_max_cached_size test, which is causing flaky failures in CI.

Guidance

• Investigate the test_max_cached_size test in test_cuda.py to understand why it's failing with an AssertionError due to unexpected allocations.
• Review the workflow logs and grep for test_max_cached_size to identify patterns or clues that might explain the flaky behavior.
• Consider running the test locally with PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 and PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 to reproduce and debug the issue.
• Check the test's implementation and assertions to ensure they are correct and not prone to false positives or negatives.

Notes

The provided information suggests that the issue is related to a specific test, but without more context or details about the test's implementation, it's challenging to provide a definitive fix. Further investigation and debugging are necessary to resolve the issue.

Recommendation

Apply workaround: Temporarily enable the test and run it locally with the provided environment variables to gather more information and debug the issue, as the root cause is not immediately apparent from the provided error message and logs.