pytorch - 💡(How to fix) Fix DISABLED test_both_thresholds (__main__.TestCachingHostAllocatorConfig) [1 comments, 1 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/test_cuda.py", line 6312, in test_both_thresholds self.assertEqual(stats["allocations.current"], 1) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4525, in assertEqual raise error_metas.pop()[0].to_error( # type: ignore[index] AssertionError: Scalars are not equal!

Expected 1 but got 2. Absolute difference: 1 Relative difference: 1.0

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/test_cuda.py TestCachingHostAllocatorConfig.test_both_thresholds

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix involves investigating and resolving the inconsistency in the test_both_thresholds test case, where the expected allocation count is 1 but the actual count is 2.

Guidance

• Investigate the test_both_thresholds test case in test_cuda.py to understand why the expected allocation count is 1 but the actual count is 2.
• Review the workflow logs and grep for test_both_thresholds to study the logs and identify potential causes of the inconsistency.
• Run the test locally using the provided command PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/test_cuda.py TestCachingHostAllocatorConfig.test_both_thresholds to reproduce and debug the issue.
• Check the stats["allocations.current"] value to see why it's returning 2 instead of the expected 1.

Notes

The issue seems to be related to a flaky test case, and the provided error message suggests an inconsistency in the allocation count. However, without more information about the test case and the code, it's difficult to provide a more specific fix.

Recommendation

Apply workaround: Investigate and resolve the inconsistency in the test_both_thresholds test case, as it's currently causing flaky test failures.