pytorch - 💡(How to fix) Fix DISABLED test_allow_reuse_disable_if_exceed_peak_cuda (__main__.GPUTests) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 59, in testPartExecutor yield File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3443, in wrapper method(*args, **kwargs) File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 16330, in new_test return value(self) File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner return func(*args, **kwds) File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 15621, in test_allow_reuse_disable_if_exceed_peak self.assertFalse(CommonTemplate._is_triggering_buffer_reuse(fn, inp)) File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 681, in assertFalse raise self.failureException(msg) AssertionError: True is not false

To execute this test, run the following from the base repo dir: python test/inductor/test_torchinductor.py GPUTests.test_allow_reuse_disable_if_exceed_peak_cuda

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix for the flaky test test_allow_reuse_disable_if_exceed_peak_cuda is to investigate and resolve the inconsistency in the CommonTemplate._is_triggering_buffer_reuse method.

Guidance

• Investigate the CommonTemplate._is_triggering_buffer_reuse method to understand why it's returning True when expected to return False.
• Run the test locally using the provided command python test/inductor/test_torchinductor.py GPUTests.test_allow_reuse_disable_if_exceed_peak_cuda to reproduce the issue.
• Study the logs from the CI workflow by following the provided debugging instructions to gain more insight into the test failures.
• Check the test file inductor/test_torchinductor.py for any recent changes that might be causing the flakiness.

Example

No code snippet is provided as the issue does not contain enough information to suggest a specific code change.

Notes

The test is flaky, meaning it sometimes passes and sometimes fails, which can make it harder to debug. The provided debugging instructions and logs from the CI workflow should help in identifying the root cause.

Recommendation

Apply workaround: Temporarily enable the test and run it locally to gather more information about the failures, then disable it again until a proper fix is found. This will allow developers to investigate the issue without affecting the CI pipeline.