pytorch - 💡(How to fix) Fix DISABLED test_nhwc_autocast_jit_trace_model (__main__.TestJitTraceAutocast)

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/test_jit_autocast.py", line 853, in test_nhwc_autocast_jit_trace_model test_nhwc_autocast_jit_trace_model(self.models[i], self.inputs[i]) File "/var/lib/jenkins/workspace/test/test_jit_autocast.py", line 848, in test_nhwc_autocast_jit_trace_model torch.testing.assert_close(y.double(), y2.double(), rtol=1e-03, atol=1e-03) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1631, in assert_close raise error_metas[0].to_error(msg) AssertionError: Tensor-likes are not close!

Mismatched elements: 4 / 24332288 (0.0%) Greatest absolute difference: 0.44140625 at index (29, 12, 78, 60) (up to 0.001 allowed) Greatest relative difference: 1.6142857142857143 at index (29, 12, 78, 60) (up to 0.001 allowed)

To execute this test, run the following from the base repo dir: python test/test_jit_autocast.py TestJitTraceAutocast.test_nhwc_autocast_jit_trace_model

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_nhwc_autocast_jit_trace_model is failing due to assertion errors in tensor comparisons, suggesting a potential issue with numerical precision or model behavior.

Guidance

• Review the provided recent examples and workflow logs to understand the failure patterns.
• Follow the debugging instructions to find relevant log snippets by grepping for test_nhwc_autocast_jit_trace_model in the workflow logs.
• Investigate the assertion error in torch.testing.assert_close to determine if the issue is related to numerical precision, model implementation, or test setup.
• Consider relaxing the tolerance values (rtol and atol) in torch.testing.assert_close to see if the test passes with less stringent comparison criteria.

Example

No code snippet is provided as the issue does not imply a specific code change, but rather an investigation into the test failure.

Notes

The issue is marked as flaky, indicating that it may not fail consistently, which can make debugging more challenging. The provided error message suggests a small percentage of mismatched elements, which might be due to numerical precision issues or model behavior.

Recommendation

Apply workaround: Relax the tolerance values in torch.testing.assert_close to temporarily mitigate the issue while further investigation is conducted to determine the root cause.