pytorch - 💡(How to fix) Fix DISABLED test_dynamic_lstm_strict (__main__.StrictExportTestExport) [1 comments, 1 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/export/test_export.py", line 1216, in test_dynamic_lstm self.assertEqual(eager_out, ep_out) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/test_case.py", line 117, in assertEqual return super().assertEqual(x, y, *args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4445, in assertEqual raise error_metas.pop()[0].to_error( # type: ignore[index] AssertionError: Tensor-likes are not close!

Mismatched elements: 22 / 262144 (0.0%) Greatest absolute difference: 2.6635825634002686e-05 at index (0, 0, 494) (up to 1e-05 allowed) Greatest relative difference: 0.0015868250047788024 at index (0, 0, 466) (up to 1.3e-06 allowed)

To execute this test, run the following from the base repo dir: PYTORCH_TEST_WITH_ASAN=1 PYTORCH_TEST_WITH_UBSAN=1 python test/export/test_export_strict.py StrictExportTestExport.test_dynamic_lstm_strict

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix involves investigating and addressing the numerical instability or precision issues in the test_dynamic_lstm_strict test, potentially by adjusting tolerance values or improving the numerical stability of the LSTM implementation.

Guidance

• Investigate the test_dynamic_lstm_strict test in export/test_export_strict.py to understand the numerical instability causing the AssertionError: Tensor-likes are not close! error.
• Review the logs from the workflow runs to identify patterns or common factors contributing to the test failures.
• Consider adjusting the tolerance values for absolute and relative differences in the assertEqual method to accommodate the observed numerical instability.
• Examine the LSTM implementation for potential improvements to numerical stability, such as using more robust numerical methods or increasing the precision of intermediate calculations.

Example

No specific code snippet is provided, as the issue requires a deeper investigation into the numerical stability of the LSTM implementation.

Notes

The fix may involve a combination of adjusting tolerance values, improving numerical stability, and potentially modifying the test to better handle numerical instability.

Recommendation

Apply a workaround by adjusting the tolerance values in the assertEqual method to temporarily mitigate the issue, while also investigating the root cause of the numerical instability.