pytorch - 💡(How to fix) Fix DISABLED test_dynamic_lstm_training_ir_to_decomp_strict (__main__.TrainingIRToRunDecompExportTestExport) [1 comments, 1 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/export/test_export.py", line 1216, in test_dynamic_lstm self.assertEqual(eager_out, ep_out) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/test_case.py", line 117, in assertEqual return super().assertEqual(x, y, *args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual raise error_metas.pop()[0].to_error( # type: ignore[index] AssertionError: Tensor-likes are not close!

Mismatched elements: 23 / 262144 (0.0%) Greatest absolute difference: 2.2914260625839233e-05 at index (0, 8, 213) (up to 1e-05 allowed) Greatest relative difference: 0.002367280190810561 at index (0, 8, 105) (up to 1.3e-06 allowed)

To execute this test, run the following from the base repo dir: PYTORCH_TEST_WITH_ASAN=1 PYTORCH_TEST_WITH_UBSAN=1 python test/export/test_export_training_ir_to_run_decomp.py TrainingIRToRunDecompExportTestExport.test_dynamic_lstm_training_ir_to_decomp_strict

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_dynamic_lstm_training_ir_to_decomp_strict is failing due to assertion errors caused by mismatched tensor elements, and a workaround or fix is needed to address the issue.

Guidance

• Review the workflow logs and grep for test_dynamic_lstm_training_ir_to_decomp_strict to identify the specific instances where the test is failing.
• Investigate the test_dynamic_lstm function in test_export.py to understand the conditions under which the AssertionError is raised.
• Check the tolerance values used in the assertEqual method to determine if they need to be adjusted to account for the small differences in tensor elements.
• Run the test locally using the provided command to reproduce the issue and gather more information.

Example

No code snippet is provided as the issue is related to a specific test case and requires further investigation.

Notes

The issue is specific to the test_dynamic_lstm_training_ir_to_decomp_strict test case, and the provided error message suggests that the issue is related to small differences in tensor elements. The test is currently disabled, and the goal is to identify the root cause and implement a fix or workaround.

Recommendation

Apply a workaround by adjusting the tolerance values used in the assertEqual method to account for the small differences in tensor elements, as the current tolerance values may be too strict.