pytorch - 💡(How to fix) Fix DISABLED test_dynamic_lstm_serdes_strict (__main__.SerDesExportTestExport) [2 comments, 2 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/export/test_export.py", line 1216, in test_dynamic_lstm self.assertEqual(eager_out, ep_out) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/test_case.py", line 117, in assertEqual return super().assertEqual(x, y, *args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual raise error_metas.pop()[0].to_error( # type: ignore[index] AssertionError: Tensor-likes are not close!

Mismatched elements: 40 / 262144 (0.0%) Greatest absolute difference: 1.989305019378662e-05 at index (1, 12, 405) (up to 1e-05 allowed) Greatest relative difference: 0.0010184849379584193 at index (0, 12, 463) (up to 1.3e-06 allowed)

To execute this test, run the following from the base repo dir: PYTORCH_TEST_WITH_ASAN=1 PYTORCH_TEST_WITH_UBSAN=1 python test/export/test_serdes.py SerDesExportTestExport.test_dynamic_lstm_serdes_strict

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

• The most likely fix involves investigating and addressing the numerical discrepancy in the test_dynamic_lstm_serdes_strict test, potentially by adjusting the tolerance values for absolute and relative differences.

Guidance

• Review the test case test_dynamic_lstm_serdes_strict in export/test_serdes.py to understand the conditions under which the AssertionError occurs.
• Investigate the numerical discrepancy by examining the logs for the test runs, focusing on the instances where the test fails, to identify patterns or specific inputs that lead to the failure.
• Consider adjusting the tolerance values for absolute and relative differences in the assertEqual method to accommodate the observed discrepancies, if they are deemed acceptable.
• Run the test with the suggested command PYTORCH_TEST_WITH_ASAN=1 PYTORCH_TEST_WITH_UBSAN=1 python test/export/test_serdes.py SerDesExportTestExport.test_dynamic_lstm_serdes_strict to reproduce and debug the issue locally.

Example

• No specific code snippet can be provided without modifying the existing test case, but adjusting the tolerance might look something like this:

  self.assertEqual(eager_out, ep_out, atol=1e-4, rtol=1e-6)

Notes

• The fix should ensure that the adjusted tolerance values do not compromise the test's effectiveness in catching significant errors.
• The numerical discrepancy might be due to various factors, including floating-point precision issues or differences in computation between eager and non-eager modes.

Recommendation

• Apply workaround: Adjust the tolerance values in the assertEqual method to accommodate the observed numerical discrepancies, after verifying that these adjustments do not compromise the integrity of the test.