pytorch - 💡(How to fix) Fix AArch64 unit test failure TestFX.test_shape_prop_layout_3d [2 comments, 2 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/passes/shape_prop.py", line 180, in run_node result = super().run_node(n) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 297, in run_node return getattr(self, n.op)(n.target, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 431, in call_module return submod(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 733, in forward return self._conv_forward(input, self.weight, self.bias) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 728, in _conv_forward return F.conv3d( RuntimeError: label is too far ----------------------------- Captured stderr call ----------------------------- Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/passes/shape_prop.py", line 180, in run_node result = super().run_node(n) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 297, in run_node return getattr(self, n.op)(n.target, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 431, in call_module return submod(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 733, in forward return self._conv_forward(input, self.weight, self.bias) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 728, in _conv_forward return F.conv3d( RuntimeError: label is too far =================================== FAILURES =================================== _______________________ TestFX.test_shape_prop_layout_3d _______________________ Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/passes/shape_prop.py", line 180, in run_node result = super().run_node(n) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 297, in run_node return getattr(self, n.op)(n.target, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 431, in call_module return submod(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 733, in forward return self._conv_forward(input, self.weight, self.bias) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 728, in _conv_forward return F.conv3d( RuntimeError: std::future_error: Broken promise

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/test_fx.py", line 2069, in test_shape_prop_layout_3d shape_prop.ShapeProp(traced_3d).propagate(x_3d) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/passes/shape_prop.py", line 230, in propagate return super().run(*fake_args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 200, in run self.env[node] = self.run_node(node) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/passes/shape_prop.py", line 185, in run_node raise RuntimeError( RuntimeError: ShapeProp error for: node=%conv_mod : [num_users=1] = call_module[target=conv_mod](args = (%x,), kwargs = {}) with meta={'nn_module_stack': OrderedDict([('conv_mod', ('conv_mod', <class 'torch.nn.modules.conv.Conv3d'>))])}

While executing %conv_mod : [num_users=1] = call_module[target=conv_mod](args = (%x,), kwargs = {}) Original traceback: None Use tlparse to see full graph. (https://github.com/pytorch/tlparse?tab=readme-ov-file#tlparse-parse-structured-pt2-logs)

To execute this test, run the following from the base repo dir: python test/test_fx.py TestFX.test_shape_prop_layout_3d

TL;DR

The most likely fix for the AArch64 unit test failure in text_fx.py is to investigate and address the RuntimeError: label is too far error, potentially related to the Conv3d module.

Guidance

• Investigate the Conv3d module and its usage in the test_shape_prop_layout_3d test to understand the cause of the RuntimeError: label is too far error.
• Check the input and weight shapes, as well as the bias, to ensure they are compatible with the Conv3d module.
• Review the PyTorch version and consider updating to a newer version if available, as the current version torch-2.12.0a0+gitc6845b4 may have known issues.
• Use the tlparse tool to visualize the graph and gain more insights into the error.

Example

No specific code snippet can be provided without more context, but reviewing the test_shape_prop_layout_3d test and the Conv3d module implementation may help identify the issue.

Notes

The error may be related to a known issue in the PyTorch version or a specific hardware architecture (AArch64). Further investigation is needed to determine the root cause.

Recommendation

Apply a workaround by investigating and addressing the RuntimeError: label is too far error, as updating to a fixed version is not explicitly implied in the issue.