pytorch - 💡(How to fix) Fix DISABLED test_dynamo_dtensor_from_local_redistribute (__main__.TestDTensorCompileWithCompiledAutograd) [2 comments, 1 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/distributed/tensor/test_dtensor_compile.py", line 1444, in test_dynamo_dtensor_from_local_redistribute self.assertExpectedInline( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/internal/common_utils.py", line 3391, in assertExpectedInline return super().assertExpectedInline(actual if isinstance(actual, str) else str(actual), expect, skip + 1) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/init.py", line 413, in assertExpectedInline assert_expected_inline( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/init.py", line 378, in assert_expected_inline assert_eq(expect, actual, msg=help_text) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/expecttest/init.py", line 450, in assertMultiLineEqualMaybeCppStack self.assertMultiLineEqual(expect, actual, *args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 1226, in assertMultiLineEqual self.fail(self.formatMessage(msg, standardMsg)) File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 675, in fail raise self.failureException(msg) AssertionError: 'def [470 chars] add = to_local + 2; to_local = None\n return (add,)' != 'def [470 chars] add = to_local.add(2); to_local = None\n return (add,)' def forward(self, L_x : torch.Tensor, L_mesh : torch.distributed.device_mesh.DeviceMesh): l_x_ = L_x_ l_mesh_ = L_mesh_ dt = torch.distributed.tensor.api.from_local(l_x, l_mesh_, [torch.distributed.tensor.placement_types.Shard(dim=0)], run_check = False); l_x_ = None redistribute = dt.redistribute(l_mesh_, [torch.distributed.tensor.placement_types.Replicate()]); dt = l_mesh_ = None to_local = redistribute.to_local(); redistribute = None

• add = to_local + 2;  to_local = None

? ^^^

• add = to_local.add(2);  to_local = None

? ^^^^^ + return (add,) : To accept the new output, re-run test with envvar EXPECTTEST_ACCEPT=1 (we recommend staging/committing your changes before doing this)

To execute this test, run the following from the base repo dir: PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 PYTORCH_TEST_WITH_SLOW_GRADCHECK=1 python test/distributed/tensor/test_dtensor_compile.py TestDTensorCompileWithCompiledAutograd.test_dynamo_dtensor_from_local_redistribute

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix is to update the test case to use the correct method for adding a value to a tensor, which is add() instead of the + operator.

Guidance

• Review the test case in test_dtensor_compile.py and update the line add = to_local + 2 to add = to_local.add(2) to fix the assertion error.
• Run the test again with the updated code to verify that the fix works.
• If the issue persists, check the workflow logs for any other errors or warnings that may be related to the test failure.
• Consider re-running the test with the EXPECTTEST_ACCEPT=1 envvar to accept the new output and update the expected result.

Example

# Updated test case
def test_dynamo_dtensor_from_local_redistribute(self):
    # ...
    to_local = redistribute.to_local()
    add = to_local.add(2)  # Update this line
    return (add,)

Notes

The provided stacktrace and error message suggest that the issue is related to the use of the + operator instead of the add() method for adding a value to a tensor. Updating the test case to use the correct method should fix the assertion error.

Recommendation

Apply the workaround by updating the test case to use the add() method instead of the + operator, as this is the most likely cause of the issue.