pytorch - 💡(How to fix) Fix DISABLED test_missing_abstract_impl_cuda (__main__.TestCustomOpTestingCUDA) [2 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/optests/generate_tests.py", line 695, in opcheck tester(op, args, kwargs, rtol=rtol, atol=atol) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/optests/generate_tests.py", line 61, in safe_schema_check result = op(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 871, in call return self._op(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_compile.py", line 54, in inner return disable_fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1277, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/schema_check_mode.py", line 184, in torch_dispatch out = func(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 871, in call return self._op(*args, **kwargs) File "/var/lib/jenkins/workspace/test/test_custom_ops.py", line 282, in foo_impl return torch.tensor(x.cpu().numpy() ** 2, device=x.device) RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.

The above exception was the direct cause of the following exception:

torch.testing._internal.optests.generate_tests.OpCheckError: opcheck(op, ...): test_schema failed with Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead. (scroll up for stack trace)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/test_custom_ops.py", line 290, in test_missing_abstract_impl with self.assertRaisesRegex( File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 239, in exit self._raiseFailure('"{}" does not match "{}"'.format( File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure raise self.test_case.failureException(msg) AssertionError: "_test_custom_op.foo.default" does not match "opcheck(op, ...): test_schema failed with Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead. (scroll up for stack trace)"

To execute this test, run the following from the base repo dir: python test/test_custom_ops.py TestCustomOpTestingCUDA.test_missing_abstract_impl_cuda

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix is to modify the foo_impl function to use tensor.detach().numpy() instead of tensor.numpy() to resolve the RuntimeError caused by calling numpy() on a Tensor that requires grad.

Guidance

• Identify the foo_impl function in test_custom_ops.py and modify the line return torch.tensor(x.cpu().numpy() ** 2, device=x.device) to use detach() before calling numpy(), like this: return torch.tensor(x.cpu().detach().numpy() ** 2, device=x.device).
• Verify that the test test_missing_abstract_impl_cuda passes after making this change by running the test using the command python test/test_custom_ops.py TestCustomOpTestingCUDA.test_missing_abstract_impl_cuda.
• Check the CI workflow logs to ensure that the test is no longer flaky and the error message is resolved.
• If the issue persists, investigate other potential causes of flakiness in the test, such as random number generation or GPU-related issues.

Example

def foo_impl(x):
    # Modified line:
    return torch.tensor(x.cpu().detach().numpy() ** 2, device=x.device)

Notes

The provided stack trace and error message suggest that the issue is specific to the foo_impl function and the use of numpy() on a Tensor that requires grad. However, the test is reported as flaky, which may indicate other underlying issues that need to be addressed.

Recommendation

Apply the workaround by modifying the foo_impl function to use detach() before calling numpy(), as this directly addresses the reported error message and is a common solution to this type of issue.