pytorch - 💡(How to fix) Fix DISABLED test_opcheck_fails_basic_cuda (__main__.TestCustomOpTestingCUDA) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 2937, in _dispatch_impl r = func(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 871, in call return self._op(*args, **kwargs) NotImplementedError: _test_custom_op::foo: attempted to run this operator with Meta tensors, but there was no fake impl or Meta kernel registered. You may have run into this message while using an operator with PT2 compilation APIs (torch.compile/torch.export); in order to use this operator with those APIs you'll need to add a fake impl. Please see the following for next steps: https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/optests/generate_tests.py", line 695, in opcheck tester(op, args, kwargs, rtol=rtol, atol=atol) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/optests/generate_tests.py", line 99, in safe_fake_check return fake_check(op, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/optests/fake_tensor.py", line 12, in fake_check op(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 871, in call return self._op(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_compile.py", line 54, in inner return disable_fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1277, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_utils.py", line 277, in torch_dispatch fake_r = func(*fake_args, **fake_kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 871, in call return self._op(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_compile.py", line 54, in inner return disable_fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1277, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 29, in wrapper return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1463, in torch_dispatch return self.dispatch(func, types, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 2240, in dispatch return self._cached_dispatch_impl(func, types, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1611, in _cached_dispatch_impl output = self._dispatch_impl(func, types, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 2939, in _dispatch_impl return maybe_run_unsafe_fallback(not_implemented_error) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 2921, in maybe_run_unsafe_fallback raise UnsupportedOperatorException(func) torch._subclasses.fake_tensor.UnsupportedOperatorException: _test_custom_op.foo.default

The above exception was the direct cause of the following exception:

torch.testing._internal.optests.generate_tests.OpCheckError: opcheck(op, ...): test_faketensor failed with _test_custom_op.foo.default (scroll up for stack trace)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/test_custom_ops.py", line 456, in test_opcheck_fails_basic with self.assertRaisesRegex( File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 239, in exit self._raiseFailure('"{}" does not match "{}"'.format( File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 163, in _raiseFailure raise self.test_case.failureException(msg) AssertionError: "Autograd has not been implemented for operator" does not match "opcheck(op, ...): test_faketensor failed with _test_custom_op.foo.default (scroll up for stack trace)"

To execute this test, run the following from the base repo dir: python test/test_custom_ops.py TestCustomOpTestingCUDA.test_opcheck_fails_basic_cuda

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_opcheck_fails_basic_cuda is failing due to an UnsupportedOperatorException and needs a fake implementation for the custom operator _test_custom_op::foo.

Guidance

1. Review the error message: The error indicates that the custom operator _test_custom_op::foo is not implemented for fake tensors, which is required for the test.
2. Add a fake implementation: Follow the instructions at https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html to add a fake implementation for the custom operator.
3. Verify the fix: Run the test test_opcheck_fails_basic_cuda to verify that the fix resolves the issue.

Example

No code example is provided as the fix requires adding a custom fake implementation, which is specific to the operator and requires knowledge of the operator's functionality.

Notes

The fix may require modifying the custom operator's code to include a fake implementation, which may involve registering a fake kernel or implementing a fake forward pass.

Recommendation

Apply a workaround by adding a fake implementation for the custom operator _test_custom_op::foo to resolve the UnsupportedOperatorException. This will allow the test to run successfully.