Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4278, in test_maml_regression class ThreeLayerNet(nn.Module): File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4300, in torch_dynamo_resume_in_test_maml_regression_at_4278 ThreeLayerNet().to(device), mechanism File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4299, in torch_dynamo_resume_in_test_maml_regression_at_4300 net, params = _get_weights_and_functional_call( File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4350, in torch_dynamo_resume_in_test_maml_regression_at_4299 task = sample_tasks(num_tasks, K) File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4356, in torch_dynamo_resume_in_test_maml_regression_at_4350 inner_losses = vmap(partial(get_loss_for_task, True))( File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4360, in torch_dynamo_resume_in_test_maml_regression_at_4356 result_grads = torch.autograd.grad(loss2, list_params) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/init.py", line 530, in grad result = _engine_run_backward( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 882, in _engine_run_backward return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/compiled_autograd.py", line 1154, in runtime_wrapper out = compiled_fn( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 473, in call return super().call(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1047, in compile_wrapper result = fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 979, in call_wrapped return self._wrapped_call(self, *args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 486, in call raise e File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 472, in call return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2480, in call result = self._torchdynamo_orig_backend( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2198, in call result = self._inner_convert( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 758, in call result = _compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1983, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 96, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1593, in compile_inner result = _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1652, in _compile_inner dynamo_output = compile_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1500, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1626, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1472, in transform tracer_output = trace_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 365, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 933, in trace_frame run_tracer() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 914, in run_tracer tracer.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1506, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1043, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2832, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1407, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 306, in realize_and_forward return getattr(self.realize(), name)(args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py", line 2724, in call_function tensor_variable = wrap_fx_proxy( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3115, in wrap_fx_proxy return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3190, in wrap_fx_proxy_cls out: VTTypeAlias = _wrap_fx_proxy( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3314, in _wrap_fx_proxy example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3751, in get_fake_value return _get_fake_value_impl(node, tx, allow_non_graph_fake) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3942, in _get_fake_value_impl _wrap_graph_break_with_torch_runtime_err( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3740, in _wrap_graph_break_with_torch_runtime_err raise exc.with_traceback(e.traceback) from None File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3737, in _wrap_graph_break_with_torch_runtime_err gb_fn() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3943, in <lambda> lambda: unimplemented( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 653, in unimplemented raise Unsupported( torch._dynamo.exc.TorchRuntimeError: RuntimeError when making fake tensor call Explanation: Dynamo failed to run FX node with fake tensors: call_function torch._dynamo.compiled_autograd.ops.ReluBackward0(([FakeTensor(..., size=(4, 20, 40))], [True], FakeTensor(..., size=(4, 20, 40), grad_fn=<Error>)), **{}): got AssertionError('fake_mode.in_kernel_invocation must be False') Hint: Your code may result in an error when running in eager. Please double check that your code doesn't contain a similar error when actually running eager/uncompiled. You can do this by removing the torch.compile call, or by using torch.compiler.set_stance("force_eager").

Developer debug context:

For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb4315.html

from user code: File "<eval_with_key>.2504", line 188, in forward relu_backward0 = torch__dynamo_compiled_autograd_ops_ReluBackward0([getitem_120], [True], getitem_8); getitem_120 = getitem_8 = None

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

To execute this test, run the following from the base repo dir: PYTORCH_TEST_WITH_DYNAMO=1 python test/functorch/test_eager_transforms.py TestExamplesCorrectnessCPU.test_maml_regression_mechanism_make_functional_cpu

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_maml_regression_mechanism_make_functional_cpu is failing due to a TorchRuntimeError caused by Dynamo's failure to run an FX node with fake tensors, suggesting an issue with the code when running in eager mode.

Guidance

• The error message hints at an issue with the code when running in eager mode, suggesting that the code may contain an error similar to the one encountered when running eager/uncompiled.
• To debug this issue, try removing the torch.compile call or use torch.compiler.set_stance("force_eager") to run the code in eager mode and check for errors.
• Set TORCHDYNAMO_VERBOSE=1 to get the internal stack trace, which may provide more information about the error.
• Set TORCH_LOGS="+dynamo" for even more developer context.

Example

No specific code example can be provided without modifying the original code, but the error message suggests modifying the forward method in the test_maml_regression_mechanism_make_functional_cpu test to handle the relu_backward0 calculation differently.

Notes

The provided error message and stack trace suggest that the issue is related to the interaction between Dynamo and the code being tested, but without more information about the code and the specific test being run, it's difficult to provide a more specific solution.

Recommendation

Apply workaround by removing the torch.compile call or using torch.compiler.set_stance("force_eager") to run the code in eager mode and check for errors, as this may help identify the root cause of the issue.