Traceback (most recent call last): File "/__w/pytorch/pytorch/test/functorch/test_eager_transforms.py", line 4703, in test_find_learning_rate_ensembling class MLPClassifier(nn.Module): File "/__w/pytorch/pytorch/test/functorch/test_eager_transforms.py", line 4725, in torch_dynamo_resume_in_test_find_learning_rate_ensembling_at_4703 MLPClassifier().to(device), mechanism File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2474, in call result = self._torchdynamo_orig_backend( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2192, in call result = self._inner_convert( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 736, in call result = _compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1961, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 96, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1571, in compile_inner result = _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1630, in _compile_inner dynamo_output = compile_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1478, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1626, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1450, in transform tracer_output = trace_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 343, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 911, in trace_frame run_tracer() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 892, in run_tracer tracer.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1813, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1480, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1017, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2787, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1381, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1674, in call_function return super().call_function(tx, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 815, in call_function return super().call_function(tx, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 494, in call_function return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) # type: ignore[attr-defined] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1408, in inline_user_function_return return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/dynamo/symbolic_convert.py", line 5242, in inline_call return tracer.inline_call() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/dynamo/symbolic_convert.py", line 5463, in inline_call self.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1813, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1480, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1017, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2787, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1381, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1674, in call_function return super().call_function(tx, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 815, in call_function return super().call_function(tx, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 494, in call_function return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) # type: ignore[attr-defined] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1408, in inline_user_function_return return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/dynamo/symbolic_convert.py", line 5242, in inline_call return tracer.inline_call() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/dynamo/symbolic_convert.py", line 5463, in inline_call self.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1813, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1480, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1017, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2787, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1381, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1674, in call_function return super().call_function(tx, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 815, in call_function return super().call_function(tx, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 494, in call_function return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) # type: ignore[attr-defined] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1408, in inline_user_function_return return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/dynamo/symbolic_convert.py", line 5242, in inline_call return tracer.inline_call() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/dynamo/symbolic_convert.py", line 5463, in inline_call self.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1813, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1480, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1017, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2787, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1381, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 494, in call_function return tx.inline_user_function_return(self, [*self.self_args(), args], kwargs) # type: ignore[attr-defined] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1408, in inline_user_function_return return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/dynamo/symbolic_convert.py", line 5242, in inline_call return tracer.inline_call() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/dynamo/symbolic_convert.py", line 5463, in inline_call self.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1813, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1480, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1017, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2787, in CALL_FUNCTION self.call_function(fn, args, {}) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1381, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 1354, in call_function return self.obj.call_method(tx, self.name, list(args), kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 970, in call_method result = wrap_fx_proxy(tx, proxy) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3084, in wrap_fx_proxy return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3159, in wrap_fx_proxy_cls out: VTTypeAlias = _wrap_fx_proxy( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3283, in _wrap_fx_proxy example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3752, in get_fake_value return _get_fake_value_impl(node, tx, allow_non_graph_fake) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3943, in _get_fake_value_impl _wrap_graph_break_with_torch_runtime_err( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3741, in _wrap_graph_break_with_torch_runtime_err raise exc.with_traceback(e.traceback) from None File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3738, in _wrap_graph_break_with_torch_runtime_err gb_fn() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3944, in <lambda> lambda: unimplemented( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 653, in unimplemented raise Unsupported( torch._dynamo.exc.TorchRuntimeError: RuntimeError when making fake tensor call Explanation: Dynamo failed to run FX node with fake tensors: call_method to((Parameter(FakeTensor(..., size=(32, 16), requires_grad=True)), device(type='cpu'), None, False), **{}): got RuntimeError('tls_on_entry.has_value() INTERNAL ASSERT FAILED at "/__w/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp":53, please report a bug to PyTorch. ') Hint: Your code may result in an error when running in eager. Please double check that your code doesn't contain a similar error when actually running eager/uncompiled. You can do this by removing the torch.compile call, or by using torch.compiler.set_stance("force_eager").

Developer debug context:

For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb4315.html

from user code: File "/__w/pytorch/pytorch/test/functorch/test_eager_transforms.py", line 4725, in torch_dynamo_resume_in_test_find_learning_rate_ensembling_at_4725 MLPClassifier().to(device), mechanism File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1383, in to return self._apply(convert) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 933, in _apply module._apply(fn) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 964, in _apply param_applied = fn(param) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1369, in convert return t.to(

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

To execute this test, run the following from the base repo dir: PYTORCH_TEST_WITH_DYNAMO=1 python test/functorch/test_eager_transforms.py TestExamplesCorrectnessCPU.test_find_learning_rate_ensembling_Dropout_mechanism_functional_call_cpu

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_find_learning_rate_ensembling_Dropout_mechanism_functional_call_cpu is failing due to a TorchRuntimeError caused by Dynamo failing to run an FX node with fake tensors, and the issue may be related to the use of torch.compile or eager mode.

Guidance

• To debug this issue, try removing the torch.compile call or using torch.compiler.set_stance("force_eager") to run the code in eager mode.
• Set TORCHDYNAMO_VERBOSE=1 to get the internal stack trace, which may provide more information about the error.
• Set TORCH_LOGS="+dynamo" to get more developer context.
• Run the test with PYTORCH_TEST_WITH_DYNAMO=1 python test/functorch/test_eager_transforms.py TestExamplesCorrectnessCPU.test_find_learning_rate_ensembling_Dropout_mechanism_functional_call_cpu to reproduce the error.

Example

No specific code example is provided, but the error message suggests that the issue is related to the MLPClassifier().to(device) line in the test_eager_transforms.py file.

Notes

The error message provides a link to a graph break site (https://meta-pytorch.github.io/compile-graph-break-site/gb/gb4315.html) that may provide more information about the issue. The test is disabled due to flakiness, and the error message suggests that the issue may be related to the use of torch.compile or eager mode.

Recommendation

Apply a workaround by removing the torch.compile call or using torch.compiler.set_stance("force_eager") to run the code in eager mode, as suggested by the error message. This may help to identify the root cause of the issue and provide more information for debugging.