pytorch - 💡(How to fix) Fix DISABLED test_find_learning_rate_ensembling_FeatureAlphaDropout_mechanism_functional_call_cpu (__main__.TestExamplesCorrectnessCPU) [2 comments, 1 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4703, in test_find_learning_rate_ensembling class MLPClassifier(nn.Module): File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4725, in torch_dynamo_resume_in_test_find_learning_rate_ensembling_at_4703 MLPClassifier().to(device), mechanism File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4754, in torch_dynamo_resume_in_test_find_learning_rate_ensembling_at_4725 batched_weights = init_fn(num_models=2) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2474, in call result = self._torchdynamo_orig_backend( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2192, in call result = self._inner_convert( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 736, in call result = _compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1961, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 96, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1571, in compile_inner result = _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1630, in _compile_inner dynamo_output = compile_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1478, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1626, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1450, in transform tracer_output = trace_frame( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 343, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 911, in trace_frame run_tracer() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 892, in run_tracer tracer.run() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1813, in run while self.step(): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1480, in step self.dispatch_table[inst.opcode](self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1017, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2867, in CALL_FUNCTION_KW self.call_function(fn, args, kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1381, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 294, in realize_and_forward return getattr(self.realize(), name)(args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py", line 2694, in call_function tensor_variable = wrap_fx_proxy( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3084, in wrap_fx_proxy return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3159, in wrap_fx_proxy_cls out: VTTypeAlias = _wrap_fx_proxy( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3283, in _wrap_fx_proxy example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3751, in get_fake_value return _get_fake_value_impl(node, tx, allow_non_graph_fake) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3942, in _get_fake_value_impl _wrap_graph_break_with_torch_runtime_err( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3740, in _wrap_graph_break_with_torch_runtime_err raise exc.with_traceback(e.traceback) from None File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3737, in _wrap_graph_break_with_torch_runtime_err gb_fn() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3943, in <lambda> lambda: unimplemented( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 653, in unimplemented raise Unsupported( torch._dynamo.exc.TorchRuntimeError: RuntimeError when making fake tensor call Explanation: Dynamo failed to run FX node with fake tensors: call_function <built-in method tensor of type object at 0x7fcdcdb77ba0>(([0.2, 0.4],), **{'device': 'cpu'}): got RuntimeError('tls_on_entry.has_value() INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/aten/src/ATen/core/PythonFallbackKernel.cpp":53, please report a bug to PyTorch. ') Hint: Your code may result in an error when running in eager. Please double check that your code doesn't contain a similar error when actually running eager/uncompiled. You can do this by removing the torch.compile call, or by using torch.compiler.set_stance("force_eager").

Developer debug context:

For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb4315.html

from user code: File "/var/lib/jenkins/workspace/test/functorch/test_eager_transforms.py", line 4759, in torch_dynamo_resume_in_test_find_learning_rate_ensembling_at_4754 lrs = torch.tensor([0.2, 0.4], device=device)

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

To execute this test, run the following from the base repo dir: PYTORCH_TEST_WITH_DYNAMO=1 python test/functorch/test_eager_transforms.py TestExamplesCorrectnessCPU.test_find_learning_rate_ensembling_FeatureAlphaDropout_mechanism_functional_call_cpu

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_find_learning_rate_ensembling_FeatureAlphaDropout_mechanism_functional_call_cpu is failing due to a TorchRuntimeError caused by Dynamo's failure to run an FX node with fake tensors, and a potential workaround is to remove the torch.compile call or use torch.compiler.set_stance("force_eager").

Guidance

• To debug this issue, set TORCHDYNAMO_VERBOSE=1 to get the internal stack trace and TORCH_LOGS="+dynamo" for more developer context.
• Remove the torch.compile call or use torch.compiler.set_stance("force_eager") to check if the code contains a similar error when running in eager mode.
• Visit the provided link https://meta-pytorch.github.io/compile-graph-break-site/gb/gb4315.html for more details about the graph break.
• Run the test with PYTORCH_TEST_WITH_DYNAMO=1 python test/functorch/test_eager_transforms.py TestExamplesCorrectnessCPU.test_find_learning_rate_ensembling_FeatureAlphaDropout_mechanism_functional_call_cpu to reproduce the error.

Example

No code snippet is provided as the issue is related to a specific test case and requires more context to provide a accurate example.

Notes

The error message suggests that the issue is related to the torch.tensor call, specifically when creating a tensor with a device argument. The provided link to the graph break site may contain more information about the specific error.

Recommendation

Apply the workaround by removing the torch.compile call or using torch.compiler.set_stance("force_eager") to check if the code contains a similar error when running in eager mode, as this may help identify the root cause of the issue.