pytorch - 💡(How to fix) Fix DISABLED test_update_batch_norm_mechanism_make_functional_originally_track_running_stats_True_cpu (__main__.TestExamplesCorrectnessCPU) [1 comments, 1 participants]

Traceback (most recent call last): File "/__w/pytorch/pytorch/test/functorch/test_eager_transforms.py", line 4471, in test_update_batch_norm nn.Conv2d(64, 64, 3), File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 2480, in call result = self._torchdynamo_orig_backend( frame, cache_entry, self.hooks, frame_state, skip=1 ) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 2198, in call result = self._inner_convert( frame, cache_entry, hooks, frame_state, skip=skip + 1 ) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 758, in call result = _compile( frame.f_code, ...<16 lines>... convert_frame_box=self._box, ) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 1983, in _compile guarded_code, tracer_output = compile_inner(code, one_graph, hooks) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_utils_internal.py", line 96, in wrapper_function return function(*args, **kwargs) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 1593, in compile_inner result = _compile_inner(code, one_graph, hooks) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 1652, in _compile_inner dynamo_output = compile_frame( code, ...<11 lines>... package=package, ) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 1500, in compile_frame bytecode, tracer_output = transform_code_object(code, transform) ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/bytecode_transformation.py", line 1626, in transform_code_object tracer_output = transformations(instructions, code_options) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 1472, in transform tracer_output = trace_frame( code, ...<14 lines>... package=package, ) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 365, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 933, in trace_frame run_tracer() ~~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/convert_frame.py", line 914, in run_tracer tracer.run() ~~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run while self.step(): ~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1506, in step self.dispatch_table[inst.opcode](self, inst) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1043, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 4431, in CALL_KW self._call(inst, call_kw=True) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 4207, in _call self.call_function(fn, args, kwargs) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1407, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/lazy.py", line 306, in realize_and_forward return getattr(self.realize(), name)(*args, **kwargs) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/user_defined.py", line 1181, in call_function result = tx.inline_user_function_return( VariableTracker.build( ...<3 lines>... kwargs, ) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1434, in inline_user_function_return return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/dynamo/symbolic_convert.py", line 5309, in inline_call return tracer.inline_call() ~~~~~~~~~~~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/dynamo/symbolic_convert.py", line 5530, in inline_call self.run() ~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run while self.step(): ~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1506, in step self.dispatch_table[inst.opcode](self, inst) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1043, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 2896, in CALL_FUNCTION_EX self.call_function(fn, argsvars.items, kwargsvars) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1407, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/functions.py", line 1672, in call_function return super().call_function(tx, args, kwargs) ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/functions.py", line 812, in call_function return super().call_function(tx, args, kwargs) ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/functions.py", line 491, in call_function return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) # type: ignore[attr-defined] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1434, in inline_user_function_return return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/dynamo/symbolic_convert.py", line 5309, in inline_call return tracer.inline_call() ~~~~~~~~~~~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/dynamo/symbolic_convert.py", line 5530, in inline_call self.run() ~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run while self.step(): ~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1506, in step self.dispatch_table[inst.opcode](self, inst) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1043, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 2896, in CALL_FUNCTION_EX self.call_function(fn, argsvars.items, kwargsvars) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1407, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/misc.py", line 1348, in call_function return self.obj.call_method(tx, self.name, list(args), kwargs) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/misc.py", line 300, in call_method return fn_vt.call_function(tx, [self.objvar] + args, kwargs) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/functions.py", line 812, in call_function return super().call_function(tx, args, kwargs) ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/functions.py", line 491, in call_function return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) # type: ignore[attr-defined] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1434, in inline_user_function_return return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/dynamo/symbolic_convert.py", line 5309, in inline_call return tracer.inline_call() ~~~~~~~~~~~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/dynamo/symbolic_convert.py", line 5530, in inline_call self.run() ~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run while self.step(): ~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1506, in step self.dispatch_table[inst.opcode](self, inst) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1043, in wrapper return inner_fn(self, inst) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 2896, in CALL_FUNCTION_EX self.call_function(fn, argsvars.items, kwargsvars) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/symbolic_convert.py", line 1407, in call_function self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/lazy.py", line 306, in realize_and_forward return getattr(self.realize(), name)(args, **kwargs) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/torch.py", line 2724, in call_function tensor_variable = wrap_fx_proxy( tx=tx, ...<4 lines>... ), ) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/builder.py", line 3117, in wrap_fx_proxy return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/builder.py", line 3192, in wrap_fx_proxy_cls out: VTTypeAlias = _wrap_fx_proxy( ~~~~~~~~~~~~~~^ target_cls, tx, proxy, example_value, subclass_type, **options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/variables/builder.py", line 3316, in _wrap_fx_proxy example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/utils.py", line 3751, in get_fake_value return _get_fake_value_impl(node, tx, allow_non_graph_fake) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/utils.py", line 3942, in _get_fake_value_impl _wrap_graph_break_with_torch_runtime_err( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ lambda: unimplemented( ^^^^^^^^^^^^^^^^^^^^^^ ...<5 lines>... ) ^ ) ^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/utils.py", line 3740, in _wrap_graph_break_with_torch_runtime_err raise exc.with_traceback(e.traceback) from None File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/utils.py", line 3737, in _wrap_graph_break_with_torch_runtime_err gb_fn() ~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/utils.py", line 3943, in <lambda> lambda: unimplemented( ~~~~~~~~~~~~~^ gb_type="RuntimeError when making fake tensor call", ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... from_exc=cause, ^^^^^^^^^^^^^^^ ) ^ File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/exc.py", line 653, in unimplemented raise Unsupported( msg, gb_type, skip_frame, real_stack=past_real_stack ) from from_exc torch._dynamo.exc.TorchRuntimeError: RuntimeError when making fake tensor call Explanation: Dynamo failed to run FX node with fake tensors: call_function <built-in method empty of type object at 0x7fa537172d90>((64,), **{'device': None, 'dtype': None}): got RuntimeError('tls_on_entry.has_value() INTERNAL ASSERT FAILED at "/__w/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp":53, please report a bug to PyTorch. ') Hint: Your code may result in an error when running in eager. Please double check that your code doesn't contain a similar error when actually running eager/uncompiled. You can do this by removing the torch.compile call, or by using torch.compiler.set_stance("force_eager").

Developer debug context:

For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb4315.html

from user code: File "/__w/pytorch/pytorch/test/functorch/test_eager_transforms.py", line 4472, in torch_dynamo_resume_in_test_update_batch_norm_at_4471 nn.BatchNorm2d( File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/_dynamo/polyfills/init.py", line 424, in instantiate_user_defined_class_object obj.init(*args, **kwargs) File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/nn/modules/batchnorm.py", line 166, in init super().init( File "/opt/conda/envs/py_3.14/lib/python3.14t/site-packages/torch/nn/modules/batchnorm.py", line 58, in init self.weight = Parameter(torch.empty(num_features, **factory_kwargs))

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

To execute this test, run the following from the base repo dir: PYTORCH_TEST_WITH_DYNAMO=1 python test/functorch/test_eager_transforms.py TestExamplesCorrectnessCPU.test_update_batch_norm_mechanism_make_functional_originally_track_running_stats_True_cpu

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_update_batch_norm_mechanism_make_functional_originally_track_running_stats_True_cpu is failing due to a TorchRuntimeError when making a fake tensor call, indicating a potential issue with the torch.compile call or the code's compatibility with eager mode.

Guidance

• Review the test code in functorch/test_eager_transforms.py to ensure it is compatible with eager mode and doesn't contain any errors that would cause a RuntimeError.
• Try removing the torch.compile call or setting torch.compiler.set_stance("force_eager") to see if the test passes in eager mode.
• Check the PyTorch documentation and the provided link to the graph break site (https://meta-pytorch.github.io/compile-graph-break-site/gb/gb4315.html) for more information on the error and potential solutions.
• Set TORCHDYNAMO_VERBOSE=1 and TORCH_LOGS="+dynamo" to get more detailed internal stack traces and logs for debugging purposes.

Example

No specific code example is provided as the issue is related to a complex test case and requires a deeper understanding of the PyTorch and TorchDynamo internals.

Notes

The error message suggests that the issue might be related to the torch.empty call in the BatchNorm2d initialization, but without more context or information about the test case, it's difficult to provide a more specific solution.

Recommendation

Apply a workaround by setting torch.compiler.set_stance("force_eager") or removing the torch.compile call to see if the test passes in eager mode, and then investigate further to find a permanent solution.