pytorch - 💡(How to fix) Fix DISABLED test_empty_conv_input_search_space_DEFAULT_kernel_size_1 (__main__.TestMaxAutotuneAsyncPipelined) [1 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 59, in testPartExecutor yield File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3443, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/mock.py", line 1379, in patched return func(*newargs, **newkeywargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 589, in instantiated_test test(self, **param_kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner return func(*args, **kwds) File "/var/lib/jenkins/workspace/test/inductor/test_max_autotune.py", line 1201, in test_empty_conv_input act = opt_f(x, weight) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1059, in compile_wrapper raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1 File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1069, in _compile_fx_inner raise InductorError(e, currentframe()).with_traceback( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1049, in _compile_fx_inner mb_compiled_graph = fx_codegen_and_compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1832, in fx_codegen_and_compile return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1593, in codegen_and_compile compiled_module = graph.compile_to_module() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2612, in compile_to_module return self._compile_to_module() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2618, in _compile_to_module self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2505, in codegen_with_cpp_wrapper return self.codegen() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2561, in codegen result = self.wrapper_code.generate(self.is_inference) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codegen/cpp_wrapper_gpu.py", line 923, in generate return super().generate(is_inference) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codegen/cpp_wrapper_cpu.py", line 1044, in generate return super().generate(is_inference) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codegen/wrapper.py", line 2054, in generate return self._generate(is_inference) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codegen/wrapper.py", line 2100, in _generate line.codegen(self.wrapper_call) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codegen/wrapper.py", line 621, in codegen kernel_name = node.get_kernel_name() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 6367, in get_kernel_name assert name is not None torch._inductor.exc.InductorError: AssertionError:

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

To execute this test, run the following from the base repo dir: python test/inductor/test_max_autotune.py TestMaxAutotuneAsyncPipelined.test_empty_conv_input_search_space_DEFAULT_kernel_size_1

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The most likely fix for the failing test is to investigate and address the InductorError: AssertionError exception in the torch/_inductor/ir.py file.

Guidance

1. Investigate the error: Set TORCHDYNAMO_VERBOSE=1 and TORCH_LOGS="+dynamo" to get more detailed internal stack traces and developer context.
2. Run the test locally: Execute the test using python test/inductor/test_max_autotune.py TestMaxAutotuneAsyncPipelined.test_empty_conv_input_search_space_DEFAULT_kernel_size_1 to reproduce the issue.
3. Check the kernel name: Verify that the node.get_kernel_name() function in torch/_inductor/ir.py returns a valid kernel name.
4. Review recent changes: Examine recent code changes and commits to identify potential causes of the error.
5. Check CI workflow logs: Follow the debugging instructions to find relevant log snippets and study the logs for clues.

Example

No specific code snippet is provided, but the error message suggests investigating the get_kernel_name() function in torch/_inductor/ir.py.

Notes

The error message indicates an AssertionError exception, which may be caused by an invalid or missing kernel name. The issue may be related to recent changes or commits, and reviewing the CI workflow logs and running the test locally can help identify the root cause.

Recommendation

Apply workaround: Set PYTORCH_PRINT_REPRO_ON_FAILURE=0 to suppress the error message and investigate the issue further.