pytorch - 💡(How to fix) Fix DISABLED test_triton_error_precompilation_and_autotuning (__main__.TestMaxAutotuneAsyncPipelined) [4 comments, 1 participants]

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 59, in testPartExecutor yield File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3443, in wrapper method(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner return func(*args, **kwds) File "/var/lib/jenkins/workspace/test/inductor/test_max_autotune.py", line 5356, in test_triton_error_precompilation_and_autotuning test_aten_chosen() File "/var/lib/jenkins/workspace/test/inductor/test_max_autotune.py", line 5350, in test_aten_chosen out, code = run_and_get_code(compiled_fn, a, b, epilogue) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2629, in run_and_get_code result = fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1059, in compile_wrapper raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1 File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1069, in _compile_fx_inner raise InductorError(e, currentframe()).with_traceback( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1049, in _compile_fx_inner mb_compiled_graph = fx_codegen_and_compile( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1832, in fx_codegen_and_compile return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1593, in codegen_and_compile compiled_module = graph.compile_to_module() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2612, in compile_to_module return self._compile_to_module() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2618, in _compile_to_module self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2505, in codegen_with_cpp_wrapper return self.codegen() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2550, in codegen self._update_scheduler() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2544, in _update_scheduler self.scheduler = Scheduler(self.operations) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 3086, in init self._init(nodes) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 3189, in _init self.nodes = self.fuse_nodes(self.nodes) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 4037, in fuse_nodes nodes = self.fuse_nodes_once(nodes, is_reorder_round=False) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 5000, in fuse_nodes_once self._try_fusion_pairs( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 4885, in _try_fusion_pairs fusion_res = self.speedup_by_fusion(node1, node2) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 4458, in speedup_by_fusion choice_timings = multi_node.choice_timings() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 5656, in choice_timings self._choice_timings[hint_override] = self._choice_timings_fn(hint_override) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/select_algorithm.py", line 3880, in get_timings timings = self.do_autotuning( File "/var/lib/jenkins/workspace/test/inductor/test_max_autotune.py", line 5335, in mock_do_autotuning raise NoValidChoicesError("Simulated: all Triton choices failed") torch._inductor.exc.InductorError: NoValidChoicesError: Simulated: all Triton choices failed

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

To execute this test, run the following from the base repo dir: python test/inductor/test_max_autotune.py TestMaxAutotuneAsyncPipelined.test_triton_error_precompilation_and_autotuning

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

TL;DR

The test test_triton_error_precompilation_and_autotuning is failing due to a NoValidChoicesError exception, indicating that all Triton choices failed during autotuning.

Guidance

• To debug this issue, set TORCHDYNAMO_VERBOSE=1 to get the internal stack trace and TORCH_LOGS="+dynamo" for more developer context.
• Run the test with the provided command python test/inductor/test_max_autotune.py TestMaxAutotuneAsyncPipelined.test_triton_error_precompilation_and_autotuning to reproduce the error.
• Investigate the mock_do_autotuning method in test_max_autotune.py to understand why it's raising a NoValidChoicesError.
• Check the workflow logs linked in the issue to find relevant log snippets by grepping for test_triton_error_precompilation_and_autotuning.

Example

No code snippet is provided as the issue is related to a specific test failure and requires further investigation.

Notes

The issue lacks information about the specific changes or updates that might have caused this test failure. Further investigation is needed to determine the root cause.

Recommendation

Apply workaround: Investigate and fix the mock_do_autotuning method to handle the NoValidChoicesError exception properly, or update the test to avoid this error.