Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix DISABLED test_triggers (__main__.CallbackTests)

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Error Message

Traceback (most recent call last): File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1381, in _record static_outputs = model(inputs) File "/tmp/torchinductor_jenkins/tmput3rg409/7c/c7c2ibbwddbas2wnlaaualq6kain33v7hx4gakc4dd25z3hhkyys.py", line 229, in partition_0 triton_per_fused_sum_threshold_backward_0.run(le, tangents_1, buf0, buf2, 10, 10, stream=raw_stream0) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 2086, in run return fast(*args, stream=stream) File "<string>", line 5, in launcher RuntimeError: CUDA driver error: 901

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/dynamo/test_callback.py", line 121, in test_triggers loss.backward() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 623, in backward torch.autograd.backward( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/init.py", line 395, in backward _engine_run_backward( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 979, in _engine_run_backward return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 333, in apply_boxed return self._get_user_fn()(self, *args) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 3473, in backward return CompiledFunction._bwd_fn( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/subclass_codegen.py:codegen(compiled_function_backward)", line 21, in _compiled_backward File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/subclass_codegen.py:codegen(compiled_function_backward)", line 14, in impl_fn File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 3551, in _backward_impl return call_func_at_runtime_with_args( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args out = normalize_as_list(f(args)) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1446, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 763, in call return self.current_callable(inputs) File "/tmp/torchinductor_jenkins/tmput3rg409/7c/c7c2ibbwddbas2wnlaaualq6kain33v7hx4gakc4dd25z3hhkyys.py", line 259, in call (buf2, buf1) = self.partitions0 File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1925, in run return compiled_fn(new_inputs) # type: ignore[arg-type] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 440, in deferred_cudagraphify return fn(inputs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/utils.py", line 3815, in run out = model(new_inputs) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2178, in run out = self._run(new_inputs, function_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2372, in _run return self.record_function(new_inputs, function_id) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2412, in record_function node = CUDAGraphNode( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1133, in init self.recording_outputs: OutputType | None = self._record( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1366, in _record with ( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/graphs.py", line 477, in exit self.cuda_graph.capture_end() File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/graphs.py", line 169, in capture_end super().capture_end() torch.AcceleratorError: CUDA error: operation failed due to a previous error during capture Search for `hipErrorStreamCaptureInvalidated' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information. For more detailed error information, run with CUDA_LOG_FILE=stderr Device-side assertion tracking was not enabled by user.

To execute this test, run the following from the base repo dir: PYTORCH_TEST_WITH_ROCM=1 python test/dynamo/test_callback.py CallbackTests.test_triggers

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

Root Cause

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

Code Example

Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1381, in _record
    static_outputs = model(inputs)
  File "/tmp/torchinductor_jenkins/tmput3rg409/7c/c7c2ibbwddbas2wnlaaualq6kain33v7hx4gakc4dd25z3hhkyys.py", line 229, in partition_0
    triton_per_fused_sum_threshold_backward_0.run(le, tangents_1, buf0, buf2, 10, 10, stream=raw_stream0)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 2086, in run
    return fast(*args, stream=stream)
  File "<string>", line 5, in launcher
RuntimeError: CUDA driver error: 901

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/test/dynamo/test_callback.py", line 121, in test_triggers
    loss.backward()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 623, in backward
    torch.autograd.backward(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 395, in backward
    _engine_run_backward(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 979, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 333, in apply_boxed
    return self._get_user_fn()(self, *args)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 3473, in backward
    return CompiledFunction._bwd_fn(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/subclass_codegen.py:codegen(compiled_function_backward)", line 21, in _compiled_backward
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/subclass_codegen.py:codegen(compiled_function_backward)", line 14, in impl_fn
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 3551, in _backward_impl
    return call_func_at_runtime_with_args(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
    out = normalize_as_list(f(args))
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1446, in _fn
    return fn(*args, **kwargs)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 763, in __call__
    return self.current_callable(inputs)
  File "/tmp/torchinductor_jenkins/tmput3rg409/7c/c7c2ibbwddbas2wnlaaualq6kain33v7hx4gakc4dd25z3hhkyys.py", line 259, in call
    (buf2, buf1) = self.partitions[0](partition0_args)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1925, in run
    return compiled_fn(new_inputs)  # type: ignore[arg-type]
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 440, in deferred_cudagraphify
    return fn(inputs)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/utils.py", line 3815, in run
    out = model(new_inputs)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2178, in run
    out = self._run(new_inputs, function_id)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2372, in _run
    return self.record_function(new_inputs, function_id)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2412, in record_function
    node = CUDAGraphNode(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1133, in __init__
    self.recording_outputs: OutputType | None = self._record(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1366, in _record
    with (
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/graphs.py", line 477, in __exit__
    self.cuda_graph.capture_end()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/graphs.py", line 169, in capture_end
    super().capture_end()
torch.AcceleratorError: CUDA error: operation failed due to a previous error during capture
Search for `hipErrorStreamCaptureInvalidated' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
For more detailed error information, run with CUDA_LOG_FILE=stderr
Device-side assertion tracking was not enabled by user.

To execute this test, run the following from the base repo dir:
    PYTORCH_TEST_WITH_ROCM=1 python test/dynamo/test_callback.py CallbackTests.test_triggers

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
RAW_BUFFERClick to expand / collapse

Platforms: linux, rocm, slow

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

Over the past 6 hours, it has been determined flaky in 5 workflow(s) with 5 failures and 5 successes.

Debugging instructions (after clicking on the recent samples link): DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs. To find relevant log snippets:

  1. Click on the workflow logs linked above
  2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
  3. Grep for test_triggers
  4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.
<details><summary>Sample error message</summary> 
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1381, in _record
    static_outputs = model(inputs)
  File "/tmp/torchinductor_jenkins/tmput3rg409/7c/c7c2ibbwddbas2wnlaaualq6kain33v7hx4gakc4dd25z3hhkyys.py", line 229, in partition_0
    triton_per_fused_sum_threshold_backward_0.run(le, tangents_1, buf0, buf2, 10, 10, stream=raw_stream0)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 2086, in run
    return fast(*args, stream=stream)
  File "<string>", line 5, in launcher
RuntimeError: CUDA driver error: 901

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/test/dynamo/test_callback.py", line 121, in test_triggers
    loss.backward()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 623, in backward
    torch.autograd.backward(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 395, in backward
    _engine_run_backward(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 979, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 333, in apply_boxed
    return self._get_user_fn()(self, *args)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 3473, in backward
    return CompiledFunction._bwd_fn(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/subclass_codegen.py:codegen(compiled_function_backward)", line 21, in _compiled_backward
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/subclass_codegen.py:codegen(compiled_function_backward)", line 14, in impl_fn
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 3551, in _backward_impl
    return call_func_at_runtime_with_args(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
    out = normalize_as_list(f(args))
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1446, in _fn
    return fn(*args, **kwargs)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 763, in __call__
    return self.current_callable(inputs)
  File "/tmp/torchinductor_jenkins/tmput3rg409/7c/c7c2ibbwddbas2wnlaaualq6kain33v7hx4gakc4dd25z3hhkyys.py", line 259, in call
    (buf2, buf1) = self.partitions[0](partition0_args)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1925, in run
    return compiled_fn(new_inputs)  # type: ignore[arg-type]
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 440, in deferred_cudagraphify
    return fn(inputs)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/utils.py", line 3815, in run
    out = model(new_inputs)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2178, in run
    out = self._run(new_inputs, function_id)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2372, in _run
    return self.record_function(new_inputs, function_id)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2412, in record_function
    node = CUDAGraphNode(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1133, in __init__
    self.recording_outputs: OutputType | None = self._record(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1366, in _record
    with (
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/graphs.py", line 477, in __exit__
    self.cuda_graph.capture_end()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/graphs.py", line 169, in capture_end
    super().capture_end()
torch.AcceleratorError: CUDA error: operation failed due to a previous error during capture
Search for `hipErrorStreamCaptureInvalidated' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
For more detailed error information, run with CUDA_LOG_FILE=stderr
Device-side assertion tracking was not enabled by user.

To execute this test, run the following from the base repo dir:
    PYTORCH_TEST_WITH_ROCM=1 python test/dynamo/test_callback.py CallbackTests.test_triggers

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
</details>

Test file path: dynamo/test_callback.py

For all disabled tests (by GitHub issue), see https://hud.pytorch.org/disabled.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @chauhang @amjames @jataylo @azahed98

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING