Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix [Dynamo] speech_transformer SIGSEGV on A10G in periodic benchmarks [1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#180741•Fetched 2026-04-19 15:03:53
View on GitHub
Comments
0
Participants
1
Timeline
38
Reactions
0
Author
NikhilAPatelNikhilAPatel
Participants
NikhilAPatelNikhilAPatel
Timeline (top)
subscribed ×17mentioned ×16labeled ×5

speech_transformer causes a SIGSEGV (exit code -11) during dynamo compilation on A10G GPUs in the periodic benchmark CI:

Run failed with return code: -11

The crash occurs during dynamo tracing, preceded by graph break warnings about Tensor.item() in the speech_transformer model:

W0418 09:24:29.944000 Graph break from Tensor.item(), consider setting:
    torch._dynamo.config.capture_scalar_outputs = True
  File ".../speech_transformer/utils/utils.py", line 108, in get_non_pad_mask
    non_pad_mask[i, input_lengths[i] :] = 0

Root Cause

speech_transformer causes a SIGSEGV (exit code -11) during dynamo compilation on A10G GPUs in the periodic benchmark CI:

Run failed with return code: -11

The crash occurs during dynamo tracing, preceded by graph break warnings about Tensor.item() in the speech_transformer model:

W0418 09:24:29.944000 Graph break from Tensor.item(), consider setting:
    torch._dynamo.config.capture_scalar_outputs = True
  File ".../speech_transformer/utils/utils.py", line 108, in get_non_pad_mask
    non_pad_mask[i, input_lengths[i] :] = 0

Code Example

Run failed with return code: -11

---

W0418 09:24:29.944000 Graph break from Tensor.item(), consider setting:
    torch._dynamo.config.capture_scalar_outputs = True
  File ".../speech_transformer/utils/utils.py", line 108, in get_non_pad_mask
    non_pad_mask[i, input_lengths[i] :] = 0
RAW_BUFFERClick to expand / collapse

Description

speech_transformer causes a SIGSEGV (exit code -11) during dynamo compilation on A10G GPUs in the periodic benchmark CI:

Run failed with return code: -11

The crash occurs during dynamo tracing, preceded by graph break warnings about Tensor.item() in the speech_transformer model:

W0418 09:24:29.944000 Graph break from Tensor.item(), consider setting:
    torch._dynamo.config.capture_scalar_outputs = True
  File ".../speech_transformer/utils/utils.py", line 108, in get_non_pad_mask
    non_pad_mask[i, input_lengths[i] :] = 0

Details

  • Affects dynamo_eager_torchbench, aot_eager_torchbench, dynamic_aot_eager_torchbench on CUDA (A10G)
  • The benchmark runner continues after the crash; subsequent models still run
  • Does not directly cause CI failure (the CI fails due to other model graph break regressions)
  • Likely OOM-related on A10G (22GB VRAM) or a C++ crash in the dynamo compiler

CI Impact

Non-blocking. The periodic benchmark framework handles individual model crashes gracefully.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @chauhang @amjames @Lucaskabela @jataylo @azahed98

extent analysis

TL;DR

Setting torch._dynamo.config.capture_scalar_outputs = True may prevent the SIGSEGV crash during dynamo compilation on A10G GPUs.

Guidance

  • Investigate the graph break warnings about Tensor.item() in the speech_transformer model to understand the root cause of the crash.
  • Consider setting torch._dynamo.config.capture_scalar_outputs = True to capture scalar outputs and potentially prevent the graph break.
  • Monitor the GPU memory usage to determine if the crash is indeed OOM-related, and adjust the model or GPU resources accordingly.
  • Review the C++ code in the dynamo compiler to check for potential crashes or memory leaks.

Example

No code snippet is provided as the issue does not contain sufficient information to create a minimal example.

Notes

The crash may be related to either an out-of-memory issue or a C++ crash in the dynamo compiler, and further investigation is needed to determine the root cause.

Recommendation

Apply workaround: Setting torch._dynamo.config.capture_scalar_outputs = True may prevent the crash, and it is a relatively safe change to make.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#generation error#database connection#vector store#embedding generation#cache error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

pytorch - 💡(How to fix) Fix [Dynamo] speech_transformer SIGSEGV on A10G in periodic benchmarks [1 participants]