Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix [dynamo] [torch.compile] sdpa become slower after torch._dynamo.run in dynamo benchmark when attention_mask is None [1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#177712•Fetched 2026-04-08 00:52:33
View on GitHub
Comments
0
Participants
1
Timeline
122
Reactions
0
Author
jianyizhjianyizh
Participants
jianyizhjianyizh
Timeline (top)
subscribed ×58mentioned ×57labeled ×6renamed ×1
RAW_BUFFERClick to expand / collapse

🐛 Describe the bug

python benchmarks/dynamo/huggingface.py --performance --bfloat16 -dcuda -n3 --no-skip --dashboard --batch-size 2 --training --total-partitions 1 --partition-id 0 --only AlbertForQuestionAnswering --backend=eager --timeout=7200 --disable-cudagraphs --export-profiler-trace I run this on 4080s for backend=eager or indcutor. In the exported trace, I can see it calls aten::_flash_attention_forward in eager run but calls aten::_efficient_attention_forward in the dynamo run. attention_mask changes from None to a tensor of all True

This will hurt performance, i.e. The total device time of sdpa bwd changes from 3.6ms to 6.4ms, fwd changes from 1.4ms to 2.0ms. Why we set attention_mask even it's None?

sdpa in trace marked as expected: <img width="577" height="389" alt="Image" src="https://github.com/user-attachments/assets/030f607d-00a5-4098-99f8-3f3f8436066f" />

sdpa in trace marked as actual: <img width="557" height="404" alt="Image" src="https://github.com/user-attachments/assets/49ce209f-aded-4e3b-8992-eaac07efb169" />

Versions

torch 2.12.0.dev20260317+cu128

cc @jerryzh168 @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @amjames @Lucaskabela @jataylo @drisspg @liangel-02 @howardzhang-cv

extent analysis

Fix Plan

To address the performance issue caused by the change in attention_mask from None to a tensor of all True, we need to modify the code to handle None values correctly.

Step-by-Step Solution

  • Identify the function call that sets attention_mask to a tensor of all True.
  • Modify the function to check if attention_mask is None before setting it.
  • If attention_mask is None, do not set it to a tensor of all True.

Example code snippet:

if attention_mask is not None:
    attention_mask = torch.tensor([True] * sequence_length)
else:
    # Do not set attention_mask if it is None
    pass

Alternatively, you can also modify the aten::_flash_attention_forward and aten::_efficient_attention_forward functions to handle None values correctly.

Code Changes

You can add a conditional statement to check if attention_mask is None before calling the attention functions:

if attention_mask is not None:
    output = aten::_flash_attention_forward(query, key, value, attention_mask)
else:
    output = aten::_flash_attention_forward(query, key, value, None)

Verification

To verify that the fix worked, you can run the benchmark script again and check the exported trace to see if the attention_mask is set correctly. You can also compare the performance metrics before and after the fix to ensure that the issue is resolved.

Extra Tips

  • Make sure to test the fix thoroughly to ensure that it does not introduce any new issues.
  • Consider adding a check to ensure that the attention_mask is not set to a tensor of all True unnecessarily, as this can impact performance.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#generation error#database connection#vector store#embedding generation#cache error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING