Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix [Inductor]T5 models performance dropped 10% on inductor mode after pattern match changes [1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#177118•Fetched 2026-04-08 00:22:10
View on GitHub
Comments
1
Participants
2
Timeline
62
Reactions
0
Author
weishi-dengweishi-deng
Participants
eellisoneellisonweishi-dengweishi-deng
Assignees
shunting314shunting314
Timeline (top)
mentioned ×25subscribed ×25labeled ×8assigned ×1
RAW_BUFFERClick to expand / collapse

🐛 Describe the bug

T5 models' performance dropped 10% in inductor mode after PR https://github.com/pytorch/pytorch/commit/d50721ab12b534dd1bc90060791add4bf7ff0575 before: 0.077 after: 0.084 on cuda（device: 4080). TORCH_COMPILE_DEBUG logs show that the fusion kernel of "triton_per_fused__softmax__to_copy_add_arange_embedding_exp_le_mul_native_dropout_permute_prepare_softmax_online_repeat_rsub_sub_unsqueeze_view_16" have been divided into 2 kernels: "triton_red_fused__to_copy_add_mul_permute_prepare_softmax_online_rsub_unsqueeze_view_12" + "triton_poi_fused__softmax__to_copy_add_exp_mul_native_dropout_permute_sub_unsqueeze_view_13"

reproducer: python benchmarks/dynamo/huggingface.py --training --amp --amp-dtype float16 -d cuda -n10 --performance --disable-cudagraphs --only T5ForConditionalGeneration --cold-start-latency --backend=inductor

Versions

torch 2.11.0.dev20260105+cu128 triton 3.6.0+git9844da95

cc @jerryzh168 @ptrblck @msaroufim @eqy @tinglvv @nWEIdia @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @aakhundov @coconutruben @jataylo

extent analysis

Fix Plan

1. Downgrade to Previous Fusion Kernel

The performance drop is likely due to the new fusion kernel introduced in the PR. We can try downgrading to the previous kernel by setting the TRITON_FUSION_KERNELS environment variable.

export TRITON_FUSION_KERNELS=triton_per_fused__softmax__to_copy_add_arange_embedding_exp_le_mul_native_dropout_permute_prepare_softmax_online_repeat_rsub_sub_unsqueeze_view_16

2. Disable Fusion

If the above step doesn't work, we can try disabling fusion altogether by setting the TRITON_FUSION environment variable to 0.

export TRITON_FUSION=0

3. Update to Latest Triton Version

If the above steps don't work, we can try updating to the latest Triton version, which may have fixed the issue.

pip install --upgrade triton

Verification

To verify that the fix worked, we can re-run the reproducer script and check if the performance has improved.

python benchmarks/dynamo/huggingface.py --training --amp --amp-dtype float16 -d cuda -n10 --performance --disable-cudagraphs --only T5ForConditionalGeneration --cold-start-latency --backend=inductor

We can also check the TORCH_COMPILE_DEBUG logs to see if the fusion kernel has been divided into two kernels.

Extra Tips

  • Make sure to clean up any environment variables set in the previous steps before trying a new fix.
  • If you're using a virtual environment, make sure to activate it before trying the fixes.
  • If you're still experiencing issues, try reaching out to the Triton community or the PyTorch team for further assistance.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#inference speed#output truncation#response parsing#generation error#database connection

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING