pytorch - 💡(How to fix) Fix Long GPU queue (g5, g6) on LF fleet

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Error Message

Error looks like

Root Cause

TBD, cc @ezyang @gchanan @kadeng @msaroufim @zxiiro

Fix Action

Mitigation

  1. Attempt to route traffic back to Meta fleet temporarily on https://github.com/pytorch/test-infra/issues/5132
  2. Disable auto revert
  3. Cancel all running or queueing periodic workflows it issues
  4. Restore traffic back to LF fleet
RAW_BUFFERClick to expand / collapse

Current Status

Ongoing

Error looks like

Queue jobs

Incident timeline (all times pacific)

May 8th

User impact

PR are taking longer to land or needs to be forge merged

Root cause

TBD, cc @ezyang @gchanan @kadeng @msaroufim @zxiiro

Mitigation

  1. Attempt to route traffic back to Meta fleet temporarily on https://github.com/pytorch/test-infra/issues/5132
  2. Disable auto revert
  3. Cancel all running or queueing periodic workflows it issues
  4. Restore traffic back to LF fleet

Prevention/followups

TBD

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

pytorch - 💡(How to fix) Fix Long GPU queue (g5, g6) on LF fleet