Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix [CI[B200] Smoke test encounters CUDA Unknown error for `dgxb200-03` and `dgxb200-04` [7 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#180376•Fetched 2026-04-16 06:34:57
View on GitHub
Comments
7
Participants
2
Timeline
51
Reactions
0
Author
nWEIdianWEIdia
Participants
huydhnhuydhnnWEIdianWEIdia
Assignees
crcrparcrcrpar
Timeline (top)
subscribed ×30mentioned ×8commented ×7labeled ×4

Error Message

Otherwise the runners are producing cuda unknown error anyways. dgxb200-04-1005 this job,  CUDA unknown error dgxb200-03-1006  this job, CUDA unknown error

RAW_BUFFERClick to expand / collapse

🐛 Describe the bug

This issue is created as fyi that we would be doing maintenance immediately on these two nodes. Jobs would be failing. Otherwise the runners are producing cuda unknown error anyways. Detailed links below:

dgxb200-04-1005 this job,  CUDA unknown error dgxb200-03-1006  this job, CUDA unknown error

cc @seemethere @malfet @pytorch/pytorch-dev-infra @ptrblck @eqy @atalman @huydhn @tinglvv

Versions

Unlikely a pytorch issue

extent analysis

TL;DR

Maintenance on the nodes dgxb200-04-1005 and dgxb200-03-1006 may resolve the cuda unknown error issue.

Guidance

  • Investigate the system logs for the nodes dgxb200-04-1005 and dgxb200-03-1006 to identify potential hardware or configuration issues.
  • Verify the CUDA installation and configuration on the affected nodes to ensure it is correct and up-to-date.
  • Check the job configurations for the linked jobs to see if there are any specific requirements or settings that could be contributing to the error.

Notes

The issue seems to be related to the nodes rather than a PyTorch issue, as stated in the "Versions" section.

Recommendation

Apply workaround: Perform maintenance on the nodes dgxb200-04-1005 and dgxb200-03-1006, as this may resolve the underlying issue causing the cuda unknown error.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#environment setup#docker error#permission error#memory optimization#batch processing

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING