Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix DISABLED test_nan_assert_float32 (__main__.ProcessGroupNCCLGroupTest) [2 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#177006•Fetched 2026-04-08 00:22:58
View on GitHub
Comments
2
Participants
2
Timeline
59
Reactions
0
Author
pytorch-bot[bot]pytorch-bot[bot]
Participants
pytorch-bot[bot]pytorch-bot[bot]wwwjnwwwjn
Timeline (top)
mentioned ×27subscribed ×27labeled ×3commented ×2

Root Cause

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

RAW_BUFFERClick to expand / collapse

Platforms: linux

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

Over the past 6 hours, it has been determined flaky in 4 workflow(s) with 0 failures and 4 successes.

Debugging instructions (after clicking on the recent samples link): DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs. To find relevant log snippets:

  1. Click on the workflow logs linked above
  2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
  3. Grep for test_nan_assert_float32
  4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.

Test file path: distributed/test_c10d_nccl.py

For all disabled tests (by GitHub issue), see https://hud.pytorch.org/disabled.

cc @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci @aditvenk @xmfan

extent analysis

Fix Plan

1. Identify and Fix the Flaky Test

The test test_nan_assert_float32 is flaky and needs to be fixed.

2. Review Test Code

Review the test code in distributed/test_c10d_nccl.py to identify the potential causes of flakiness.

3. Implement a Retry Mechanism

Implement a retry mechanism in the test to handle temporary failures.

import torch
import time

def test_nan_assert_float32():
    # Test code here
    try:
        # Test code that may fail
        torch.randn(1).nan()
    except Exception as e:
        # Retry the test up to 3 times
        for _ in range(3):
            try:
                torch.randn(1).nan()
                break
            except Exception as e:
                time.sleep(1)  # Wait for 1 second before retrying
        else:
            # If all retries fail, raise the exception
            raise

4. Update the Test to Use a More Robust Assertion

Update the test to use a more robust assertion that can handle temporary failures.

import torch

def test_nan_assert_float32():
    # Test code here
    assert torch.randn(1).isnan().any(), "Expected NaN value"

5. Verify the Fix

Verify that the fix works by running the test multiple times and checking that it passes consistently.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#parallel task#integration issue#index setup#retrieval issue#search optimization

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING