Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix DISABLED test_nan_assert_float8_e5m2 (__main__.ProcessGroupNCCLGroupTest) [1 comments, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#176975•Fetched 2026-04-08 00:23:22
View on GitHub
Comments
1
Participants
1
Timeline
76
Reactions
0
Author
pytorch-bot[bot]pytorch-bot[bot]
Participants
pytorch-bot[bot]pytorch-bot[bot]
Timeline (top)
mentioned ×36subscribed ×36labeled ×3commented ×1

Root Cause

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

RAW_BUFFERClick to expand / collapse

Platforms: linux

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

Over the past 6 hours, it has been determined flaky in 5 workflow(s) with 0 failures and 5 successes.

Debugging instructions (after clicking on the recent samples link): DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs. To find relevant log snippets:

  1. Click on the workflow logs linked above
  2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
  3. Grep for test_nan_assert_float8_e5m2
  4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.

Test file path: distributed/test_c10d_nccl.py

For all disabled tests (by GitHub issue), see https://hud.pytorch.org/disabled.

cc @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci @aditvenk @xmfan

extent analysis

Fix Plan

Fix Name: Flaky Test Fix for test_nan_assert_float8_e5m2

Step-by-Step Solution:

1. Identify the root cause of flakiness

  • Review the workflow logs and grep for test_nan_assert_float8_e5m2 to identify the specific instances that failed or succeeded.
  • Analyze the logs to determine the root cause of the flakiness.

2. Modify the test to handle flaky conditions

  • In the distributed/test_c10d_nccl.py file, modify the test_nan_assert_float8_e5m2 test to handle flaky conditions.
  • Use a retry mechanism to re-run the test until it succeeds.

Example Code:

import time

def retry_test(func):
    max_retries = 3
    for i in range(max_retries):
        try:
            func()
            break
        except Exception as e:
            if i < max_retries - 1:
                time.sleep(1)  # wait for 1 second before retrying
            else:
                raise e

@retry_test
def test_nan_assert_float8_e5m2():
    # original test code here
    pass

3. Update the test to use a flaky test framework

  • Consider using a flaky test framework like pytest-flaky to handle flaky tests.
  • Update the test to use the flaky test framework.

Example Code:

import pytest

@pytest.mark.flaky(reruns=3, reruns_delay=1)
def test_nan_assert_float8_e5m2():
    # original test code here
    pass

4. Verify the fix

  • Run the test again in CI to verify that it no longer fails.
  • Review the workflow logs to ensure that the test is no longer flaky.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#GPU setup#container setup#orchestration issue#cache issue#memory leak

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

pytorch - 💡(How to fix) Fix DISABLED test_nan_assert_float8_e5m2 (__main__.ProcessGroupNCCLGroupTest) [1 comments, 1 participants]