pytorch - 💡(How to fix) Fix DISABLED test_nan_assert_float8_e4m3fn (__main__.ProcessGroupNCCLGroupTest) [2 comments, 2 participants]

Fix Plan

Step 1: Identify the Flaky Test

• Locate the test file distributed/test_c10d_nccl.py and identify the specific test function test_nan_assert_float8_e4m3fn that is flaky.

Step 2: Analyze the Test Logs

• Follow the debugging instructions to analyze the test logs:
  1. Click on the workflow logs linked above
  2. Click on the Test step of the job so that it is expanded
  3. Grep for test_nan_assert_float8_e4m3fn
  4. Study the logs to identify the failure pattern

Step 3: Implement a Retry Mechanism

• Modify the test function to implement a retry mechanism using a library like tenacity:

import tenacity

@tenacity.retry(wait=tenacity.wait_exponential(multiplier=1, min=4, max=10))
def test_nan_assert_float8_e4m3fn():
    # test code here

• Adjust the retry settings as needed to suit the test's requirements.

Step 4: Update the Test Configuration

• Update the test configuration to enable the retry mechanism:

import pytest

@pytest.mark.flaky(reruns=3, reruns_delay=0.5)
def test_nan_assert_float8_e4m3fn():
    # test code here

• Adjust the reruns and reruns_delay settings as needed to suit the test's requirements.

Step 5: Verify the Fix

• Re-run the test in CI to verify that the fix works.
• Monitor the test logs to ensure that the retry mechanism is working correctly.