pytorch - 💡(How to fix) Fix [CUDA][B200][CI Test] Smoke Test Jobs Timing Out (3.5hrs+) [3 comments, 1 participants]

Fix Plan

The fix involves increasing the timeout limit for the B200 Smoke Test Periodic Jobs to prevent non-deterministic time-outs.

Steps to Fix

• Increase the timeout limit in the GitHub Actions workflow file:
  • Locate the timeout parameter in the workflow file (e.g., .github/workflows/b200_smoke_test.yml)
  • Update the timeout value to a higher limit (e.g., timeout: 4h instead of timeout: 2h)
• Example code snippet:

jobs:
  b200_smoke_test:
    runs-on: ubuntu-latest
    timeout: 4h  # Increased timeout limit
    steps:
      # ... rest of the workflow steps ...

• Commit and push the changes to the repository to update the workflow configuration.

Verification

• Verify that the updated workflow runs without timing out by checking the GitHub Actions job logs.
• Monitor the HUD dashboard to ensure that the B200 Smoke Test Periodic Jobs are completing successfully without time-outs.

Extra Tips

• Consider implementing a retry mechanism for the job to handle intermittent failures.
• Review the job's resource utilization to ensure it's not exceeding the allocated resources, which could contribute to time-outs.