pytorch - 💡(How to fix) Fix DISABLED deit_tiny_patch16_224.fb_in1k fail_accuracy on ROCm dynamic_inductor_timm_training [1 comments, 2 participants]

TL;DR

Investigate and address the numerical accuracy regression specific to the ROCm platform for the deit_tiny_patch16_224.fb_in1k model.

Guidance

• Review the changes made to the model or the ROCm configuration since the last successful run to identify potential causes of the numerical accuracy regression.
• Compare the accuracy results on ROCm with those on CUDA and CPU to understand the discrepancy and potentially isolate the issue.
• Consider re-running the benchmark with increased precision or different numerical settings on the ROCm platform to see if it mitigates the accuracy failure.
• Investigate if there are any known issues or updates related to the ROCm version being used that could be contributing to the regression.

Notes

The issue seems to be specific to the ROCm platform, and the model passes accuracy checks on CUDA and CPU, indicating a potential platform-specific problem. Without more detailed information about the changes made or the specific ROCm version, it's challenging to provide a precise fix.

Recommendation

Apply workaround: Given the model's previous success on ROCm and its current success on other platforms, it's reasonable to suspect a temporary or configuration-related issue. Therefore, applying a workaround, such as adjusting numerical precision settings or updating the ROCm version if possible, might resolve the issue.