transformers - 💡(How to fix) Fix [Bug] FP8 save_pretrained moe [3 comments, 3 participants]

Fix Plan

1. Update transformers library to the latest version

Ensure you're running the latest version of the transformers library, which may have fixed the issue.

pip install --upgrade transformers

2. Use save_pretrained with use_cache=False

The save_pretrained method can sometimes cause issues when saving large models. Try setting use_cache=False to see if it resolves the problem.

model.save_pretrained('/root/test', safe_serialization=True, max_shard_size='5GB', use_cache=False)

3. Check for Out-of-Memory (OOM) errors

If the model is too large, it may cause OOM errors when saving. Try reducing the max_shard_size or using a more efficient storage solution.

model.save_pretrained('/root/test', safe_serialization=True, max_shard_size='2GB')

4. Verify the issue is resolved

Run the script again and check if the issue persists.

# Run the script with the updated parameters
python script.py

Verification

• Run the script with the updated parameters and check if the issue is resolved.
• Verify that the model is saved successfully without any errors.

Extra Tips

• Always use the latest version of the transformers library to ensure you have the latest bug fixes and features.
• Be cautious when working with large models, as they can cause Out-of-Memory (OOM) errors.
• Consider using a more efficient storage solution, such as torch.save or joblib, to save large models.