transformers - 💡(How to fix) Fix transformers>=5.1.0 fails when loading physical-intelligence/fast [2 comments, 3 participants]

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "../lib/python3.10/site-packages/transformers/models/auto/processing_auto.py", line 394, in from_pretrained
    return processor_class.from_pretrained(
  File "../lib/python3.10/site-packages/transformers/processing_utils.py", line 1402, in from_pretrained
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, processor_dict, **kwargs)
  File "../lib/python3.10/site-packages/transformers/processing_utils.py", line 1516, in _get_arguments_from_pretrained
    tokenizer = cls._load_tokenizer_from_pretrained(
  File "../lib/python3.10/site-packages/transformers/processing_utils.py", line 1469, in _load_tokenizer_from_pretrained
    tokenizer = auto_processor_class.from_pretrained(
  File "../lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 725, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "../lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1712, in from_pretrained
    return cls._from_pretrained(
  File "../lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1900, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "../lib/python3.10/site-packages/transformers/tokenization_utils_tokenizers.py", line 274, in __init__
    raise ValueError(
ValueError: Couldn't instantiate the backend tokenizer from one of: 
(1) a `tokenizers` library serialization file, 
(2) a slow tokenizer instance to convert or 
(3) an equivalent slow tokenizer class to instantiate and convert. 
You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.

2. **Specify `use_fast` Parameter**: When loading the `AutoProcessor`, try setting `use_fast=False` explicitly to avoid any potential issues with fast tokenizers:
   ```python
from transformers import AutoProcessor
p = AutoProcessor.from_pretrained("../physical-intelligence/fast", trust_remote_code=True, use_fast=False)

### Verification
To verify that the fix worked, successfully execute the code without encountering the `ValueError`. You should be able to load the `AutoProcessor` or the tokenizer and model without any errors.

### Extra Tips
- Ensure your environment is clean and managed (e.g., using `venv` or `conda`) to avoid version conflicts.
- If you're working in a team or planning to deploy your model, consider adding the required libraries and versions to your project's `requirements.txt` to ensure reproducibility.
- Always check the Hugging Face model hub for the most current information on model compatibility and requirements.