transformers - 💡(How to fix) Fix Please update `tokenizers` version check [1 comments, 2 participants]

ImportError: tokenizers>=0.22.0,<=0.23.0 is required for a normal functioning of this module, but found tokenizers==0.23.1.

TL;DR

Update the transformers library to support tokenizers version 0.23.1 by modifying the version check in setup.py and dependency_versions_table.py.

Guidance

• Identify the lines of code in setup.py and dependency_versions_table.py that enforce the tokenizers version check and update them to allow 0.23.1.
• Verify that the updated transformers library works correctly with tokenizers version 0.23.1 by running tests and checking for any errors.
• Consider testing the updated library with different use cases to ensure that the changes do not introduce any regressions.
• If possible, submit a pull request to the transformers repository with the updated code to make the fix available to others.

Example

# In setup.py, update the line:
install_requires = [..., "tokenizers>=0.22.0,<=0.23.0",...]
# to:
install_requires = [..., "tokenizers>=0.22.0,<=0.23.1",...]

# In dependency_versions_table.py, update the line:
TOKENIZERS = ("tokenizers", ["0.22.0", "0.23.0"])
# to:
TOKENIZERS = ("tokenizers", ["0.22.0", "0.23.1"])

Notes

The fix assumes that the tokenizers version 0.23.1 is compatible with the transformers library. Thorough testing is necessary to ensure that the updated library works correctly.

Recommendation

Apply the workaround by updating the setup.py and dependency_versions_table.py files to support tokenizers version 0.23.1, as this will allow the use of the latest tokenizers version with the transformers library.