transformers - 💡(How to fix) Fix transformers >= 5.0.0 fails loading tokenizer for EMBEDDIA/est-roberta [6 comments, 3 participants]

from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("EMBEDDIA/est-roberta") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Programmid\Miniconda3\envs\py312_transformers_problem\Lib\site-packages\transformers\models\auto\tokenization_auto.py", line 749, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Programmid\Miniconda3\envs\py312_transformers_problem\Lib\site-packages\transformers\tokenization_utils_base.py", line 1721, in from_pretrained return cls._from_pretrained( ^^^^^^^^^^^^^^^^^^^^^ File "C:\Programmid\Miniconda3\envs\py312_transformers_problem\Lib\site-packages\transformers\tokenization_utils_base.py", line 1910, in _from_pretrained tokenizer = cls(*init_inputs, **init_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Programmid\Miniconda3\envs\py312_transformers_problem\Lib\site-packages\transformers\models\camembert\tokenization_camembert.py", line 118, in init unk_index = next((i for i, (tok, _) in enumerate(self._vocab) if tok == str(unk_token)), 0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Programmid\Miniconda3\envs\py312_transformers_problem\Lib\site-packages\transformers\models\camembert\tokenization_camembert.py", line 118, in <genexpr> unk_index = next((i for i, (tok, _) in enumerate(self._vocab) if tok == str(unk_token)), 0) ^^^^^^^^ ValueError: too many values to unpack (expected 2)

Fix Plan

The issue arises from the tokenization_camembert.py file, which is not compatible with the EMBEDDIA/est-roberta model. To fix this, we need to use the correct tokenizer for the model.

• Check the model's documentation to find the correct tokenizer class.
• Use the AutoTokenizer with the use_fast parameter set to False to use the slow tokenizer.

from transformers import AutoTokenizer

# Load the tokenizer with use_fast=False
tokenizer = AutoTokenizer.from_pretrained("EMBEDDIA/est-roberta", use_fast=False)

Alternatively, you can try to use the BertTokenizer or RobertaTokenizer directly:

from transformers import BertTokenizer, RobertaTokenizer

# Load the tokenizer
tokenizer = RobertaTokenizer.from_pretrained("EMBEDDIA/est-roberta")

Verification

To verify that the fix worked, you can try to load the tokenizer and use it to encode a sentence:

input_text = "This is a test sentence."
inputs = tokenizer(input_text, return_tensors="pt")

print(inputs)

If the tokenizer loads successfully and encodes the sentence without errors, the fix has worked.

Extra Tips

• Make sure to check the model's documentation for the recommended tokenizer and configuration.
• If you're using a custom model, ensure that the tokenizer is compatible with the model's architecture.
• You can also try to update the transformers library to the latest version to see if the issue is resolved.