from transformers import AutoTokenizer bert_tokenizer = AutoTokenizer.from_pretrained("vesteinn/ScandiBERT")

Traceback (most recent call last): File "<python-input-2>", line 1, in <module> bert_tokenizer = AutoTokenizer.from_pretrained("vesteinn/ScandiBERT") File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/transformers/models/auto/tokenization_auto.py", line 712, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/transformers/tokenization_utils_base.py", line 1712, in from_pretrained return cls._from_pretrained( ~~~~~~~~~~~~~~~~~~~~^ resolved_vocab_files, ^^^^^^^^^^^^^^^^^^^^^ ...<9 lines>... **kwargs, ^^^^^^^^^ ) ^ File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/transformers/tokenization_utils_base.py", line 1897, in _from_pretrained init_kwargs = cls.convert_to_native_format(**init_kwargs) File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/transformers/tokenization_utils_tokenizers.py", line 127, in convert_to_native_format if vocab and isinstance(vocab[0], (list, tuple)): ~~~~~^^^ KeyError: 0

Fix Plan

Downgrade Tokenizers

The issue is caused by a compatibility problem between transformers 5.2.0 and tokenizers 0.22.2. Downgrading tokenizers to 0.22.1 should resolve the issue.

Steps

1. Downgrade Tokenizers:

pip install --force-reinstall tokenizers==0.22.1

2. **Verify Downgrade**:
   Check the installed version of `tokenizers`:
   ```bash
pip show tokenizers

It should show tokenizers 0.22.1.

Alternative Solution

If downgrading tokenizers is not feasible, you can also try downgrading transformers to a version that is compatible with tokenizers 0.22.2. However, this may require further investigation to find the correct version.

Example Use Case

After downgrading tokenizers, you should be able to load the bert_tokenizer without any exceptions:

>>> from transformers import AutoTokenizer
>>> bert_tokenizer = AutoTokenizer.from_pretrained("vesteinn/ScandiBERT")

This should load the tokenizer without any errors.