Traceback (most recent call last): File "/mnt/RAPID/tmp/repro_mistral_regex_bug.py", line 23, in <module> tokenizer = AutoTokenizer.from_pretrained( "mistralai/Mistral-Nemo-Instruct-2407", trust_remote_code=True, fix_mistral_regex=True, ) File "/root/miniconda3/envs/myconda/lib/python3.13/site-packages/transformers/models/auto/tokenization_auto.py", line 723, in from_pretrained return tokenizer_class_from_name(tokenizer_config_class).from_pretrained( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ pretrained_model_name_or_path, *inputs, **kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/root/miniconda3/envs/myconda/lib/python3.13/site-packages/transformers/tokenization_utils_base.py", line 1721, in from_pretrained return cls._from_pretrained( ~~~~~~~~~~~~~~~~~~~~^ resolved_vocab_files, ^^^^^^^^^^^^^^^^^^^^^ ...<9 lines>... **kwargs, ^^^^^^^^^ ) ^ File "/root/miniconda3/envs/myconda/lib/python3.13/site-packages/transformers/tokenization_utils_base.py", line 1910, in _from_pretrained tokenizer = cls(*init_inputs, **init_kwargs) File "/root/miniconda3/envs/myconda/lib/python3.13/site-packages/transformers/tokenization_utils_tokenizers.py", line 477, in init self._tokenizer = self._patch_mistral_regex( ~~~~~~~~~~~~~~~~~~~~~~~~~^ self._tokenizer, ^^^^^^^^^^^^^^^^ ...<3 lines>... **kwargs, ^^^^^^^^^ ) ^ File "/root/miniconda3/envs/myconda/lib/python3.13/site-packages/transformers/tokenization_utils_tokenizers.py", line 1363, in _patch_mistral_regex current_pretokenizer = tokenizer.backend_tokenizer.pre_tokenizer ^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'tokenizers.Tokenizer' object has no attribute 'backend_tokenizer'

Fix Plan

To fix the issue, you need to update the tokenization_utils_tokenizers.py file. Here are the steps:

• Open the tokenization_utils_tokenizers.py file located in your transformers library installation.
• Find the _patch_mistral_regex method.
• Update the line current_pretokenizer = tokenizer.backend_tokenizer.pre_tokenizer to current_pretokenizer = tokenizer.pre_tokenizer.
• Update any corresponding write-back lines that use .backend_tokenizer to use the object directly.

Example code snippet:

# Before
current_pretokenizer = tokenizer.backend_tokenizer.pre_tokenizer

# After
current_pretokenizer = tokenizer.pre_tokenizer

Make sure to update all occurrences of .backend_tokenizer in the _patch_mistral_regex method.

Verification

To verify that the fix worked, run your original code again:

import transformers

print(f"transformers version: {transformers.__version__}")
print("Loading Mistral tokenizer with fix_mistral_regex=True ...")

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "mistralai/Mistral-Nemo-Instruct-2407",
    trust_remote_code=True,
    fix_mistral_regex=True,
)

If the fix is successful, the code should run without raising any errors.

Extra Tips

• Make sure to update the transformers library to the latest version if possible.
• If you are using a virtual environment, ensure that the changes are made in the correct environment.
• If you are using a package manager like pip, you may need to reinstall the transformers library after making the changes.