transformers - 💡(How to fix) Fix [Bug] extra_special_tokens as list in tokenizer_config.json causes AttributeError when loading gemma4 (and other v5-format models) with transformers v4 [1 comments, 2 participants]

- transformers version: 4.x (any release before v5)
- Platform: macOS (Apple Silicon) / Linux
- Python version: 3.12 / 3.13
- PyTorch version: latest
- Model: google/gemma-4-E4B-it

---

pip install transformers  # installs v4.x

---

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-E4B-it")

---

File .../transformers/tokenization_utils_base.py:1181
    self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys())
AttributeError: 'list' object has no attribute 'keys'

---

"extra_special_tokens": ["<token_a>", "<token_b>"]

---

def _set_model_specific_special_tokens(self, special_tokens):
    if isinstance(special_tokens, list):
        # v5-style list format — nothing to register, skip
        return
    self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys())
    for key, value in special_tokens.items():
        ...

---

import json
from pathlib import Path
from huggingface_hub import hf_hub_download

config_path = Path(hf_hub_download("google/gemma-4-E4B-it", "tokenizer_config.json"))
with open(config_path) as f:
    config = json.load(f)
if isinstance(config.get("extra_special_tokens"), list):
    config["extra_special_tokens"] = {}
    with open(config_path, "w") as f:
        json.dump(config, f)

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-E4B-it")

---

pip install transformers  # installs v4.x

---

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-E4B-it")