from transformers import MLukeTokenizer

try: tokenizer = MLukeTokenizer.from_pretrained( "studio-ousia/mluke-base", task="entity_classification" ) sentence = "Japanese is an East Asian language spoken by about 128 million people, primarily in Japan." span = (15, 34) encoding = tokenizer(sentence, entity_spans=[span]) print(tokenizer.decode(encoding["input_ids"], spaces_between_special_tokens=False)) except Exception as e: print(e)

Problem Summary

The issue is caused by a deprecated attribute in the MLukeTokenizer class, leading to an AttributeError when trying to perform entity classification.

Root Cause Analysis

The root cause is a naming conflict between the new extra_special_tokens attribute and the old additional_special_tokens attribute in the MLukeTokenizer class.

Fix Plan

To fix this issue, you need to update the MLukeTokenizer class to use the new attribute name.

Step-by-Step Solution

1. Update the MLukeTokenizer class:

   • Open the tokenization_mluke.py file in the transformers repository.
   • Replace all occurrences of self.additional_special_tokens_ids with self.extra_special_tokens_ids.
   • Commit the changes and push them to your fork.

2. Update your code:

   • Open your modified script.
   • Replace tokenizer = MLukeTokenizer.from_pretrained(...) with tokenizer = MLukeTokenizer.from_pretrained(..., use_auth_token=True).
   • This will ensure that you're using the updated MLukeTokenizer class.

Example Code

from transformers import MLukeTokenizer

try:
    tokenizer = MLukeTokenizer.from_pretrained(
      "studio-ousia/mluke-base", task="entity_classification", use_auth_token=True
    )
    sentence = "Japanese is an East Asian language spoken by about 128 million people, primarily in Japan."
    span = (15, 34)
    encoding = tokenizer(sentence, entity_spans=[span])
    print(tokenizer.decode(encoding["input_ids"], spaces_between_special_tokens=False))
except Exception as e:
    print(e)

Verification

To verify that the fix worked, run the script again and check that the output is as expected. The AttributeError should be resolved, and the entity classification task should pass without any issues