PR #45199: fix(models): Resolve regressions in Wav2Vec2PhonemeCTCTokenizer (wav2vec2-lv-60-espeak-cv-ft)

• Repository: huggingface/transformers
• Author: harshaljanjani
• State: open | merged: False
• Link: https://github.com/huggingface/transformers/pull/45199

Description (problem / solution / changelog)

What does this PR do?

The following Wav2Vec2PhonemeCTC use cases were identified and fixed in this PR:

→ 05c0e1d ("rm slow tokenizers") added self.backend = kwargs.pop("backend", None). Wav2Vec2PhonemeCTCTokenizer already used self.backend for its phonemizer EspeakBackend object set in init_backend. Regardless of call order, one clobbers the other; either the base class overwrites the phonemizer object with None (breaking phonemize()), or the phonemizer object overwrites the base class's serializable value (breaking save_pretrained with EspeakBackend is not JSON serializable). Renamed to self._phonemizer_backend so both attributes coexist. Followed the same naming convention used for _word_delimiter_token and _phone_delimiter_token in the same file. → Same commit consolidated tokenization_utils.py into tokenization_python.py. In the old code, _encode_plus had return_offsets_mapping as a named param and raised NotImplementedError before reaching tokenize(). After the refactor, return_offsets_mapping is no longer a named param in _encode_plus, so it flows through **kwargs → tokenize() → prepare_for_tokenization(), which had a fixed signature. Added **kwargs to match the base class contract at tokenization_python.py#L836-L838. No other models are affected; Wav2Vec2PhonemeCTCTokenizer is the only override of prepare_for_tokenization that was missing **kwargs :) → For more details on reproducing the bug and the output screenshots, please visit the linked issue!

Fixes #45198

cc: @Rocketknight1 @itazap

CI coverage fixed by this PR (as suggested for inclusion in the PR):

CI run test coverage of this behavior:

→ models/wav2vec2_phoneme/test_tokenization_wav2vec2_phoneme.py::Wav2Vec2PhonemeCTCTokenizerTest:

test_batch_encode_dynamic_overflowing, test_batch_encode_plus_batch_sequence_length, test_batch_encode_plus_padding, test_call, test_case_insensitive, test_change_phonemizer_lang, test_chat_template, test_chat_template_batched, test_decode_with_del, test_empty_input_string, test_encode, test_encode_basic_padding, test_encode_decode, test_encode_decode_with_del, test_encode_decode_with_del_filter, test_encode_plus_with_padding_0, test_encode_plus_with_padding_1, test_encode_with_del, test_mask_output, test_maximum_encoding_length_pair_input, test_maximum_encoding_length_single_input, test_number_of_added_tokens, test_offsets, test_offsets_batch, test_padding_to_multiple_of, test_phonemize, test_phonemize_with_word_del, test_prepare_seq2seq_batch, test_pretokenized_inputs, test_right_and_left_truncation, test_save_and_load_tokenizer, test_special_tokens_mask, test_special_tokens_mask_input_pairs, test_token_type_ids, test_tokenizer_add_new_tokens

The remaining post-fix failures tmk are not caused by Wav2Vec2PhonemeCTCTokenizer but originate from the tokenization_python.py base class changes (phone delimiter token no longer inserted between phoneme tokens during encoding, added/special token serialization changes, and TypeError in offset computation). These would need fixes in the base class, not in the model tokenizer :)

Repro output after the fixes (feel free to cross-check):

Code Agent Policy

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• Did you fix any necessary existing tests?

Changed files

• src/transformers/models/wav2vec2_phoneme/tokenization_wav2vec2_phoneme.py (modified, +3/-2)