Issue Station
Sign In
Tools hereToolbox

transformers - ✅(Solved) Fix [BUG] AutoTokenizer fails to load FastSpeech2ConformerTokenizer [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
huggingface/transformers#44442•Fetched 2026-04-08 00:28:31
View on GitHub
Comments
0
Participants
1
Timeline
3
Reactions
0
Author
harshaljanjaniharshaljanjani
Participants
harshaljanjaniharshaljanjani
Timeline (top)
closed ×1cross-referenced ×1labeled ×1

Error Message

from transformers import AutoTokenizer

try: tokenizer = AutoTokenizer.from_pretrained("espnet/fastspeech2_conformer") print("Loaded tokenizer:", type(tokenizer)) except Exception as e: print(e)

Fix Action

Fixed

PR fix notes

PR #44443: fix(tokenizer): Only strip Fast from class names in AutoTokenizer if used as a suffix

Description (problem / solution / changelog)

What does this PR do?

The following failing use case was identified and fixed in this PR:

→ TOKENIZER_MAPPING_NAMES doesn't contain any class names where "Fast" appears in the middle of the name, and the canonical pattern is that "Fast" is appended to the end of the class name. But in the current version, the code uses .replace(.) (so if a model were named FastTextTokenizerFast, it would give TextTokenizer), which doesn't seem quite right. Resultingly in the current set of models, FastSpeech2ConformerTokenizer instantiation fails; this change fixes that by using endswith(.) instead, which should follow the convention more closely :)

cc: @Rocketknight1

Fixes #44442.

Current Output:

<img width="500" height="200" alt="Image" src="https://github.com/user-attachments/assets/532044d0-a936-4acc-ba22-e6b0525dd793" /><br>

Output After the Fix:

<img width="600" height="300" alt="4" src="https://github.com/user-attachments/assets/3dc2d496-23bf-4fda-8de8-94b177ebd697" />

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
  • Did you fix any necessary existing tests?

Changed files

  • src/transformers/models/auto/tokenization_auto.py (modified, +7/-7)

Code Example

from transformers import AutoTokenizer

try:
    tokenizer = AutoTokenizer.from_pretrained("espnet/fastspeech2_conformer")
    print("Loaded tokenizer:", type(tokenizer))
except Exception as e:
    print(e)
RAW_BUFFERClick to expand / collapse

System Info

  • transformers version: 5.0.0.dev0
  • Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
  • Python version: 3.12.3
  • huggingface_hub version: 1.3.2
  • safetensors version: 0.7.0
  • accelerate version: 1.12.0
  • Accelerate config: not installed
  • DeepSpeed version: not installed
  • PyTorch version (accelerator?): 2.9.1+cu128 (CUDA)
  • GPU type: NVIDIA L4
  • NVIDIA driver version: 550.90.07
  • CUDA version: 12.4

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer

try:
    tokenizer = AutoTokenizer.from_pretrained("espnet/fastspeech2_conformer")
    print("Loaded tokenizer:", type(tokenizer))
except Exception as e:
    print(e)

Trying to load espnet/fastspeech2_conformer tokenizer using AutoTokenizer crashes with a ValueError; the output I've attached below (feel free to reproduce) shows that AutoTokenizer is failing to instantiate a fast tokenizer backend.

Current Output:

<img width="600" height="300" alt="Image" src="https://github.com/user-attachments/assets/532044d0-a936-4acc-ba22-e6b0525dd793" />

Expected behavior

→ AutoTokenizer.from_pretrained("espnet/fastspeech2_conformer") should correctly instantiate and return a FastSpeech2ConformerTokenizer obj.

extent analysis

Fix Plan

Fix Name

Update transformers version to a stable release.

Steps

  1. Update transformers version: Run the following command to update the transformers version to a stable release (e.g., 4.25.1):

pip install transformers==4.25.1

2. **Verify the update**: Check the `transformers` version by running:
   ```python
import transformers
print(transformers.__version__)
  1. Re-run the reproduction script: Run the reproduction script again to verify that the issue is resolved:

from transformers import AutoTokenizer

try: tokenizer = AutoTokenizer.from_pretrained("espnet/fastspeech2_conformer") print("Loaded tokenizer:", type(tokenizer)) except Exception as e: print(e)

4. **Optional: Downgrade `transformers` version**: If the issue persists, try downgrading the `transformers` version to the previous stable release (e.g., `4.24.3`) and re-run the reproduction script.

## Verification
Verify that the `AutoTokenizer.from_pretrained` method correctly instantiates and returns a `FastSpeech2ConformerTokenizer` object without raising a `ValueError`.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

→ AutoTokenizer.from_pretrained("espnet/fastspeech2_conformer") should correctly instantiate and return a FastSpeech2ConformerTokenizer obj.

#api#ssr#installation#tensor shape#autograd error#database connection#vector store#embedding generation#cache error#pipeline error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING