Issue Station
Sign In
Tools hereToolbox

transformers - ✅(Solved) Fix CLIPTokenizer uses 10**30 as `model_max_length` [1 pull requests]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Error Message

Traceback (most recent call last): File "/home/oleg.konin/mlapi-glm/test.py", line 3, in <module> tok1 = CLIPTokenizer.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", subfolder="tokenizer", local_files_only=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained return cls._from_pretrained( ^^^^^^^^^^^^^^^^^^^^^ File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained tokenizer = cls(*init_inputs, **init_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/models/clip/tokenization_clip.py", line 306, in init with open(vocab_file, encoding="utf-8") as vocab_handle: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: expected str, bytes or os.PathLike object, not NoneType

Fix Action

Fixed

PR fix notes

PR #45541: Fix local_files_only tokenizer fallback when tokenizer files are missing (Issue 45538)

Description (problem / solution / changelog)

What does this PR do?

<!-- Congratulations! You've made it this far! You're not quite done yet though. Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution. Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change. Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost. -->

Fixes https://github.com/huggingface/transformers/issues/45538

Root Cause

from_pretrained(..., local_files_only=True) let missing tokenizer files resolve to None and still proceeded into _from_pretrained(...). That allowed a stub tokenizer to initialize instead of raising, which led to the bogus huge model_max_length.

Describe the Fix

The fix adds a fail-closed check in PreTrainedTokenizerBase.from_pretrained.

After resolving files, it now checks whether all real tokenizer assets for that class resolved to None. If they did, it raises the existing Can't load tokenizer... OSError instead of continuing into _from_pretrained(...) and constructing a stub tokenizer

Local Tests

  • Check with BertTokenizer and CLIPTokenizer using a nonexistent model id and local_files_only=True. Before the fix, they loaded unexpectedly with the huge fallback max length. After the fix, they raised the expected OSError.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by code agents. We are currently bottlenecked by our ability to review and respond to them. As a result, we ask that new users do not submit pure code agent PRs at this time. You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result, this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

  • I confirm that this is not a pure code agent PR.

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@ArthurZucker @Cyrilvallez

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @ If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**. Please tag fewer than 3 people. Models: - text models: @ArthurZucker @Cyrilvallez - vision models: @yonigozlan @molbap - audio models: @eustlb @ebezzam @vasqu - multimodal models: @zucchini-nlp - graph models: @clefourrier Library: - generate: @zucchini-nlp (visual-language models) or @gante (all others) - continuous batching: @remi-or @ArthurZucker @McPatate - pipelines: @Rocketknight1 - tokenizers: @ArthurZucker and @itazap - trainer: @SunMarc - attention: @vasqu @ArthurZucker @CyrilVallez - model loading (from pretrained, etc): @CyrilVallez - distributed: @3outeille @ArthurZucker - CIs: @ydshieh Integrations: - ray/raytune: @richardliaw, @amogkam - Big Model Inference: @SunMarc - quantization: @SunMarc - kernels: @drbh - peft: @BenjaminBossan @githubnemo Devices/Backends: - AMD ROCm: @ivarflakstad - Intel XPU: @IlyasMoutawwakil - Ascend NPU: @ivarflakstad Documentation: @stevhliu Research projects are not maintained and should be taken as is. -->

Changed files

  • src/transformers/tokenization_utils_base.py (modified, +14/-9)

Code Example

transformers==5.5.4
python3.12

---

transformers==4.57.6
python3.12

---

from transformers import CLIPTokenizer

tok1 = CLIPTokenizer.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", subfolder="tokenizer", local_files_only=True)

tok2 = CLIPTokenizer.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", subfolder="tokenizer")
print(tok1.model_max_length, tok2.model_max_length)
assert tok1.model_max_length == tok2.model_max_length

---

Traceback (most recent call last):
  File "/home/oleg.konin/mlapi-glm/test.py", line 3, in <module>
    tok1 = CLIPTokenizer.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", subfolder="tokenizer", local_files_only=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/models/clip/tokenization_clip.py", line 306, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType

---

tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████| 705/705 [00:00<00:00, 4.32MB/s]
vocab.json: 1.06MB [00:00, 20.6MB/s]
merges.txt: 525kB [00:00, 7.86MB/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████████| 588/588 [00:00<00:00, 6.32MB/s]
1000000000000000019884624838656 77
Traceback (most recent call last):
  File "/home/oleg.konin/mlapi-glm/test.py", line 7, in <module>
    assert tok1.model_max_length == tok2.model_max_length
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
RAW_BUFFERClick to expand / collapse

System Info

transformers==5.5.4
python3.12

vs

transformers==4.57.6
python3.12

Who can help?

@ArthurZucker @Cyrilvallez

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

This script:

from transformers import CLIPTokenizer

tok1 = CLIPTokenizer.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", subfolder="tokenizer", local_files_only=True)

tok2 = CLIPTokenizer.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", subfolder="tokenizer")
print(tok1.model_max_length, tok2.model_max_length)
assert tok1.model_max_length == tok2.model_max_length
  1. empty your ~/.cache/huggingface/hub cache folder
  2. run with trasformers 4.57.6
  3. observe the expected behavior (the model doesn't load)
  4. empty your ~/.cache/huggingface/hub cache folder
  5. run with transformers 5.5.4
  6. observe the tokenizer's stub being loaded without any exception

Expected behavior

gives me a different behavior

  • + transformers==4.57.6:
Traceback (most recent call last):
  File "/home/oleg.konin/mlapi-glm/test.py", line 3, in <module>
    tok1 = CLIPTokenizer.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", subfolder="tokenizer", local_files_only=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/models/clip/tokenization_clip.py", line 306, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType
  • + transformers==5.5.4:
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████| 705/705 [00:00<00:00, 4.32MB/s]
vocab.json: 1.06MB [00:00, 20.6MB/s]
merges.txt: 525kB [00:00, 7.86MB/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████████| 588/588 [00:00<00:00, 6.32MB/s]
1000000000000000019884624838656 77
Traceback (most recent call last):
  File "/home/oleg.konin/mlapi-glm/test.py", line 7, in <module>
    assert tok1.model_max_length == tok2.model_max_length
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

In the second case, I think an exception should be thrown, otherwise it makes you think that the model is already present on disk.

extent analysis

TL;DR

The issue is likely due to a change in how the transformers library handles model loading in version 5.5.4, and a workaround may involve modifying the model loading code to handle the case where the model is not found.

Guidance

  • The error message in the transformers==4.57.6 case suggests that the vocab_file is None, which is causing the TypeError. This could be due to the model not being found in the cache or on the hub.
  • In the transformers==5.5.4 case, the model is being loaded without throwing an exception, but the model_max_length attribute is not being set correctly, causing the AssertionError.
  • To mitigate this issue, you could try adding error handling to the model loading code to catch the case where the model is not found, and throw a meaningful exception.
  • You could also try checking the cache and hub to ensure that the model is being loaded correctly.

Example

try:
    tok1 = CLIPTokenizer.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", subfolder="tokenizer", local_files_only=True)
except Exception as e:
    print(f"Error loading model: {e}")

Notes

The exact cause of the issue is unclear, and it may be related to changes in the transformers library or the model itself. Further investigation would be needed to determine the root cause.

Recommendation

Apply workaround: add error handling to the model loading code to catch the case where the model is not found, and throw a meaningful exception. This will help to ensure that the issue is properly handled and provide a better user experience.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

gives me a different behavior

  • + transformers==4.57.6:
Traceback (most recent call last):
  File "/home/oleg.konin/mlapi-glm/test.py", line 3, in <module>
    tok1 = CLIPTokenizer.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", subfolder="tokenizer", local_files_only=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/oleg.konin/mlapi-glm/.venv/lib/python3.12/site-packages/transformers/models/clip/tokenization_clip.py", line 306, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType
  • + transformers==5.5.4:
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████| 705/705 [00:00<00:00, 4.32MB/s]
vocab.json: 1.06MB [00:00, 20.6MB/s]
merges.txt: 525kB [00:00, 7.86MB/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████████| 588/588 [00:00<00:00, 6.32MB/s]
1000000000000000019884624838656 77
Traceback (most recent call last):
  File "/home/oleg.konin/mlapi-glm/test.py", line 7, in <module>
    assert tok1.model_max_length == tok2.model_max_length
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

In the second case, I think an exception should be thrown, otherwise it makes you think that the model is already present on disk.

#api#generation error#database connection#vector store#embedding generation

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

transformers - ✅(Solved) Fix CLIPTokenizer uses 10**30 as `model_max_length` [1 pull requests]