transformers - 💡(How to fix) Fix Unable to resume model training with a locally downloaded model and trust remote code

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Error Message

[rank1]: Traceback (most recent call last): [rank1]: File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module> [rank1]: main() [rank1]: File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main [rank1]: trainer_stats = trainer.train(resume_from_checkpoint=resume) [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train [rank1]: self._load_from_checkpoint(resume_from_checkpoint) [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint [rank1]: loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code) [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper [rank1]: return func(*args, **kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in init [rank1]: super().init( [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in init [rank1]: modules, self.module_kwargs = self._load_modules( [rank1]: ^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules [rank1]: return self._load_config_modules(model_name_or_path, **load_kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules [rank1]: module = module_class.load( [rank1]: ^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load [rank1]: return cls(model_name_or_path=model_name_or_path, **init_kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper [rank1]: return func(*args, **kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in init [rank1]: self.model = self._load_model( [rank1]: ^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model [rank1]: return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained [rank1]: model_class = get_class_from_dynamic_module( [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module [rank1]: final_module = get_cached_module_file( [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file [rank1]: resolved_module_file = cached_file( [rank1]: ^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file [rank1]: file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files [rank1]: raise OSError( [rank1]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files. [rank3]: Traceback (most recent call last): [rank3]: File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module> [rank3]: main() [rank3]: File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main [rank3]: trainer_stats = trainer.train(resume_from_checkpoint=resume) [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train [rank3]: self._load_from_checkpoint(resume_from_checkpoint) [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint [rank3]: loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code) [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper [rank3]: return func(*args, **kwargs) [rank3]: ^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in init [rank3]: super().init( [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in init [rank3]: modules, self.module_kwargs = self._load_modules( [rank3]: ^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules [rank3]: return self._load_config_modules(model_name_or_path, **load_kwargs) [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules [rank3]: module = module_class.load( [rank3]: ^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load [rank3]: return cls(model_name_or_path=model_name_or_path, **init_kwargs) [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper [rank3]: return func(*args, **kwargs) [rank3]: ^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in init [rank3]: self.model = self._load_model( [rank3]: ^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model [rank3]: return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs) [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained [rank3]: model_class = get_class_from_dynamic_module( [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module [rank3]: final_module = get_cached_module_file( [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file [rank3]: resolved_module_file = cached_file( [rank3]: ^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file [rank3]: file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs) [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank3]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files [rank3]: raise OSError( [rank3]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files. [rank2]: Traceback (most recent call last): [rank2]: File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module> [rank2]: main() [rank2]: File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main [rank2]: trainer_stats = trainer.train(resume_from_checkpoint=resume) [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train [rank2]: self._load_from_checkpoint(resume_from_checkpoint) [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint [rank2]: loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code) [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper [rank2]: return func(*args, **kwargs) [rank2]: ^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in init [rank2]: super().init( [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in init [rank2]: modules, self.module_kwargs = self._load_modules( [rank2]: ^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules [rank2]: return self._load_config_modules(model_name_or_path, **load_kwargs) [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules [rank2]: module = module_class.load( [rank2]: ^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load [rank2]: return cls(model_name_or_path=model_name_or_path, **init_kwargs) [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper [rank2]: return func(*args, **kwargs) [rank2]: ^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in init [rank2]: self.model = self._load_model( [rank2]: ^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model [rank2]: return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs) [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained [rank2]: model_class = get_class_from_dynamic_module( [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module [rank2]: final_module = get_cached_module_file( [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file [rank2]: resolved_module_file = cached_file( [rank2]: ^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file [rank2]: file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs) [rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank2]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files [rank2]: raise OSError( [rank2]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files. [rank0]: Traceback (most recent call last): [rank0]: File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module> [rank0]: main() [rank0]: File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main [rank0]: trainer_stats = trainer.train(resume_from_checkpoint=resume) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train [rank0]: self._load_from_checkpoint(resume_from_checkpoint) [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint [rank0]: loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper [rank0]: return func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in init [rank0]: super().init( [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in init [rank0]: modules, self.module_kwargs = self._load_modules( [rank0]: ^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules [rank0]: return self._load_config_modules(model_name_or_path, **load_kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules [rank0]: module = module_class.load( [rank0]: ^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load [rank0]: return cls(model_name_or_path=model_name_or_path, **init_kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper [rank0]: return func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in init [rank0]: self.model = self._load_model( [rank0]: ^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model [rank0]: return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained [rank0]: model_class = get_class_from_dynamic_module( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module [rank0]: final_module = get_cached_module_file( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file [rank0]: resolved_module_file = cached_file( [rank0]: ^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file [rank0]: file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files [rank0]: raise OSError( [rank0]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.

Code Example

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="voyageai/voyage-4-nano",
    local_dir="/e/data1/datasets/playground/mmlaion/shared/enrico/models/voyage-4-nano",
)

---

model = SentenceTransformer(
        base_model_path, 
        trust_remote_code=trust_remote_code
    )

---

[rank1]: Traceback (most recent call last):
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank1]:     main()
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank1]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank1]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank1]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank1]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank1]:     super().__init__(
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank1]:     modules, self.module_kwargs = self._load_modules(
[rank1]:                                   ^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank1]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank1]:     module = module_class.load(
[rank1]:              ^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank1]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank1]:     self.model = self._load_model(
[rank1]:                  ^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank1]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank1]:     model_class = get_class_from_dynamic_module(
[rank1]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank1]:     final_module = get_cached_module_file(
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank1]:     resolved_module_file = cached_file(
[rank1]:                            ^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank1]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank1]:     raise OSError(
[rank1]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank3]: Traceback (most recent call last):
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank3]:     main()
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank3]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank3]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank3]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank3]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank3]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank3]:     super().__init__(
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank3]:     modules, self.module_kwargs = self._load_modules(
[rank3]:                                   ^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank3]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank3]:     module = module_class.load(
[rank3]:              ^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank3]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank3]:     self.model = self._load_model(
[rank3]:                  ^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank3]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank3]:     model_class = get_class_from_dynamic_module(
[rank3]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank3]:     final_module = get_cached_module_file(
[rank3]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank3]:     resolved_module_file = cached_file(
[rank3]:                            ^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank3]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank3]:     raise OSError(
[rank3]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank2]: Traceback (most recent call last):
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank2]:     main()
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank2]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank2]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank2]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank2]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank2]:     super().__init__(
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank2]:     modules, self.module_kwargs = self._load_modules(
[rank2]:                                   ^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank2]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank2]:     module = module_class.load(
[rank2]:              ^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank2]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank2]:     self.model = self._load_model(
[rank2]:                  ^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank2]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank2]:     model_class = get_class_from_dynamic_module(
[rank2]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank2]:     final_module = get_cached_module_file(
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank2]:     resolved_module_file = cached_file(
[rank2]:                            ^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank2]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank2]:     raise OSError(
[rank2]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank0]:     main()
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank0]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank0]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank0]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank0]:     super().__init__(
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank0]:     modules, self.module_kwargs = self._load_modules(
[rank0]:                                   ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank0]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank0]:     module = module_class.load(
[rank0]:              ^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank0]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank0]:     self.model = self._load_model(
[rank0]:                  ^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank0]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank0]:     model_class = get_class_from_dynamic_module(
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank0]:     final_module = get_cached_module_file(
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank0]:     resolved_module_file = cached_file(
[rank0]:                            ^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank0]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank0]:     raise OSError(
[rank0]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
RAW_BUFFERClick to expand / collapse

System Info

requires-python = ">=3.12" dependencies = [ "accelerate>=1.13.0", "datasets>=4.8.5", "sentence-transformers>=5.4.1", "torch>=2.11.0", "transformers>=5.8.0", ]

Who can help?

@tomaszcichy98

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

When downloading a model locally using:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="voyageai/voyage-4-nano",
    local_dir="/e/data1/datasets/playground/mmlaion/shared/enrico/models/voyage-4-nano",
)

I am unable to resume training as there are missing files due to custom code not being saved on disk.

    model = SentenceTransformer(
        base_model_path, 
        trust_remote_code=trust_remote_code
    )

I get the error:

[rank1]: Traceback (most recent call last):
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank1]:     main()
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank1]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank1]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank1]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank1]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank1]:     super().__init__(
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank1]:     modules, self.module_kwargs = self._load_modules(
[rank1]:                                   ^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank1]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank1]:     module = module_class.load(
[rank1]:              ^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank1]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank1]:     self.model = self._load_model(
[rank1]:                  ^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank1]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank1]:     model_class = get_class_from_dynamic_module(
[rank1]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank1]:     final_module = get_cached_module_file(
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank1]:     resolved_module_file = cached_file(
[rank1]:                            ^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank1]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank1]:     raise OSError(
[rank1]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank3]: Traceback (most recent call last):
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank3]:     main()
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank3]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank3]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank3]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank3]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank3]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank3]:     super().__init__(
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank3]:     modules, self.module_kwargs = self._load_modules(
[rank3]:                                   ^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank3]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank3]:     module = module_class.load(
[rank3]:              ^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank3]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank3]:     self.model = self._load_model(
[rank3]:                  ^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank3]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank3]:     model_class = get_class_from_dynamic_module(
[rank3]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank3]:     final_module = get_cached_module_file(
[rank3]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank3]:     resolved_module_file = cached_file(
[rank3]:                            ^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank3]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank3]:     raise OSError(
[rank3]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank2]: Traceback (most recent call last):
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank2]:     main()
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank2]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank2]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank2]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank2]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank2]:     super().__init__(
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank2]:     modules, self.module_kwargs = self._load_modules(
[rank2]:                                   ^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank2]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank2]:     module = module_class.load(
[rank2]:              ^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank2]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank2]:     self.model = self._load_model(
[rank2]:                  ^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank2]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank2]:     model_class = get_class_from_dynamic_module(
[rank2]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank2]:     final_module = get_cached_module_file(
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank2]:     resolved_module_file = cached_file(
[rank2]:                            ^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank2]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank2]:     raise OSError(
[rank2]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank0]:     main()
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank0]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank0]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank0]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank0]:     super().__init__(
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank0]:     modules, self.module_kwargs = self._load_modules(
[rank0]:                                   ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank0]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank0]:     module = module_class.load(
[rank0]:              ^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank0]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank0]:     self.model = self._load_model(
[rank0]:                  ^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank0]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank0]:     model_class = get_class_from_dynamic_module(
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank0]:     final_module = get_cached_module_file(
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank0]:     resolved_module_file = cached_file(
[rank0]:                            ^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank0]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank0]:     raise OSError(
[rank0]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.

This currently happens with the models: https://huggingface.co/voyageai/voyage-4-nano https://huggingface.co/jinaai/jina-embeddings-v5-text-small-retrieval

Expected behavior

I should be able to resume model training.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

I should be able to resume model training.

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING