transformers - ✅(Solved) Fix incompatiblity between torch 2.4.1 and transformers 5.8.0 [1 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
huggingface/transformers#45800Fetched 2026-05-07 03:31:19
View on GitHub
Comments
1
Participants
2
Timeline
4
Reactions
0
Author
Timeline (top)
commented ×1cross-referenced ×1labeled ×1subscribed ×1

Error Message

predict | Traceback (most recent call last): predict | File "/app/tmp.py", line 1, in <module> predict | from rfdetr import RFDETRMedium predict | File "/usr/local/lib/python3.12/dist-packages/rfdetr/init.py", line 7, in <module> predict | from rfdetr.detr import ( predict | File "/usr/local/lib/python3.12/dist-packages/rfdetr/detr.py", line 52, in <module> predict | from rfdetr.models import PostProcess, build_model predict | File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/init.py", line 17, in <module> predict | from rfdetr.models.lwdetr import build_model predict | File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/lwdetr.py", line 30, in <module> predict | from rfdetr.models.backbone import build_backbone predict | File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/backbone/init.py", line 15, in <module> predict | from rfdetr.models.backbone.backbone import Backbone predict | File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/backbone/backbone.py", line 22, in <module> predict | from peft import PeftModel predict | File "/usr/local/lib/python3.12/dist-packages/peft/init.py", line 17, in <module> predict | from .auto import ( predict | File "/usr/local/lib/python3.12/dist-packages/peft/auto.py", line 31, in <module> predict | from .config import PeftConfig predict | File "/usr/local/lib/python3.12/dist-packages/peft/config.py", line 30, in <module> predict | from .utils import CONFIG_NAME, PeftType, TaskType predict | File "/usr/local/lib/python3.12/dist-packages/peft/utils/init.py", line 15, in <module> predict | from .constants import ALLOWED_COMPUTE_DTYPES, UPCAST_DTYPES predict | File "/usr/local/lib/python3.12/dist-packages/peft/utils/constants.py", line 16, in <module> predict | from transformers import BloomPreTrainedModel predict | File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist predict | File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2226, in getattr predict | module = self._get_module(self._class_to_module[name]) predict | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ predict | File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2460, in _get_module predict | raise e predict | File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2458, in _get_module predict | return importlib.import_module("." + module_name, self.name) predict | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ predict | File "/usr/lib/python3.12/importlib/init.py", line 90, in import_module predict | return _bootstrap._gcd_import(name[level:], package, level) predict | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ predict | File "/usr/local/lib/python3.12/dist-packages/transformers/models/bloom/modeling_bloom.py", line 26, in <module> predict | from ...modeling_layers import GradientCheckpointingLayer predict | File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py", line 27, in <module> predict | from .processing_utils import Unpack predict | File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 79, in <module> predict | from .modeling_utils import PreTrainedAudioTokenizerBase predict | File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 69, in <module> predict | from .integrations.finegrained_fp8 import ALL_FP8_EXPERTS_FUNCTIONS predict | File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/finegrained_fp8.py", line 30, in <module> predict | from .moe import ExpertsInterface, use_experts_implementation predict | File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/moe.py", line 250, in <module> predict | torch.library.custom_op("transformers::grouped_mm_fallback", _grouped_mm_fallback, mutates_args=()) predict | File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 142, in custom_op predict | return inner(fn) predict | ^^^^^^^^^ predict | File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 119, in inner predict | schema_str = torch._custom_op.impl.infer_schema(fn, mutates_args) predict | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ predict | File "/usr/local/lib/python3.12/dist-packages/torch/_library/infer_schema.py", line 42, in infer_schema predict | error_fn( predict | File "/usr/local/lib/python3.12/dist-packages/torch/_library/infer_schema.py", line 21, in error_fn predict | raise ValueError( predict | ValueError: infer_schema(func): Parameter input has unsupported type torch.Tensor. The valid types are: dict_keys([<class 'torch.Tensor'>, typing.Optional[torch.Tensor], typing.Sequence[torch.Tensor], typing.List[torch.Tensor], typing.Sequence[typing.Optional[torch.Tensor]], typing.List[typing.Optional[torch.Tensor]], <class 'int'>, typing.Optional[int], typing.Sequence[int], typing.List[int], typing.Optional[typing.Sequence[int]], typing.Optional[typing.List[int]], <class 'float'>, typing.Optional[float], typing.Sequence[float], typing.List[float], typing.Optional[typing.Sequence[float]], typing.Optional[typing.List[float]], <class 'bool'>, typing.Optional[bool], typing.Sequence[bool], typing.List[bool], typing.Optional[typing.Sequence[bool]], typing.Optional[typing.List[bool]], <class 'str'>, typing.Optional[str], typing.Union[int, float, bool], typing.Union[int, float, bool, NoneType], typing.Sequence[typing.Union[int, float, bool]], typing.List[typing.Union[int, float, bool]], <class 'torch.dtype'>, typing.Optional[torch.dtype], <class 'torch.device'>, typing.Optional[torch.device]]). Got func with signature (input: 'torch.Tensor', weight: 'torch.Tensor', offs: 'torch.Tensor') -> 'torch.Tensor')

Fix Action

Fixed

PR fix notes

PR #45804: Fix import error in moe.py by providing explicit schema to custom_op

Description (problem / solution / changelog)

Fixes #45800

In integrations/moe.py, the from __future__ import annotations import at the top makes all type annotations evaluate as strings, so torch.Tensor is stored as "torch.Tensor" rather than the actual class object at definition time. Because of this, torch.library.custom_op (in PyTorch 2.4.1) fails during schema inference.

Solution

Pass an explicit schema string to torch.library.custom_op so PyTorch skips the annotation inference step entirely:

torch.library.custom_op(
    "transformers::grouped_mm_fallback",
    _grouped_mm_fallback,
    mutates_args=(),
    schema="(Tensor input, Tensor weight, Tensor offs) -> Tensor",
)

Verified in an isolated venv with the exact versions from the issue report:

torch==2.4.1
torchvision==0.19.1
transformers==5.8.0
rfdetr

The import succeeds with this fix.

cc: @SunMarc

Changed files

  • src/transformers/integrations/moe.py (modified, +6/-1)

Code Example

FROM nvcr.io/nvidia/tensorrt:25.04-py3
ARG DEBIAN_FRONTEND=noninteractive

# Install dependencies
RUN apt-get update && apt-get install libgl1 -y
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu124
RUN pip install rfdetr

---

from rfdetr import RFDETRMedium
model = RFDETRMedium()

---

predict  | Traceback (most recent call last):
predict  |   File "/app/tmp.py", line 1, in <module>
predict  |     from rfdetr import RFDETRMedium
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/__init__.py", line 7, in <module>
predict  |     from rfdetr.detr import (
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/detr.py", line 52, in <module>
predict  |     from rfdetr.models import PostProcess, build_model
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/__init__.py", line 17, in <module>
predict  |     from rfdetr.models.lwdetr import build_model
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/lwdetr.py", line 30, in <module>
predict  |     from rfdetr.models.backbone import build_backbone
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/backbone/__init__.py", line 15, in <module>
predict  |     from rfdetr.models.backbone.backbone import Backbone
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/backbone/backbone.py", line 22, in <module>
predict  |     from peft import PeftModel
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/__init__.py", line 17, in <module>
predict  |     from .auto import (
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/auto.py", line 31, in <module>
predict  |     from .config import PeftConfig
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/config.py", line 30, in <module>
predict  |     from .utils import CONFIG_NAME, PeftType, TaskType
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/utils/__init__.py", line 15, in <module>
predict  |     from .constants import ALLOWED_COMPUTE_DTYPES, UPCAST_DTYPES
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/utils/constants.py", line 16, in <module>
predict  |     from transformers import BloomPreTrainedModel
predict  |   File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2226, in __getattr__
predict  |     module = self._get_module(self._class_to_module[name])
predict  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2460, in _get_module
predict  |     raise e
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2458, in _get_module
predict  |     return importlib.import_module("." + module_name, self.__name__)
predict  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
predict  |     return _bootstrap._gcd_import(name[level:], package, level)
predict  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/models/bloom/modeling_bloom.py", line 26, in <module>
predict  |     from ...modeling_layers import GradientCheckpointingLayer
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py", line 27, in <module>
predict  |     from .processing_utils import Unpack
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 79, in <module>
predict  |     from .modeling_utils import PreTrainedAudioTokenizerBase
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 69, in <module>
predict  |     from .integrations.finegrained_fp8 import ALL_FP8_EXPERTS_FUNCTIONS
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/finegrained_fp8.py", line 30, in <module>
predict  |     from .moe import ExpertsInterface, use_experts_implementation
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/moe.py", line 250, in <module>
predict  |     torch.library.custom_op("transformers::grouped_mm_fallback", _grouped_mm_fallback, mutates_args=())
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 142, in custom_op
predict  |     return inner(fn)
predict  |            ^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 119, in inner
predict  |     schema_str = torch._custom_op.impl.infer_schema(fn, mutates_args)
predict  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/infer_schema.py", line 42, in infer_schema
predict  |     error_fn(
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/infer_schema.py", line 21, in error_fn
predict  |     raise ValueError(
predict  | ValueError: infer_schema(func): Parameter input has unsupported type torch.Tensor. The valid types are: dict_keys([<class 'torch.Tensor'>, typing.Optional[torch.Tensor], typing.Sequence[torch.Tensor], typing.List[torch.Tensor], typing.Sequence[typing.Optional[torch.Tensor]], typing.List[typing.Optional[torch.Tensor]], <class 'int'>, typing.Optional[int], typing.Sequence[int], typing.List[int], typing.Optional[typing.Sequence[int]], typing.Optional[typing.List[int]], <class 'float'>, typing.Optional[float], typing.Sequence[float], typing.List[float], typing.Optional[typing.Sequence[float]], typing.Optional[typing.List[float]], <class 'bool'>, typing.Optional[bool], typing.Sequence[bool], typing.List[bool], typing.Optional[typing.Sequence[bool]], typing.Optional[typing.List[bool]], <class 'str'>, typing.Optional[str], typing.Union[int, float, bool], typing.Union[int, float, bool, NoneType], typing.Sequence[typing.Union[int, float, bool]], typing.List[typing.Union[int, float, bool]], <class 'torch.dtype'>, typing.Optional[torch.dtype], <class 'torch.device'>, typing.Optional[torch.device]]). Got func with signature (input: 'torch.Tensor', weight: 'torch.Tensor', offs: 'torch.Tensor') -> 'torch.Tensor')
RAW_BUFFERClick to expand / collapse

System Info

torch 2.4.1 transformers 5.8.0

Who can help?

No response

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

dockerfile:

FROM nvcr.io/nvidia/tensorrt:25.04-py3
ARG DEBIAN_FRONTEND=noninteractive

# Install dependencies
RUN apt-get update && apt-get install libgl1 -y
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu124
RUN pip install rfdetr

python code:

from rfdetr import RFDETRMedium
model = RFDETRMedium()

error logs:

predict  | Traceback (most recent call last):
predict  |   File "/app/tmp.py", line 1, in <module>
predict  |     from rfdetr import RFDETRMedium
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/__init__.py", line 7, in <module>
predict  |     from rfdetr.detr import (
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/detr.py", line 52, in <module>
predict  |     from rfdetr.models import PostProcess, build_model
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/__init__.py", line 17, in <module>
predict  |     from rfdetr.models.lwdetr import build_model
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/lwdetr.py", line 30, in <module>
predict  |     from rfdetr.models.backbone import build_backbone
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/backbone/__init__.py", line 15, in <module>
predict  |     from rfdetr.models.backbone.backbone import Backbone
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/backbone/backbone.py", line 22, in <module>
predict  |     from peft import PeftModel
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/__init__.py", line 17, in <module>
predict  |     from .auto import (
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/auto.py", line 31, in <module>
predict  |     from .config import PeftConfig
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/config.py", line 30, in <module>
predict  |     from .utils import CONFIG_NAME, PeftType, TaskType
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/utils/__init__.py", line 15, in <module>
predict  |     from .constants import ALLOWED_COMPUTE_DTYPES, UPCAST_DTYPES
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/utils/constants.py", line 16, in <module>
predict  |     from transformers import BloomPreTrainedModel
predict  |   File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2226, in __getattr__
predict  |     module = self._get_module(self._class_to_module[name])
predict  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2460, in _get_module
predict  |     raise e
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2458, in _get_module
predict  |     return importlib.import_module("." + module_name, self.__name__)
predict  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
predict  |     return _bootstrap._gcd_import(name[level:], package, level)
predict  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/models/bloom/modeling_bloom.py", line 26, in <module>
predict  |     from ...modeling_layers import GradientCheckpointingLayer
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py", line 27, in <module>
predict  |     from .processing_utils import Unpack
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 79, in <module>
predict  |     from .modeling_utils import PreTrainedAudioTokenizerBase
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 69, in <module>
predict  |     from .integrations.finegrained_fp8 import ALL_FP8_EXPERTS_FUNCTIONS
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/finegrained_fp8.py", line 30, in <module>
predict  |     from .moe import ExpertsInterface, use_experts_implementation
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/moe.py", line 250, in <module>
predict  |     torch.library.custom_op("transformers::grouped_mm_fallback", _grouped_mm_fallback, mutates_args=())
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 142, in custom_op
predict  |     return inner(fn)
predict  |            ^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 119, in inner
predict  |     schema_str = torch._custom_op.impl.infer_schema(fn, mutates_args)
predict  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/infer_schema.py", line 42, in infer_schema
predict  |     error_fn(
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/infer_schema.py", line 21, in error_fn
predict  |     raise ValueError(
predict  | ValueError: infer_schema(func): Parameter input has unsupported type torch.Tensor. The valid types are: dict_keys([<class 'torch.Tensor'>, typing.Optional[torch.Tensor], typing.Sequence[torch.Tensor], typing.List[torch.Tensor], typing.Sequence[typing.Optional[torch.Tensor]], typing.List[typing.Optional[torch.Tensor]], <class 'int'>, typing.Optional[int], typing.Sequence[int], typing.List[int], typing.Optional[typing.Sequence[int]], typing.Optional[typing.List[int]], <class 'float'>, typing.Optional[float], typing.Sequence[float], typing.List[float], typing.Optional[typing.Sequence[float]], typing.Optional[typing.List[float]], <class 'bool'>, typing.Optional[bool], typing.Sequence[bool], typing.List[bool], typing.Optional[typing.Sequence[bool]], typing.Optional[typing.List[bool]], <class 'str'>, typing.Optional[str], typing.Union[int, float, bool], typing.Union[int, float, bool, NoneType], typing.Sequence[typing.Union[int, float, bool]], typing.List[typing.Union[int, float, bool]], <class 'torch.dtype'>, typing.Optional[torch.dtype], <class 'torch.device'>, typing.Optional[torch.device]]). Got func with signature (input: 'torch.Tensor', weight: 'torch.Tensor', offs: 'torch.Tensor') -> 'torch.Tensor')

Expected behavior

it should not generate any errors

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

it should not generate any errors

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

transformers - ✅(Solved) Fix incompatiblity between torch 2.4.1 and transformers 5.8.0 [1 pull requests, 1 comments, 2 participants]