transformers - 💡(How to fix) Fix `dtype` is silently ignored when loading composite (multimodal) checkpoints through AutoModelForCausalLM (affects all Qwen3.5 repos)

System Info

• transformers version: 5.10.0.dev0 (main @ effde20942); also reproduced on the v5.9.0 release
• Platform: macOS-26.1-arm64-arm-64bit (Apple Silicon)
• Python version: 3.11.15
• PyTorch version (GPU?): 2.11.0 (False; MPS available, repro runs on CPU)
• Huggingface_hub version: 1.14.0
• Safetensors version: 0.7.0
• Accelerate version: 1.13.0

Who can help?

@Cyrilvallez @zucchini-nlp

Information

• The official example scripts
• My own modified scripts

Reproduction

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32)
print(next(model.parameters()).dtype)  # torch.bfloat16, expected torch.float32

Same result with dtype="float32", with or without device_map, and with dtype=torch.float64. The concrete class honors the same argument, so the bug is specific to the Auto path:

from transformers import Qwen3_5ForCausalLM

model = Qwen3_5ForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32)
print(next(model.parameters()).dtype)  # torch.float32, correct

Fully offline reproduction (no Hub access, also a regression-test candidate):

import torch
from transformers import AutoModelForCausalLM, Qwen3_5Config, Qwen3_5ForConditionalGeneration

config = Qwen3_5Config(
    text_config=dict(
        vocab_size=64, hidden_size=32, intermediate_size=64, num_hidden_layers=2,
        num_attention_heads=2, num_key_value_heads=1, head_dim=16,
        linear_conv_kernel_dim=2, linear_key_head_dim=8, linear_value_head_dim=8,
        linear_num_key_heads=1, linear_num_value_heads=2, max_position_embeddings=128,
        layer_types=["linear_attention", "full_attention"],
        rope_parameters={"rope_type": "default", "rope_theta": 10000.0,
                         "partial_rotary_factor": 1.0, "mrope_section": [3, 3, 2]},
    ),
    vision_config=dict(depth=2, hidden_size=32, out_hidden_size=32, intermediate_size=64,
                       num_heads=2, patch_size=4, temporal_patch_size=2, spatial_merge_size=1),
)
Qwen3_5ForConditionalGeneration(config).to(torch.bfloat16).save_pretrained("/tmp/tiny_composite")

model = AutoModelForCausalLM.from_pretrained("/tmp/tiny_composite", dtype=torch.float32)
print(type(model).__name__, next(model.parameters()).dtype)
# Qwen3_5ForCausalLM torch.bfloat16, expected torch.float32

Root cause (line numbers as of effde20942)

1. _BaseAutoModelClass.from_pretrained pops dtype from kwargs only when it is the string "auto" (src/transformers/models/auto/auto_factory.py:330-333). A concrete torch.dtype or dtype string stays in kwargs.

2. AutoConfig.from_pretrained(..., return_unused_kwargs=True, **kwargs) (auto_factory.py:338) then absorbs dtype into the loaded composite config's top level via standard config-kwarg handling, and it is no longer in the returned unused kwargs:

   from transformers import AutoConfig
   cfg, unused = AutoConfig.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32, return_unused_kwargs=True)
   print(type(cfg).__name__, cfg.dtype, cfg.text_config.dtype, "dtype" in unused)
   # Qwen3_5Config torch.float32 torch.bfloat16 False

3. Because the resolved model class is text-only, model_class.config_class matches the composite's text_config, so the config is swapped: config = config.get_text_config() (auto_factory.py:394-398, same pattern in from_config around lines 240-244). The top-level dtype, which now holds the user's value, is dropped here. Note this block already propagates quantization_config from the parent config for exactly this reason (#45494), but not dtype.

4. model_class.from_pretrained(...) is then called with no dtype in kwargs, so it falls back to its default "auto", and _get_dtype (src/transformers/modeling_utils.py:837-838) resolves it from the extracted sub-config (text_config.dtype: bfloat16 in all Qwen3.5 checkpoints) or, when that is unset, from the checkpoint weights. The user's request is silently gone.

Impact

• All Qwen/Qwen3.5-* repos ship architectures: ["Qwen3_5ForConditionalGeneration"] (MoE: Qwen3_5MoeForConditionalGeneration) with text_config.dtype: bfloat16 (verified on 0.8B/2B/4B/9B/122B-A10B), so every AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-*", dtype=...) ignores the requested dtype.
• The failure is silent: no warning, and users believe they are running fp32/fp16. As the offline repro shows, it does not even require dtype in the sub-config json (it then falls back to the weights dtype), so it affects any composite checkpoint loaded through an Auto class whose resolved class's config_class equals sub_configs["text_config"].
• Concrete fallout: in #46190 the reporter's float32 control run silently executed in bfloat16, which made expected bf16 rounding look like a large correctness bug in GatedDeltaNet.

Expected behavior

dtype=torch.float32 through the Auto classes should produce fp32 parameters, identical to loading through the concrete class.

Possible fixes

1. Mirror #45494: in both get_text_config() extraction sites, carry the parent config's dtype over to the extracted text config when it is not None.
2. Alternatively, pop a user-provided dtype before the AutoConfig.from_pretrained call (extending the existing "auto" special case at auto_factory.py:330-333) and pass it explicitly to model_class.from_pretrained, so the user value never round-trips through the config object.

I am happy to open a PR with either approach plus an offline regression test based on the snippet above, lmk.