Issue Station
Sign In
Tools hereToolbox

transformers - ✅(Solved) Fix from_pretrained no longer uses mmap for CPU weights in transformers 5.x causing full materialization [1 pull requests, 5 comments, 4 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
huggingface/transformers#44262•Fetched 2026-04-08 00:29:30
View on GitHub
Comments
5
Participants
4
Timeline
13
Reactions
0
Author
nikitas-cerebrasnikitas-cerebras
Participants
CyrilvallezCyrilvallezNielBuysNielBuysnikitas-cerebrasnikitas-cerebrasRocketknight1Rocketknight1
Timeline (top)
commented ×5subscribed ×4mentioned ×2closed ×1

Fix Action

Fixed

PR fix notes

PR #44760: Add Mistral 4

Description (problem / solution / changelog)

What does this PR do?

<!-- Congratulations! You've made it this far! You're not quite done yet though. Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution. Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change. Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost. --> <!-- Remove if not applicable -->

This PR adds support to the Mistral 4 family

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @ If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**. Please tag fewer than 3 people. Models: - text models: @ArthurZucker @Cyrilvallez - vision models: @yonigozlan @molbap - audio models: @eustlb @ebezzam @vasqu - multimodal models: @zucchini-nlp - graph models: @clefourrier Library: - generate: @zucchini-nlp (visual-language models) or @gante (all others) - continuous batching: @remi-or @ArthurZucker @McPatate - pipelines: @Rocketknight1 - tokenizers: @ArthurZucker and @itazap - trainer: @SunMarc - attention: @vasqu @ArthurZucker @CyrilVallez - model loading (from pretrained, etc): @CyrilVallez - distributed: @3outeille @ArthurZucker - CIs: @ydshieh Integrations: - ray/raytune: @richardliaw, @amogkam - Big Model Inference: @SunMarc - quantization: @SunMarc - kernels: @drbh - peft: @BenjaminBossan @githubnemo Devices/Backends: - AMD ROCm: @ivarflakstad - Intel XPU: @IlyasMoutawwakil - Ascend NPU: @ivarflakstad Documentation: @stevhliu Research projects are not maintained and should be taken as is. -->

@ArthurZucker @Cyrilvallez @patrickvonplaten

Changed files

  • docs/source/en/_toctree.yml (modified, +2/-0)
  • docs/source/en/model_doc/mistral4.md (added, +116/-0)
  • src/transformers/models/__init__.py (modified, +1/-0)
  • src/transformers/models/auto/configuration_auto.py (modified, +2/-0)
  • src/transformers/models/auto/modeling_auto.py (modified, +5/-0)
  • src/transformers/models/ministral3/modeling_ministral3.py (modified, +2/-2)
  • src/transformers/models/ministral3/modular_ministral3.py (modified, +2/-2)
  • src/transformers/models/mistral4/__init__.py (added, +27/-0)
  • src/transformers/models/mistral4/configuration_mistral4.py (added, +149/-0)
  • src/transformers/models/mistral4/convert_mistral4_weight_to_hf.py (added, +608/-0)
  • src/transformers/models/mistral4/modeling_mistral4.py (added, +730/-0)
  • src/transformers/models/mistral4/modular_mistral4.py (added, +291/-0)
  • src/transformers/processing_utils.py (modified, +5/-0)
  • tests/models/mistral4/__init__.py (added, +17/-0)
  • tests/models/mistral4/test_modeling_mistral4.py (added, +133/-0)

Code Example

import psutil
from transformers import AutoModel


rss_before = psutil.Process().memory_info().rss
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    dtype="auto",
    device_map="auto",
    max_memory={"cpu": "1024GB", 0: 0},
)
rss_after = psutil.Process().memory_info().rss
print(f"Memory: {(rss_after - rss_before) >> 20} MiB")

---

Loading checkpoint shards: 100%|██████████| 16/16 [00:00<00:00, 17.03it/s]
Memory: 1321 MiB

---

Loading weights: 100%|██████████| 530/530 [00:13<00:00, 39.48it/s, Materializing param=norm.weight]
Memory: 110705 MiB
RAW_BUFFERClick to expand / collapse

I have the following piece of code:

import psutil
from transformers import AutoModel


rss_before = psutil.Process().memory_info().rss
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    dtype="auto",
    device_map="auto",
    max_memory={"cpu": "1024GB", 0: 0},
)
rss_after = psutil.Process().memory_info().rss
print(f"Memory: {(rss_after - rss_before) >> 20} MiB")

When running it on transformers 4.57, I get:

Loading checkpoint shards: 100%|██████████| 16/16 [00:00<00:00, 17.03it/s]
Memory: 1321 MiB

When running it on transformers 5.2, I get:

Loading weights: 100%|██████████| 530/530 [00:13<00:00, 39.48it/s, Materializing param=norm.weight]
Memory: 110705 MiB

I see that the mode loading was reworked. It seems like on the old transformers version model weights are loaded with memory mapping, while on the new transformers version all the weights are materialized.

I wonder how to achieve the old behavior with the new transformers?

extent analysis

Fix Plan

Achieve Old Behavior with New Transformers

To achieve the old behavior with memory mapping, you can use the map_location argument when loading the model weights.

Step-by-Step Solution

  1. Import map_location from torch: Add the following line at the top of your code.

from torch import map_location

2. **Use `map_location` when loading the model**: Update the `from_pretrained` call to use `map_location`.
   ```python
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    dtype="auto",
    device_map="auto",
    max_memory={"cpu": "1024GB", 0: 0},
    map_location='cpu'  # or 'cuda:0' if you're using a GPU
)

This will load the model weights using memory mapping, similar to the old behavior.

Example Code

import psutil
from transformers import AutoModel
from torch import map_location

rss_before = psutil.Process().memory_info().rss
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    dtype="auto",
    device_map="auto",
    max_memory={"cpu": "1024GB", 0: 0},
    map_location='cpu'  # or 'cuda:0' if you're using a GPU
)
rss_after = psutil.Process().memory_info().rss
print(f"Memory: {(rss_after - rss_before) >> 20} MiB")

Verification

Run the code and check the memory usage. It should be similar to the old behavior with transformers 4.57.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#serialization error#model compatibility#GPU setup#container setup#orchestration issue

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

transformers - ✅(Solved) Fix from_pretrained no longer uses mmap for CPU weights in transformers 5.x causing full materialization [1 pull requests, 5 comments, 4 participants]