Issue Station
Sign In
Tools hereToolbox

transformers - ✅(Solved) Fix [BitsAndBytesConfig] Providing llm_int8_skip_modules clears the default lm_head exclusion, causing AssertionError in 4-bit inference [1 pull requests, 5 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
huggingface/transformers#45674•Fetched 2026-04-29 06:11:27
View on GitHub
Comments
5
Participants
2
Timeline
7
Reactions
0
Author
softguy777softguy777
Participants
softguy777softguy777SunMarcSunMarc
Timeline (top)
commented ×5cross-referenced ×2

Error Message

At minimum, the docstring for BitsAndBytesConfig.llm_int8_skip_modules should warn that lm_head must be included manually when this parameter is set.

Root Cause

In transformers/quantizers/base.py, get_modules_to_not_convert():

if skip_modules is None or add_default_skips:
    modules_to_not_convert = get_keys_to_not_convert(model)  # auto-detects lm_head
else:
    modules_to_not_convert = []  # ← cleared when user provides ANY list!

if skip_modules is not None:
    modules_to_not_convert.extend(skip_modules)

When llm_int8_skip_modules=None (default), get_keys_to_not_convert() automatically finds and excludes lm_head and other output projection layers.

The moment the user provides any list (e.g., to protect a multimodal audio/vision tower from quantization), the auto-exclusion is disabled and lm_head gets quantized → AssertionError in bitsandbytes.

This is particularly easy to hit with multimodal models like Gemma 4 E2B-IT, where users need to explicitly skip the audio/vision towers to prevent quality degradation, but are not aware that doing so also removes the lm_head protection.

Fix Action

Fix / Workaround

Workaround (user must manually re-add lm_head):

# ✅ Works: lm_head added explicitly
llm_int8_skip_modules=["lm_head", "model.audio_tower"]

PR fix notes

PR #45683: Exclude audio modules from conversion process

Description (problem / solution / changelog)

Add logic to exclude audio modules from conversion to prevent uint8 crash in multimodal models.

As discussed in the issue, this prevents the torch.finfo() TypeError on uint8 weights during 4-bit inference for multimodal models like Gemma 4, and preserves audio encoder fidelity.

What does this PR do?

<!-- Congratulations! You've made it this far! You're not quite done yet though. Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution. Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change. Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost. --> <!-- Remove if not applicable -->

Fixes #45672, Fixes #45674

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by code agents. We are currently bottlenecked by our ability to review and respond to them. As a result, we ask that new users do not submit pure code agent PRs at this time. You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result, this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

  • I confirm that this is not a pure code agent PR.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @ If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**. Please tag fewer than 3 people. Models: - text models: @ArthurZucker @Cyrilvallez - vision models: @yonigozlan @molbap - audio models: @eustlb @ebezzam @vasqu - multimodal models: @zucchini-nlp - graph models: @clefourrier Library: - generate: @zucchini-nlp (visual-language models) or @gante (all others) - continuous batching: @remi-or @ArthurZucker @McPatate - pipelines: @Rocketknight1 - tokenizers: @ArthurZucker and @itazap - trainer: @SunMarc - attention: @vasqu @ArthurZucker @CyrilVallez - model loading (from pretrained, etc): @CyrilVallez - distributed: @3outeille @ArthurZucker - CIs: @ydshieh Integrations: - ray/raytune: @richardliaw, @amogkam - Big Model Inference: @SunMarc - quantization: @SunMarc - kernels: @drbh - peft: @BenjaminBossan @githubnemo Devices/Backends: - AMD ROCm: @ivarflakstad - Intel XPU: @IlyasMoutawwakil - Ascend NPU: @ivarflakstad Documentation: @stevhliu Research projects are not maintained and should be taken as is. -->

Changed files

  • src/transformers/quantizers/base.py (modified, +6/-0)

Code Example

File "bitsandbytes/nn/modules.py", line 415, in fix_4bit_weight_quant_state_from_module
    assert module.weight.shape[1] == 1
AssertionError

---

if skip_modules is None or add_default_skips:
    modules_to_not_convert = get_keys_to_not_convert(model)  # auto-detects lm_head
else:
    modules_to_not_convert = []  # ← cleared when user provides ANY list!

if skip_modules is not None:
    modules_to_not_convert.extend(skip_modules)

---

import torch
from transformers import AutoModelForImageTextToText, BitsAndBytesConfig

# ❌ Crashes: providing any list clears the lm_head default exclusion
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    llm_int8_skip_modules=["model.audio_tower"],
)
model = AutoModelForImageTextToText.from_pretrained(
    "google/gemma-4-e2b-it",
    quantization_config=bnb_config,
    device_map="auto",
)
# model.generate(...) → AssertionError in lm_head

---

# ✅ Works: lm_head added explicitly
llm_int8_skip_modules=["lm_head", "model.audio_tower"]

---

# In get_modules_to_not_convert():
modules_to_not_convert = get_keys_to_not_convert(model)  # always start with defaults

if skip_modules is not None:
    modules_to_not_convert.extend(skip_modules)  # user additions merged, not replacing

modules_to_not_convert = list(set(modules_to_not_convert))
return modules_to_not_convert
RAW_BUFFERClick to expand / collapse

Environment

  • transformers: 5.5.4
  • bitsandbytes: 0.49.2
  • torch: 2.11.0+cu126
  • CUDA: 12.6
  • OS: Windows 11
  • GPU: NVIDIA RTX 3090

Bug Description

When specifying llm_int8_skip_modules in BitsAndBytesConfig, the default module exclusion list (which normally protects lm_head from being quantized) is silently cleared. This causes a crash during inference:

File "bitsandbytes/nn/modules.py", line 415, in fix_4bit_weight_quant_state_from_module
    assert module.weight.shape[1] == 1
AssertionError

Root Cause

In transformers/quantizers/base.py, get_modules_to_not_convert():

if skip_modules is None or add_default_skips:
    modules_to_not_convert = get_keys_to_not_convert(model)  # auto-detects lm_head
else:
    modules_to_not_convert = []  # ← cleared when user provides ANY list!

if skip_modules is not None:
    modules_to_not_convert.extend(skip_modules)

When llm_int8_skip_modules=None (default), get_keys_to_not_convert() automatically finds and excludes lm_head and other output projection layers.

The moment the user provides any list (e.g., to protect a multimodal audio/vision tower from quantization), the auto-exclusion is disabled and lm_head gets quantized → AssertionError in bitsandbytes.

This is particularly easy to hit with multimodal models like Gemma 4 E2B-IT, where users need to explicitly skip the audio/vision towers to prevent quality degradation, but are not aware that doing so also removes the lm_head protection.

Minimal Reproducible Example

import torch
from transformers import AutoModelForImageTextToText, BitsAndBytesConfig

# ❌ Crashes: providing any list clears the lm_head default exclusion
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    llm_int8_skip_modules=["model.audio_tower"],
)
model = AutoModelForImageTextToText.from_pretrained(
    "google/gemma-4-e2b-it",
    quantization_config=bnb_config,
    device_map="auto",
)
# model.generate(...) → AssertionError in lm_head

Workaround (user must manually re-add lm_head):

# ✅ Works: lm_head added explicitly
llm_int8_skip_modules=["lm_head", "model.audio_tower"]

Suggested Fix

Change the behavior so that the user-provided list is additive rather than replacing the auto-detected defaults:

# In get_modules_to_not_convert():
modules_to_not_convert = get_keys_to_not_convert(model)  # always start with defaults

if skip_modules is not None:
    modules_to_not_convert.extend(skip_modules)  # user additions merged, not replacing

modules_to_not_convert = list(set(modules_to_not_convert))
return modules_to_not_convert

At minimum, the docstring for BitsAndBytesConfig.llm_int8_skip_modules should warn that lm_head must be included manually when this parameter is set.

Related

extent analysis

TL;DR

To fix the crash during inference, modify the get_modules_to_not_convert() function to merge user-provided skip modules with auto-detected defaults instead of replacing them.

Guidance

  • Identify the get_modules_to_not_convert() function in transformers/quantizers/base.py and modify it to always start with auto-detected defaults.
  • When a user provides a list of modules to skip, extend the default list with the user-provided modules instead of replacing the defaults.
  • Ensure that the lm_head module is included in the list of skipped modules when using BitsAndBytesConfig with llm_int8_skip_modules.
  • Update the docstring for BitsAndBytesConfig.llm_int8_skip_modules to warn users that lm_head must be included manually when this parameter is set.

Example

# Modified get_modules_to_not_convert() function
modules_to_not_convert = get_keys_to_not_convert(model)  # always start with defaults

if skip_modules is not None:
    modules_to_not_convert.extend(skip_modules)  # user additions merged, not replacing

modules_to_not_convert = list(set(modules_to_not_convert))
return modules_to_not_convert

Notes

  • This fix assumes that the get_keys_to_not_convert() function correctly auto-detects the lm_head module and other output projection layers.
  • The modified get_modules_to_not_convert() function should be tested thoroughly to ensure that it correctly handles user-provided skip modules and auto-detected defaults.

Recommendation

Apply the suggested fix to modify the get_modules_to_not_convert() function to merge user-provided skip modules with auto-detected defaults. This will prevent the crash during inference and ensure that the lm_head module is correctly excluded from quantization.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#mixed precision#training loop#device allocation#model download#tokenizer error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

transformers - ✅(Solved) Fix [BitsAndBytesConfig] Providing llm_int8_skip_modules clears the default lm_head exclusion, causing AssertionError in 4-bit inference [1 pull requests, 5 comments, 2 participants]