Issue Station
Sign In
Tools hereToolbox

transformers - 💡(How to fix) Fix transformers 4.30.0 incompatible with rust [1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
huggingface/transformers#45095•Fetched 2026-04-08 01:45:13
View on GitHub
Comments
0
Participants
1
Timeline
3
Reactions
0
Author
starsgostarsgo
Participants
starsgostarsgo
Timeline (top)
closed ×1subscribed ×1unsubscribed ×1

Error Message

error: casting &T to &mut T is undefined behavior, even if the reference is unused, consider instead using an UnsafeCell error: could not compile tokenizers (lib) due to 1 previous error; 6 warnings emitted when installing tokenizers，it insists using rust , when rust install tokenizers ,this error occur

RAW_BUFFERClick to expand / collapse

error: casting &T to &mut T is undefined behavior, even if the reference is unused, consider instead using an UnsafeCell --> tokenizers-lib\src\models\bpe\trainer.rs:526:47 | 522 | let w = &words[*i] as *const _ as *mut _; | -------------------------------- casting happened here ... 526 | let word: &mut Word = &mut (*w); | ^^^^^^^^^ | = note: for more information, visit https://doc.rust-lang.org/book/ch15-05-interior-mutability.html = note: #[deny(invalid_reference_casting)] on by default

warning: hiding a lifetime that's elided elsewhere is confusing --> tokenizers-lib\src\models\unigram\model.rs:355:17 | 355 | pub fn iter(&self) -> UnigramIterator { | ^^^^^ ^^^^^^^^^^^^^^^ the same lifetime is hidden here | | | the lifetime is elided here | = help: the same lifetime is referred to in inconsistent ways, making the signature confusing help: use '_ for type paths | 355 | pub fn iter(&self) -> UnigramIterator<'_> { | ++++

warning: hiding a lifetime that's elided elsewhere is confusing --> tokenizers-lib\src\models\unigram\trie.rs:33:36 | 33 | pub fn common_prefix_search<T>(&self, iterator: T) -> TrieIterator<Label, T> | ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^ the same lifetime is hidden here | | | the lifetime is elided here | = help: the same lifetime is referred to in inconsistent ways, making the signature confusing help: use '_ for type paths | 33 | pub fn common_prefix_search<T>(&self, iterator: T) -> TrieIterator<'_, Label, T> | +++

warning: tokenizers (lib) generated 6 warnings error: could not compile tokenizers (lib) due to 1 previous error; 6 warnings emitted when installing tokenizers，it insists using rust , when rust install tokenizers ,this error occur

extent analysis

Fix Plan

To fix the error, we need to address the undefined behavior caused by casting &T to &mut T. We will use UnsafeCell as suggested by the error message. Additionally, we will fix the lifetime warnings by using the _ lifetime parameter.

Step 1: Replace casting with UnsafeCell

use std::cell::UnsafeCell;

// Replace the line causing the error
// let w = &words[*i] as *const _ as *mut _;
let w = UnsafeCell::new(&words[*i]);

// Replace the line using the casted reference
// let word: &mut Word = &mut (*w);
let word: &mut Word = unsafe { &mut *w.get() };

Step 2: Fix lifetime warnings

// Replace the line causing the warning in model.rs
// pub fn iter(&self) -> UnigramIterator {
pub fn iter(&self) -> UnigramIterator<'_> {
    // ...
}

// Replace the line causing the warning in trie.rs
// pub fn common_prefix_search<T>(&self, iterator: T) -> TrieIterator<Label, T>
pub fn common_prefix_search<T>(&self, iterator: T) -> TrieIterator<'_, Label, T> {
    // ...
}

Verification

After applying these changes, re-run the installation command for tokenizers. The error and warnings should be resolved, and the installation should complete successfully.

Extra Tips

  • When working with Rust, it's essential to understand the concept of ownership and borrowing to avoid undefined behavior.
  • The UnsafeCell type provides a way to opt-out of Rust's borrowing rules, but it should be used with caution and only when necessary.
  • Always address warnings and errors in your code to ensure the stability and security of your application.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#chain error#conversation history#tool integration#LLM response#prompt template

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

transformers - 💡(How to fix) Fix transformers 4.30.0 incompatible with rust [1 participants]