Issue Station
Sign In
Tools hereToolbox

ollama - ✅(Solved) Fix gemma4:31b doesn't output some Polish characters [1 pull requests, 1 comments, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
ollama/ollama#15231•Fetched 2026-04-08 02:34:03
View on GitHub
Comments
1
Participants
1
Timeline
13
Reactions
1
Author
szmarczakszmarczak
Participants
szmarczakszmarczak
Assignees
dhiltgendhiltgen
Timeline (top)
cross-referenced ×4referenced ×2subscribed ×2assigned ×1

Fix Action

Fixed

PR fix notes

PR #15232: tokenizer: add byte fallback for SentencePiece BPE encoding

Description (problem / solution / changelog)

When BPE merging produces tokens not in the vocabulary, fall back to encoding each UTF-8 byte as <0xHH> byte tokens instead of silently dropping the character. Also teach Decode to convert <0xHH> tokens back to raw bytes.

Fixes #15229, fixes #15231

Changed files

  • model/models/gemma4/tokenizer_reference_test.go (added, +341/-0)
  • tokenizer/bytepairencoding.go (modified, +55/-14)
  • tokenizer/bytepairencoding_test.go (modified, +117/-35)

Code Example

>>> Say "ąęśćżźółń" nothing more. Do not think.
Thinking...
The user wants me to say the string "śżź" and nothing else.
They explicitly said "nothing more" and "Do not think."

Constraint Checklist & Confidence Score:
1. Say "śżź"? Yes.
2. Nothing more? Yes.
3. Do not think? (This is a stylistic instruction to the AI, usually meant to bypass internal reasoning/chain of thought, though the system still processes 
it).

Confidence Score: 5/5

Mental Sandbox:
- If I output "śżź", I follow all instructions.
- If I output "Here is the text: śżź", I violate "nothing more".
- If I include a preamble, I violate "nothing more".

Conclusion: Output exactly "śżź".
...done thinking.

śżź

---
RAW_BUFFERClick to expand / collapse

What is the issue?

>>> Say "ąęśćżźółń" nothing more. Do not think.
Thinking...
The user wants me to say the string "śżź" and nothing else.
They explicitly said "nothing more" and "Do not think."

Constraint Checklist & Confidence Score:
1. Say "śżź"? Yes.
2. Nothing more? Yes.
3. Do not think? (This is a stylistic instruction to the AI, usually meant to bypass internal reasoning/chain of thought, though the system still processes 
it).

Confidence Score: 5/5

Mental Sandbox:
- If I output "śżź", I follow all instructions.
- If I output "Here is the text: śżź", I violate "nothing more".
- If I include a preamble, I violate "nothing more".

Conclusion: Output exactly "śżź".
...done thinking.

śżź
<img width="816" height="352" alt="Image" src="https://github.com/user-attachments/assets/6ddbc092-18cb-4a20-a058-c430cb66d3f5" />

Relevant log output

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.20.0-rc0

extent analysis

TL;DR

The issue seems to be related to the AI's understanding and response to specific instructions, and the most likely fix is to ensure the AI model is correctly interpreting the "nothing more" and "Do not think" constraints.

Guidance

  • Review the AI model's constraint handling mechanism to ensure it correctly interprets stylistic instructions like "Do not think" and literal instructions like "nothing more".
  • Verify that the model's output matches the expected response "śżź" without any additional text or processing.
  • Check the model's confidence score calculation to ensure it accurately reflects the model's adherence to the given constraints.
  • Consider testing the model with similar constraint-based inputs to identify any patterns or inconsistencies in its response.

Notes

The provided issue lacks specific technical details about the AI model's architecture or training data, making it challenging to provide a more targeted solution. The guidance provided is based on the information given and may need to be adapted to the specific implementation details of the AI model.

Recommendation

Apply workaround: The current implementation seems to be working as intended based on the provided constraints, so no immediate action is required, but ongoing monitoring and testing are recommended to ensure the model continues to perform as expected.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#model compatibility#GPU setup#container setup#orchestration issue#cache issue

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING