What is the issue?

All three BERT-derived embedding models I tested in Ollama produce incorrect embeddings for text containing combining diacritics, while the same models via the HuggingFace transformers pipeline behave correctly. The defect is not model-specific — it appears to be a systematic loss of BasicTokenizer's strip_accents preprocessing somewhere in the HF→gguf conversion path (likely in llama.cpp's convert_hf_to_gguf.py, surfaced to end-users via Ollama).

Unrelated words containing diacritics collapse to cosine similarity ≈ 1.0 because they all tokenize to [CLS] [UNK] [SEP]. Same-word ASCII-vs-diacritic pairs (e.g. Hokkaidō ↔ Hokkaido) no longer cluster.

This affects every production RAG/search/clustering deployment using these models on any multilingual or Unicode-containing corpus.

Affected models (verified)

The "Unrelated diac↔diac" column is the headline: unrelated words like Hokkaidō and Éire are embedded as near-identical vectors.

Reproduction

Minimal repro (Python, requires ollama package and the models pulled locally):

import ollama, numpy as np

def embed(model, text):
    r = ollama.embeddings(model=model, prompt=text)
    v = np.array(r["embedding"])
    return v / np.linalg.norm(v)

def cos(a, b): return float(a @ b)

for model in ["mxbai-embed-large", "nomic-embed-text", "all-minilm"]:
    h_diac  = embed(model, "Hokkaidō")
    h_ascii = embed(model, "Hokkaido")
    eire    = embed(model, "Éire")
    print(f"{model}: Hokkaidō↔Hokkaido={cos(h_diac, h_ascii):.3f}  "
          f"Hokkaidō↔Éire={cos(h_diac, eire):.3f}")

Expected: same-word pair ≈ 1.0, unrelated pair ≈ low. Actual: unrelated pair ≈ 0.9–1.0.

Full verification pipeline and 147k-pair empirical analysis:

• Repro script: https://github.com/emmaleonhart/latent-space-cartography/blob/main/scripts/verify_tokenizer_divergence.py
• Writeup with full tables + mechanism walkthrough: https://emmaleonhart.github.io/latent-space-cartography/

Likely root cause

The upstream HuggingFace tokenizers for these models use BasicTokenizer with do_lower_case=True, which (by default) applies NFD normalization and strips combining diacritical marks before WordPiece. Verified directly against mixedbread-ai/mxbai-embed-large-v1:

Hokkaidō  ->  ['hokkaido']   # identical to Hokkaido
Éire      ->  ['e', '##ire'] # identical to Eire
Zürich    ->  ['zurich']     # identical to Zurich

The gguf-converted tokenizer used by Ollama does not appear to apply this preprocessing. Raw diacritics hit WordPiece, miss the vocab, and fall back to [UNK]. Because [UNK] is a fixed token, every short diacritic-only string that can't be WordPiece-split produces [CLS] [UNK] [SEP], which embeds to a single attractor point.

The fix is almost certainly in llama.cpp/convert_hf_to_gguf.py — preserving the BasicTokenizer.strip_accents behavior through conversion. I'd expect it to require writing a flag into the gguf metadata and honoring it at tokenization time, but I haven't traced the conversion code in detail and the maintainers will know better.

Scope

Not tested exhaustively, but the mechanism implies any BERT-derived embedding model converted via this pipeline is affected. The three models above cover the most common production choices for local RAG.

Relevant log output

OS

Windows

GPU

Nvidia

CPU

AMD

Ollama version

0.17.1