System Info

• transformers version: 5.10.2
• tokenizers version: 0.22.2
• Python version: 3.11.2
• Platform: Linux (Penn State HPC, RHEL 8.10)
• PyTorch version: not relevant (tokenizer-only repro)

Who can help?

@ArthurZucker @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

CodeLlama tokenizer drops one leading space on decode round-trip (v5 regression)

Reproduction

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf")
for s in ["   leading spaces", "    indented_line"]:
    decoded = tok.decode(tok.encode(s, add_special_tokens=False))
    print(f"{'OK' if decoded == s else 'BUG'}: {s!r} -> {decoded!r}")

Output on transformers 5.10.2 + tokenizers 0.22.2:

BUG: '   leading spaces' -> '  leading spaces'
BUG: '    indented_line' -> '   indented_line'

The same script on transformers 4.55.4 (last v4) round-trips correctly:

OK: '   leading spaces' -> '   leading spaces'
OK: '    indented_line' -> '    indented_line'

Exactly one leading space is lost on round-trip; trailing whitespace, internal whitespace, and other characters round-trip correctly.

Scope

All CodeLlama variants tested are affected:

Root cause

src/transformers/models/code_llama/tokenization_code_llama.py, lines 163–164:

self._tokenizer.decoder = decoders.Sequence(
    [decoders.Replace("▁", " "), decoders.ByteFallback(), decoders.Fuse(), decoders.Strip(content=" ", left=1)]
)

The trailing decoders.Strip(content=" ", left=1) unconditionally strips one leading space on decode. This was intended to remove the synthetic SentencePiece ▁ prefix introduced by the normalizer (Prepend ▁ → Replace " " → ▁). But it also eats one leading space when the user's input itself starts with a space, because by that point in the decode pipeline the synthetic prefix and user-provided leading spaces are indistinguishable.

v4's CodeLlamaTokenizerFast had different decode handling that distinguished these. The v5 migration that removed the Fast class (#40936 "rm slow tokenizers", #42563 "Refactor-tokenization-more", #42894 "use TokenizersBackend") landed this Strip-step decoder for all CodeLlama checkpoints.

Impact

This silently corrupts whitespace-sensitive workloads on CodeLlama. In code generation specifically, where indentation is semantically meaningful:

input  = "    if x > 0:\n        return x"
output = tok.decode(tok.encode(input, add_special_tokens=False))
# output = "   if x > 0:\n        return x"   # 3 spaces instead of 4 — silent indentation corruption

Any pipeline using encode → decode for round-trip text comparison (BLEU/chrF/exact-match metrics, RL reward signals on code tokens, fine-tuning data validation) produces silently-wrong values when leading whitespace is present.

Suggested fix direction

Two possible shapes; deferring to maintainers:

1. Track the synthetic prefix. Instead of Strip(left=1), the decoder should only strip a leading space when the corresponding encoded sequence began with the synthetic ▁ prefix and not with a user-provided leading space. This requires the decoder to know whether add_prefix_space would have fired.
2. Replace with v4-style decode logic. Port whatever CodeLlamaTokenizerFast.decode did in v4 (which round-tripped correctly) into the current implementation. Happy to implement once a direction is preferred.

Expected behavior

tok.decode(tok.encode(s)) should round-trip leading whitespace, as it does on transformers v4 and on most other tokenizers in v5 (Qwen, StarCoder2, Phi-3 all round-trip correctly on the same test strings).

This is a tokenization-correctness bug. Code generation and any whitespace-sensitive evaluation pipeline silently produce wrong values on CodeLlama checkpoints.