PR #38883: [Transformers v5] Skip ColBERT jina HF comparison due to remote code incompatibility

• Repository: vllm-project/vllm
• Author: Lidang-Jiang
• State: closed | merged: False
• Link: https://github.com/vllm-project/vllm/pull/38883

Description (problem / solution / changelog)

Summary

The test_colbert_hf_comparison[jina] test fails under Transformers v5 because jinaai/jina-colbert-v2 uses custom remote code (jinaai/xlm-roberta-flash-implementation) with its own flash_attn integration that is incompatible with v5's dtype handling changes.

Root cause: Transformers v5 deprecated torch_dtype in favor of dtype. The remote code model's custom XLMRobertaFlashConfig handles torch_dtype explicitly in __init__, but v5's from_pretrained may resolve the model dtype differently, causing flash_attn to receive float32 tensors (it only supports fp16/bf16), producing NaN outputs. Even when forced to bfloat16, the numerical results differ significantly from v4 (max_diff=0.21 vs tolerance=0.01).

This skips the jina backend in test_colbert_hf_comparison when running under Transformers v5, until the model's remote code is updated for v5 compatibility.

• Not a duplicate: No open PR addresses #38737. Verified with gh pr list --search "38737" and gh pr list --search "ColBERTJina".
• AI assistance was used (Claude) for root cause analysis. All changes reviewed by human submitter.

Fixes #38737

$ CUDA_VISIBLE_DEVICES=3 python -m pytest tests/models/language/pooling/test_colbert.py::test_colbert_hf_comparison[jina] -v --timeout=600

tests/models/language/pooling/test_colbert.py::test_colbert_hf_comparison[jina] FAILED

=================================== FAILURES ===================================
_______________________ test_colbert_hf_comparison[jina] _______________________
tests/models/language/pooling/test_colbert.py:149: in _assert_embeddings_close
    torch.testing.assert_close(
E   AssertionError: Embedding mismatch for text 0

======================= 1 failed in 43.07s ========================

Diagnosis confirmed the HF reference model produces different embeddings:

• Default loading (bf16 + flash_attn): max_diff=0.21 vs tolerance 0.01
• Eager attention + float32: max_diff=0.17 (different code path, still incompatible)
• Model parameters load correctly (no NaN in weights, 1/294 near-zero params)
• flash_attn assertion: assert qkv.dtype in [torch.float16, torch.bfloat16] fails when model loaded in float32

$ CUDA_VISIBLE_DEVICES=2 python -m pytest tests/models/language/pooling/test_colbert.py::test_colbert_hf_comparison[jina] -v --timeout=60

tests/models/language/pooling/test_colbert.py::test_colbert_hf_comparison[jina] SKIPPED [100%]

======================= 1 skipped in 0.42s ========================

Test plan

• test_colbert_hf_comparison[jina] correctly skipped under Transformers v5
• Other ColBERT backends (bert, modernbert, lfm2) unaffected by this change
• ruff check and ruff format pass
• All pre-commit hooks pass

Changed files

• tests/models/language/pooling/test_colbert.py (modified, +11/-0)

PR #39176: fix(test): recompute Jina ColBERT rotary inv_freq cleared by transformers v5 weight loader

• Repository: vllm-project/vllm
• Author: ieBoytsov
• State: closed | merged: True
• Link: https://github.com/vllm-project/vllm/pull/39176

Description (problem / solution / changelog)

Purpose

Summary

• Transformers 5.0 introduced a new weight materialization system that clears non-persistent buffers (persistent=False) during from_pretrained
• The Jina ColBERT model's RotaryEmbedding registers inv_freq as a non-persistent buffer, so it gets wiped to uninitialized memory after weight loading
• This caused NaN outputs in the test_colbert_hf_comparison[jina] HF reference model, failing the embedding comparison
• Fix: recompute inv_freq for all rotary embedding modules after loading the HF model in _load_hf_model

Fix: https://github.com/vllm-project/vllm/issues/38737

Test plan

• pytest tests/models/language/pooling/test_colbert.py::test_colbert_hf_comparison[jina] passes

Test Result

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Changed files

• tests/models/language/pooling/test_colbert.py (modified, +8/-0)