PR #37873: [Bugfix] RoBERTa position_id accumulation in CUDA graph padding region

• Repository: vllm-project/vllm
• Author: yanghui1-arch
• State: closed | merged: True
• Link: https://github.com/vllm-project/vllm/pull/37873

Description (problem / solution / changelog)

Fix a crash that affected all RoBERTa-based embedding models (BAAI/bge-m3, XLM-RoBERTa, stsb-roberta, bge-reranker-v2-m3) when CUDA graphs are enabled. After approximately max_position_embeddings / 2 requests the server crashes with: Assertion 'index out of bounds: 0 <= tmp25 < 8194' failed.

• gpu_model_runner keeps a persistent GPU buffer self.positions that is reused across every request. Each request refreshes only the first num_scheduled_tokens entries via copy_to_gpu; the remaining padding slots [num_scheduled_tokens : num_input_tokens_padded] are not reset.
• RoBERTa models call replace_roberta_positions outside the CUDA graph (before BertModel.forward), which does an in-place position_ids += padding_idx + 1 on the full padded tensor, including the stale padding slots. Because those slots are never reset by copy_to_gpu, each request adds another +(padding_idx + 1) to them.

For BAAI/bge-m3 (max_position_embeddings = 8194, padding_idx = 1, offset = 2) with short sentences (6 tokens) padded to 8:

padding slot value after K requests  =  V_init + 2K
overflow when                         >= 8194
K                                     = (8194 - V_init) / 2  ≈  3999

Fixes #37868

Related #37648 #37868

Purpose

Test Plan

Reproduce the bug

CUDA_LAUNCH_BLOCKING=1 vllm serve BAAI/bge-m3 --port 9001 \
  --hf-overrides '{"architectures": ["BgeM3EmbeddingModel"]}' \
  --runner pooling

# Send 10000 sequential requests
for i in $(seq 1 10000); do
  response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:9001/pooling \
    -H "Content-Type: application/json" \
    -d '{"model": "BAAI/bge-m3", "task": "embed",
         "input": ["test sentence number '$i'"]}')
  if [ "$response" != "200" ]; then
    echo "FAILED at index $i (HTTP $response)"; break
  fi
done

Existing tests

pytest tests/models/language/pooling/test_bge_m3.py -v -s
pytest tests/models/language/pooling/test_embedding.py -v -s -k "stsb-roberta"
pytest tests/models/language/pooling/test_multi_vector_retrieval.py -v -s
pytest tests/models/language/pooling/test_scoring.py -v -s

Test Result

Tested on RTX 5090 (Blackwell, sm_120), vLLM 0.18.1rc1.dev38+ga16133a0f:

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Changed files

• vllm/v1/worker/gpu_model_runner.py (modified, +2/-0)

PR #37884: [Bugfix] Fix RoBERTa position_ids accumulation on CUDA graph padding

• Repository: vllm-project/vllm
• Author: he-yufeng
• State: closed | merged: True
• Link: https://github.com/vllm-project/vllm/pull/37884

Description (problem / solution / changelog)

Purpose

Fix a crash in all RoBERTa-based pooling/embedding models (BGE-M3, XLM-RoBERTa, stsb-roberta, bge-reranker-v2-m3) when CUDA graphs are enabled. After ~4000 sequential requests the server dies with an out-of-bounds position embedding index.

Root cause: replace_roberta_positions() did an in-place position_ids += padding_idx + 1 on the persistent GPU positions buffer. The model runner refreshes only the first num_scheduled_tokens entries via copy_to_gpu each request — the remaining CUDA-graph padding slots keep their stale values. Every request adds another +(padding_idx + 1) to those slots, and eventually the values exceed max_position_embeddings.

For BAAI/bge-m3 (padding_idx=1, offset=2) with short inputs padded to 8 tokens, overflow happens after (8194 - V_init) / 2 ≈ 4000 requests.

Fix: move the padding_idx + 1 offset into RobertaEmbedding.forward as a non-in-place add (position_ids + offset instead of position_ids += offset). This computes the correct positions each call without mutating the persistent buffer. Also fix the same in-place += pattern in the transformers LegacyMixin.

Fixes #37648 Fixes #37868

Related: #37873 (alternative fix that zeroes the padding region in _preprocess)

Test Plan

Reproduce the bug (before fix)

vllm serve BAAI/bge-m3 --port 9001 \
  --hf-overrides '{"architectures": ["BgeM3EmbeddingModel"]}'

for i in $(seq 1 5000); do
  resp=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:9001/pooling \
    -H "Content-Type: application/json" \
    -d '{"model": "BAAI/bge-m3", "task": "embed", "input": ["test '$i'"]}')
  [ "$resp" != "200" ] && echo "FAIL at $i (HTTP $resp)" && break
done

Existing tests

pytest tests/models/language/pooling/test_bge_m3.py -v -s
pytest tests/models/language/pooling/test_embedding.py -v -s -k "stsb-roberta"
pytest tests/models/language/pooling/test_scoring.py -v -s

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

Changed files

• vllm/model_executor/models/roberta.py (modified, +8/-24)
• vllm/model_executor/models/transformers/legacy.py (modified, +4/-2)