vllm - 💡(How to fix) Fix [New Model]: Add DebertaV2ForSequenceClassification (DeBERTa-v2/v3 cross-encoder / reranker)

Your current environment

vLLM main branch (post v0.9.x)

🚀 New model request

Model family: DeBERTa-v2 / DeBERTa-v3 (Microsoft)
Architecture class: DebertaV2ForSequenceClassification
HuggingFace examples:

• cross-encoder/nli-deberta-v3-small / -base / -large
• OpenAssistant/reward-model-deberta-v3-large-v2
• BAAI/bge-reranker-base
• meta-llama/Prompt-Guard-86M

Motivation

DeBERTa-v3 is one of the most widely used encoder models for reranking and NLI — popular checkpoints include:

• cross-encoder/nli-deberta-v3-small / -base / -large
• Capreolus/deberta-v3-base-msmarco
• OpenAssistant/reward-model-deberta-v3-large-v2
• meta-llama/Prompt-Guard-86M
• All microsoft/deberta-v2-* and microsoft/deberta-v3-* variants

PR #20215 attempted to add this support but has been stalled for ~10 months, is in draft state, and contains critical bugs (see below). I am proposing a clean, production-ready implementation.

Problems with PR #20215

1. Runtime crash — DebertaV2Model.forward() returns a BaseModelOutput object, but the code does hidden_states[:, 0] on that object directly → TypeError at inference time (never caught because no CI was run).
2. Performance regression — disentangled attention (c2p + p2c) is implemented with nested Python for loops over seq_len, giving O(seq_len²) Python overhead — unusable at typical sequence lengths.
3. No tensor parallelism — uses plain nn.Linear instead of ColumnParallelLinear / RowParallelLinear.

Proposed Implementation

• vllm/model_executor/models/deberta_v2.py: ~630-line implementation
  • DebertaV2ForSequenceClassification using DispatchPooler pattern
  • DebertaV2ContextPooler as a SequencePoolingMethod (CLS pooling)
  • Fully vectorized disentangled attention via torch.einsum + torch.gather (no Python loops)
  • Log-scale position bucketing supporting both DeBERTa-v2 and v3 (position_buckets config field)
  • ColumnParallelLinear / RowParallelLinear throughout for tensor parallelism
  • AutoWeightsLoader + WeightsMapper (remaps pooler.* → context_pooler.*)
• Registry, tests, docs updated following existing patterns

A draft PR is open at: https://github.com/JLiu4Coding/vllm/tree/model/deberta-v2-sequence-classification

The closest model vllm already supports.

RobertaForSequenceClassification (vllm/model_executor/models/roberta.py)

DeBERTa-v2 shares the same overall structure as RoBERTa (encoder-only transformer, CLS pooling, sequence classification head) but replaces standard multi-head self-attention with disentangled attention, where queries and keys are each split into separate content and position components.

What's your difficulty of supporting the model you want?

DeBERTa uses disentangled attention, which is architecturally incompatible with vLLM's standard attention layer. Specifically:

1. New attention mechanism — queries and keys each have a content component and a position component, producing up to three attention score terms (content-to-content, content-to-position, position-to-content). This cannot be expressed using vLLM's existing attention kernels and requires a custom implementation.

2. Relative position encoding — instead of absolute position embeddings, DeBERTa uses a shared position embedding table indexed by relative bucket distance. The bucketing logic differs between DeBERTa-v2 (linear) and DeBERTa-v3 (log-scale, controlled by the position_buckets config field).

3. Weight remapping — HuggingFace stores the classification pooler weights under pooler.* but the vLLM implementation uses context_pooler.*, requiring a WeightsMapper.

The proposed implementation resolves all three with a fully vectorized custom attention (torch.einsum + torch.gather) and AutoWeightsLoader + WeightsMapper.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.