vllm - 💡(How to fix) Fix Support SentenceTransformer Dense projection layers for embedding models (stella_en_1.5B_v5) [1 participants]

Summary

stella_en_1.5B_v5 (and other SentenceTransformer models with a 2_Dense_* projection layer) cannot be served correctly via vLLM's --runner pooling mode. vLLM only loads weights from the root model.safetensors and ignores the modules.json / 2_Dense_* subdirectories that SentenceTransformer models use for linear projection layers.

Impact

stella_en_1.5B_v5 is the highest-performing open embedding model in the 1–2B parameter range on MTEB retrieval (nDCG@10: 61.01 — beats text-embedding-004's 55.70 and e5-mistral-7b's 56.89). It's widely used for RAG/semantic search pipelines.

Without the projection layer, vLLM returns raw 1536-dim mean-pool vectors instead of the correct normalized 1024-dim embeddings. This produces embeddings that do not match sentence-transformers output and likely degrades retrieval quality significantly.

Reproduction

vllm serve dunzhang/stella_en_1.5B_v5 \
  --runner pooling \
  --trust-remote-code \
  --dtype bfloat16 \
  --override-pooler-config '{"pooling_type": "MEAN"}'

The server starts and returns embeddings, but they are 1536-dim instead of 1024-dim, and do not match sentence-transformers output because 2_Dense_1024/model.safetensors is never loaded.

Expected behavior

vLLM should detect the modules.json in SentenceTransformer models and load any Dense projection modules, applying them as part of the pooling step.

Related

• #10119 — stella not supported (closed as not planned, June 2025)
• #22614 — unmerged PR attempting to add generic ST Dense projection loading (closed stale, Nov 2025)

Workaround

Currently the only correct way to serve stella is via sentence-transformers directly or Infinity (ARM64 image not available). Sentence-transformers gives ~46 vecs/s vs vLLM's ~2950 vecs/s — a 64x throughput gap that makes stella impractical for high-volume pipelines.

Request

Either:

1. Revive and merge a cleaned-up version of #22614
2. Add a SentenceTransformerDensePooler to the vLLM pooling infrastructure that reads modules.json and loads projection layers
3. Or provide a supported path to pass custom projection weights via --override-pooler-config