vllm - 💡(How to fix) Fix [Feature]: Support n_positions config field for nomic_bert models to enable inference beyond max_position_embeddings [1 comments, 1 participants]

🚀 The feature, motivation and pitch

Support n_positions config field for nomic_bert like models to enable inference beyond max_position_embeddings

nomic-embed-text-v1.5 publishes n_positions: 8192 in its config.json to signal that RoPE frequencies are precomputed to 8192 tokens, enabling inference beyond the training length of 2048. vLLM currently has no awareness of this field and always derives max context from max_position_embeddings, capping at 2048. This RFE requests that vLLM recognise n_positions for nomic_bert model types and use it as the effective max context length.

Config.json https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/blob/main/config.json

"n_positions": 8192,              /← TRUE max positions (8192, not 2048!)
"max_position_embeddings": 2048,  ← what vLLM reads by default (misleading)
"max_trained_positions": 2048,    ← trained up to 2048

"rope_parameters": {
    "rope_theta": 1000.0,          ← RoPE base frequency
    "rope_type": "default"        ← standard RoPE (no dynamic scaling)
},
"rotary_emb_base": 1000,          ← confirms RoPE is active
"rotary_emb_fraction": 1.0,        ← RoPE applied to 100% of head dims
"rotary_emb_interleaved": false,
"rotary_emb_scale_base": null,     ← no trained scaling factor set
"rotary_scaling_factor": null,    ← no scaling factor set

1.When model_type == "nomic_bert" and n_positions is present, use n_positions as the effective max context length instead of max_position_embeddings 2. No --rope-scaling flag should be required — the precomputed frequency buffer already covers the full range 3. --rope-theta should still be respected or auto-derived from rotary_emb_base in config

Alternatives

Current workaround:

--max-model-len 8192 \
--rope-scaling '{"type": "dynamic", "factor": 4.0}' \
--rope-theta 1000 \
--trust-remote-code

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.