vllm - 💡(How to fix) Fix [Bug]: ReRank API online inference doesn't work well with given template [3 comments, 2 participants]

Your current environment

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

Describe the bug

I am serving Qwen/Qwen3-Reranker-4B with vllm serve and a custom --chat-template intended for Qwen3-Reranker scoring. The template is successfully loaded at startup, but during inference:

1. The reranking scores are identical to the case where no template is specified.
2. The logged prompt does not appear to be the rendered/template-expanded prompt.
3. In debug / warning logs, I can see the template source being loaded, but the Jinja variables (for example {{ messages | ... }}) are still printed literally instead of being rendered into the final prompt.

This makes it look like the score template is loaded but not actually applied during /rerank or /score inference.

According to the docs/examples, Qwen3-Reranker should support score templates in vLLM, and the official example shows using vllm serve ... --runner pooling --chat-template ... for online scoring/reranking.

How I start my online vllm

CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Reranker-4B \
  --host 127.0.0.1 \
  --port 8021 \
  --runner pooling \
  --hf_overrides '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' \
  --chat-template qwen3_reranker.jinja \
  --enable-log-requests

My request is below:

import requests

url = "http://127.0.0.1:8021/v1/rerank"
payload = {
    "model": "Qwen/Qwen3-Reranker-4B",
    "query": "星之卡比",
    "documents": [
        "key: kirby-hero-20260310-234352.png_1408x768\n\ndesc: A cute pink Kirby character from Nintendo game, round pink ball with big eyes, cheerful expression, simple cartoon style, on pastel pink background, kawaii style, clean illustration"
    ]
}
resp = requests.post(url, json=payload, timeout=120)
print(resp.status_code)
print(resp.text)

What I saw in logs:

No template infos at all, and the ** scores is exactly the same as no template start up.**

Expected behavior

I would expect one of the following:

The score template is actually rendered and applied during reranking/scoring, which should make the final prompt differ from the no-template case. The logged prompt should reflect the rendered prompt (or there should be a documented way to inspect the rendered prompt). If score templates are not currently applied to /rerank in this setup, the docs should clarify that.

Why I think this may be a bug

The docs/examples indicate that:

Qwen3-Reranker supports score templates in vLLM. There is an official online score example using vllm serve, --runner pooling, and --chat-template. The Qwen3-Reranker example template is documented.

However, in my setup the template seems to be loaded but has no observable effect on prompt rendering or scores.

Additional notes

This is not about Chat Completions API. I understand Qwen3-Reranker should be used with /score or /rerank, not /v1/chat/completions. The main issue is that --chat-template appears to be ignored or not observable in the rerank/score execution path. If this is intended behavior and the logged prompt is expected to be the pre-rendered/raw input, could the docs clarify how to verify that the score template is actually applied?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.