Issue Station
Sign In
Tools hereToolbox

vllm - 💡(How to fix) Fix [Feature]: How to disable chat template when using vllm serve [1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#38809•Fetched 2026-04-08 02:34:45
View on GitHub
Comments
0
Participants
1
Timeline
1
Reactions
0
Author
sleepwalker2017sleepwalker2017
Participants
sleepwalker2017sleepwalker2017
Timeline (top)
labeled ×1
RAW_BUFFERClick to expand / collapse

🚀 The feature, motivation and pitch

Hi all, I'm deploying qwen 3.5 2B model, and when we using the hf model ,we use it like this, no template is used.

`

    model_inputs = processor(
        text           = prompt_text,
        images         = images,
        return_tensors = "pt",
    )

    # ──────────────────────────────────
    input_ids      = model_inputs["input_ids"].to(device)
    attention_mask = model_inputs["attention_mask"].to(device)
    pixel_values   = model_inputs["pixel_values"].to(device)
    image_grid_thw = model_inputs["image_grid_thw"]          # 保持 CPU
    if image_grid_thw.dim() == 1:
        image_grid_thw = image_grid_thw.unsqueeze(0)


      if use_cache_mode:
          new_token_ids = _greedy_generate(
              model          = model,
              input_ids      = input_ids,
              attention_mask = attention_mask,
              pixel_values   = pixel_values,
              image_grid_thw = image_grid_thw,
              max_new_tokens = 4,
          )`

As far as I know, the openai completion api doesn't apply templates but it doesn't support image inputs.

The openai chat api supports image inputs, but always apply a chat template, which results in the fact that, the input after the template always has some template tokens different from the huggingface usage format.

Do you have has suggestions about that? Thank you!

Alternatives

No response

Additional context

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The issue can be addressed by exploring alternative APIs or models that support both image inputs and custom input formats without requiring templates.

Guidance

  • Investigate the OpenAI API documentation to see if there are any endpoints or parameters that allow for custom input formats without templates.
  • Consider using a different model or API that supports image inputs and does not require templates, such as the Hugging Face Transformers library.
  • Look into preprocessing the input data to remove or modify the template tokens that are being added by the OpenAI Chat API.
  • Explore the possibility of using a combination of APIs or models to achieve the desired functionality, such as using the OpenAI Completion API for text generation and a separate model for image processing.

Example

No code snippet is provided as the issue does not contain enough information to create a specific example.

Notes

The solution may depend on the specific requirements of the project, such as the type of images being used and the desired output format. Additionally, the OpenAI API terms of service and usage guidelines should be reviewed to ensure that any solution complies with their policies.

Recommendation

Apply a workaround by exploring alternative APIs or models that support both image inputs and custom input formats without requiring templates, as the current API being used has limitations that are causing the issue.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#memory management#API rate limit#retriever error#indexing error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING