vllm - 💡(How to fix) Fix [Bug]: When function_call is used in the request and tool_choice is set to required,the system performs schema formatting in advance before the request is sent to the model for inference. [2 comments, 2 participants]

wangbairu · 2026-04-15T03:52:32Z

[vllm] Your current environment text model: Qwen3-8B serve cmd: vllm serve /home/data/Qwen3-8B \ --served-model-name Qwen3-8B \ --trust-remote-code \ --async-s… ### Your current environment ```text model: Qwen3-8B serve cmd: vllm serve /home/data/Qwen3-8B \ --served-model-name Qwen3-8B \ --trust-remote-code \ --async-scheduling \ --no-enable-prefix-caching \ --distributed-executor-backend mp \ --tensor-parallel-size 1 \ --max-model-len 40000 \ --max-num-batched-tokens 40960 \ --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \ --additional-config '{"pa_shape_list":[48,64,72,80], "weight_prefetch_config":{"enabled":true}}' \ --port 20269 \ --block-size 128 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --gpu-memory-utilization 0.9 ``` ### 🐛 Describe the bug - As a result, for the same input prompt: 1. Without formatting: the model outputs fewer tokens, with normal accuracy. 2. With formatting: inference performance is roughly halved (about 2x slower) - the code path: vllm/tool_parsers/utils.py ```python tool_choice: "required" if tool_choice == "required": return _get_json_schema_from_tools(tools) ``` - The output 1. With formating: ```txt e2e: 1.499s tokens: 48 [9640, 262, 341, 286, 330, 606, 788, 330, 1836, 69364, 756, 286, 330, 13786, 788, 341, 310, 330, 1074, 788, 330, 99407, 99683, 39426, 23836, 756, 310, 330, 1028, 788, 330, 17, 15, 17, 21, 12, 15, 18, 12, 16, 24, 698, 286, 456, 262, 456, 60, 151645] '[\n {\n "name": "search_weather",\n "parameters": {\n "loc": "旅顺口区",\n "date": "2026-03-19"\n }\n }\n]' ``` 2. Without formating: ```txt e2e: 0.655s tokens: 38 [151657, 198, 4913, 606, 788, 330, 1836, 69364, 497, 330, 16370, 788, 5212, 1074, 788, 330, 99407, 99683, 39426, 23836, 497, 330, 1028, 788, 330, 17, 15, 17, 21, 12, 15, 18, 12, 16, 24, 95642, 151658, 151645] ' \n{"name": "search_weather", "arguments": {"loc": "旅顺口区", "date": "2026-03-19"}}\n ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.

vllm2026-04-15 03:52:32

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

vllm-project/vllm#39867•Fetched 2026-04-17 08:24:06

View on GitHub

Comments

Participants

Timeline

Reactions

Author

wangbairu

Participants

sfeng33

wangbairu

Timeline (top)

commented ×2subscribed ×2labeled ×1

Code Example

model: Qwen3-8B

serve cmd:
vllm serve /home/data/Qwen3-8B \    
         --served-model-name Qwen3-8B \    
         --trust-remote-code \    
         --async-scheduling \   
         --no-enable-prefix-caching \    
         --distributed-executor-backend mp \   
         --tensor-parallel-size 1 \    
         --max-model-len 40000 \    
         --max-num-batched-tokens 40960 \    
         --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \    
         --additional-config '{"pa_shape_list":[48,64,72,80], "weight_prefetch_config":{"enabled":true}}' \    
         --port 20269 \    
         --block-size 128 \    
         --enable-auto-tool-choice \   
         --tool-call-parser hermes  \   
         --gpu-memory-utilization 0.9

---

tool_choice: "required"
    if tool_choice == "required":
        return _get_json_schema_from_tools(tools)

---

e2e: 1.499s
tokens: 48
[9640, 262, 341, 286, 330, 606, 788, 330, 1836, 69364, 756, 286, 330, 13786, 788, 341, 310, 330, 1074, 788, 330, 99407, 99683, 39426, 23836, 756, 310, 330, 1028, 788, 330, 17, 15, 17, 21, 12, 15, 18, 12, 16, 24, 698, 286, 456, 262, 456, 60, 151645]
'[\n    {\n        "name": "search_weather",\n        "parameters": {\n            "loc": "旅顺口区",\n            "date": "2026-03-19"\n        }\n    }\n]'

---

e2e: 0.655s
tokens: 38
[151657, 198, 4913, 606, 788, 330, 1836, 69364, 497, 330, 16370, 788, 5212, 1074, 788, 330, 99407, 99683, 39426, 23836, 497, 330, 1028, 788, 330, 17, 15, 17, 21, 12, 15, 18, 12, 16, 24, 95642, 151658, 151645]
'<tool_call>\n{"name": "search_weather", "arguments": {"loc": "旅顺口区", "date": "2026-03-19"}}\n</tool_call>

RAW_BUFFERClick to expand / collapse

Your current environment

model: Qwen3-8B

serve cmd:
vllm serve /home/data/Qwen3-8B \    
         --served-model-name Qwen3-8B \    
         --trust-remote-code \    
         --async-scheduling \   
         --no-enable-prefix-caching \    
         --distributed-executor-backend mp \   
         --tensor-parallel-size 1 \    
         --max-model-len 40000 \    
         --max-num-batched-tokens 40960 \    
         --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \    
         --additional-config '{"pa_shape_list":[48,64,72,80], "weight_prefetch_config":{"enabled":true}}' \    
         --port 20269 \    
         --block-size 128 \    
         --enable-auto-tool-choice \   
         --tool-call-parser hermes  \   
         --gpu-memory-utilization 0.9

🐛 Describe the bug

As a result, for the same input prompt:

Without formatting: the model outputs fewer tokens, with normal accuracy.
With formatting: inference performance is roughly halved (about 2x slower)

the code path: vllm/tool_parsers/utils.py

tool_choice: "required"
    if tool_choice == "required":
        return _get_json_schema_from_tools(tools)

The output

With formating:

e2e: 1.499s
tokens: 48
[9640, 262, 341, 286, 330, 606, 788, 330, 1836, 69364, 756, 286, 330, 13786, 788, 341, 310, 330, 1074, 788, 330, 99407, 99683, 39426, 23836, 756, 310, 330, 1028, 788, 330, 17, 15, 17, 21, 12, 15, 18, 12, 16, 24, 698, 286, 456, 262, 456, 60, 151645]
'[\n    {\n        "name": "search_weather",\n        "parameters": {\n            "loc": "旅顺口区",\n            "date": "2026-03-19"\n        }\n    }\n]'

Without formating:

e2e: 0.655s
tokens: 38
[151657, 198, 4913, 606, 788, 330, 1836, 69364, 497, 330, 16370, 788, 5212, 1074, 788, 330, 99407, 99683, 39426, 23836, 497, 330, 1028, 788, 330, 17, 15, 17, 21, 12, 15, 18, 12, 16, 24, 95642, 151658, 151645]
'<tool_call>\n{"name": "search_weather", "arguments": {"loc": "旅顺口区", "date": "2026-03-19"}}\n</tool_call>

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The issue can be addressed by investigating the impact of formatting on the model's inference performance, potentially by modifying the tool_call_parser or adjusting the compilation_config.

Guidance

Investigate the hermes tool call parser to see if it's introducing additional overhead when formatting is present.
Compare the output tokens and performance metrics with and without formatting to identify any patterns or correlations.
Consider adjusting the compilation_config to optimize performance for formatted inputs.
Review the vllm/tool_parsers/utils.py code path to ensure that the tool_choice logic is not contributing to the performance discrepancy.

Example

No code snippet is provided as the issue does not explicitly imply a specific code change.

Notes

The issue seems to be related to the interaction between the model, the hermes tool call parser, and the formatting of the input. Further investigation is needed to determine the root cause of the performance discrepancy.

Recommendation

Apply workaround: Investigate and optimize the hermes tool call parser and compilation_config to improve performance with formatted inputs, as the issue seems to be related to the interaction between these components.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #autograd error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - 💡(How to fix) Fix [Bug]: When function_call is used in the request and tool_choice is set to required,the system performs schema formatting in advance before the request is sent to the model for inference. [2 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

TRENDING

vllm - 💡(How to fix) Fix [Bug]: When function_call is used in the request and tool_choice is set to required,the system performs schema formatting in advance before the request is sent to the model for inference. [2 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Code Example

Your current environment

🐛 Describe the bug

Before submitting a new issue...

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

RELATED_DISCOVERY

TRENDING