vllm - 💡(How to fix) Fix [Feature]: Support streaming output for tool_calls arguments

🚀 The feature, motivation and pitch

... ... (APIServer pid=1) INFO 05-21 00:19:06 [logger.py:92] Generated response chatcmpl-97319e51c49a6410 (streaming delta): output: '[reasoning: get]', output_token_ids: [98628], finish_reason: None

(APIServer pid=1) INFO 05-21 00:19:06 [logger.py:92] Generated response chatcmpl-97319e51c49a6410 (streaming delta): output: '[reasoning: data]', output_token_ids: [96902], finish_reason: None

(APIServer pid=1) INFO 05-21 00:19:06 [logger.py:92] Generated response chatcmpl-97319e51c49a6410 (streaming delta): output: '[reasoning: 。]', output_token_ids: [1710], finish_reason: None

(APIServer pid=1) INFO 05-21 00:19:07 [logger.py:92] Generated response chatcmpl-97319e51c49a6410 (streaming delta): output: '[tool_calls: "content": "Artificial intelligence has rapidly transformed the way we interact with technology. Large language models, in particular, now possess the remarkable ability to understand complex instructions, generate creative content, and seamlessly integrate with external tools through function calling. This evolution allows developers to build dynamic applications that can fetch real-time data, perform calculations, and automate intricate workflows. As these models continue to improve in reasoning and efficiency, the barrier between human intent and machine execution diminishes. We are entering an era where AI acts not just as a conversational partner, but as a capable assistant that actively participates in solving real-world problems."]', output_token_ids: [29], finish_reason: None ... ...

In vLLM v0.19, regarding the last log entry shown above, why aren’t the arguments within tool_calls streamed? If the content is very long, it easily leads to request timeouts. Is it possible to add support for streaming this output?

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.