vllm - 💡(How to fix) Fix [RFC]: Add support for Cohere's /v2/chat HTTP API entry point

Motivation.

vLLM does not currently support the Cohere Chat v2 api. While similar to chat completions, this api has a number of different fields, including notably, citations present within the assistant messages which should then be rendered into the prompt. This functionality is important to many of our users and so it would be needed for them to be able to use cohere models with vLLM directly.

Proposed Change.

The proposed change is to add a cohere/v2/chat entry point that renders the request using the Melody library's prompt rendering functions. The flow would be very similar to the create_chat_completion function, but with render_chat calling the melody library to produce the rendered chat template prompt. Common logic could also be potentially refactored into a shared class. Output parsing logic is already in place and would not need to change.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.