vllm - 💡(How to fix) Fix [Feature]: Server overloaded response [1 comments, 2 participants]

TL;DR

The vllm server should be modified to return a "server_overloaded" response to clients when it is overloaded, similar to other inference providers like azure-openai.

Guidance

• Investigate the vllm server's current error handling mechanisms to determine how to implement a "server_overloaded" response.
• Consider adding a load monitoring system to detect when the server is overloaded and trigger the response.
• Review the documentation and chatbot responses to ensure that the new feature is properly documented and communicated to users.
• Research how other inference providers, such as azure-openai, implement their "server_overloaded" responses to gather inspiration and best practices.

Notes

The implementation details of the "server_overloaded" response are not specified, and it is unclear what the exact requirements are for this feature.

Recommendation

Apply workaround: Implement a custom solution to return a "server_overloaded" response when the vllm server is overloaded, as there is no clear indication that this feature is already available or planned.