litellm - 💡(How to fix) Fix [Bug]: "Response with id '{response_id}' not found" in /ui/chat with vllm [2 comments, 2 participants]

Error occurred while generating model response. Please try again. Error: Error: 404 litellm.BadRequestError: Hosted_vllmException - {"error":{"message":"Response with id 'resp_998bca5d44e22037' not found.","type":"invalid_request_error","param":"response_id","code":404}}. Received Model Group=Qwen/Qwen3.6-35B-A3B-FP8 Available Model Group Fallbacks=None

TL;DR

The issue can likely be resolved by modifying the /ui/chat implementation to not reuse response_ids or by ensuring that the backend vLLM stores and retrieves responses correctly.

Guidance

• Investigate the Responses API flow in /ui/chat to determine why it's attempting to reuse a response_id that cannot be found by the backend vLLM.
• Verify that the vLLM backend is correctly storing and retrieving responses for multi-turn conversations.
• Check the LiteLLM configuration to ensure that the hosted_vllm backend is properly set up to handle continuous chat sessions.
• Consider adding logging or debugging statements to the /ui/chat code to track the response_id generation and usage.

Example

No code snippet is provided as the issue does not contain sufficient information to create a specific example.

Notes

The issue seems to be specific to the /ui/chat implementation and the interaction with the hosted_vllm backend. The fact that the LiteLLM Playground works correctly suggests that the issue is not with the vLLM model itself, but rather with how /ui/chat is using it.

Recommendation

Apply a workaround to modify the /ui/chat implementation to not reuse response_ids or ensure correct response storage and retrieval by the backend vLLM, as the root cause of the issue appears to be related to this aspect of the code.