litellm - 💡(How to fix) Fix [Bug]: responses api: Session continuation only works after 10 secs

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

For non OpenAI providers (which lacks of a native response API endpoint) you have to wait about 10 secs (the time that LiteLLM waits to add logs to S3) to be able to retrieve old messages and keep the conversations.

This makes unusable for chat apps, or some fast processing apps.

Steps to Reproduce

Make a Responses request with for example a Gemini model Keep that response ID and before 10 secs try to keep the conversation Fail Repeat with OpenAI Works (because OpenAI store messages in her side)

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.85.1

Twitter / LinkedIn details

No response