litellm - 💡(How to fix) Fix [Bug]: Critical: Response leakage / Cross-talk between users in Redis Cluster environment on OpenShift [2 comments, 2 participants]

TL;DR

Review and adjust the configuration of LiteLLM with Redis Cluster to ensure proper request scoping and isolation between clients, especially when using shared API keys.

Guidance

• Investigate the caching mechanism in LiteLLM to determine if it's correctly scoped to individual requests or if it's possible for cache entries to be shared across requests from different clients.
• Verify that the Redis Cluster configuration is set up to handle concurrent requests correctly and that the connection pooling is properly managed to prevent requests from being mixed up.
• Check the documentation for LiteLLM and Redis Cluster to see if there are any known limitations or configuration requirements for using shared API keys in a distributed setup.
• Consider implementing a unique identifier for each request to help track and isolate responses, ensuring they are returned to the correct client.

Example

No specific code example can be provided without more details on the LiteLLM and Redis Cluster configuration, but reviewing the caching and request scoping logic in the LiteLLM Proxy component may be a good starting point.

Notes

The exact cause of the issue is unclear without more information on the LiteLLM configuration and the specific Redis Cluster setup. Further investigation into the caching and request handling mechanisms is needed to determine the best course of action.

Recommendation

Apply workaround: Implement a unique request identifier to help ensure responses are correctly matched with their originating requests, and review the LiteLLM and Redis Cluster configuration for any necessary adjustments to support shared API keys and multiple replicas.