PR #13339: Fix: Responses API Redis session timing for conversation context

• Repository: BerriAI/litellm
• Author: jatorre
• State: closed | merged: False
• Link: https://github.com/BerriAI/litellm/pull/13339

Description (problem / solution / changelog)

Fix: Responses API Redis session timing for conversation context

Description

This PR fixes a critical timing issue in the Responses API where conversation context was not being maintained between requests when using Redis cache. The issue was caused by batch processing delays in storing session data.

Problem

When using the Responses API with previous_response_id, the conversation context was not available immediately for subsequent requests. This was particularly problematic for:

• Multi-turn conversations requiring context
• High-throughput applications needing immediate session availability
• ChatGPT-like interfaces requiring seamless conversation flow

Solution

Added a Redis-first session storage mechanism that:

1. Stores session data immediately after response generation
2. Retrieves session data directly from Redis on subsequent requests
3. Falls back gracefully to existing enterprise/database logic if Redis is unavailable

Changes

• Added _patch_store_session_in_redis() method for immediate session storage
• Added _patch_get_session_from_redis() method for fast session retrieval
• Modified pre_completion_batch_processing() to check Redis first
• All changes are backward compatible and fail gracefully

Testing

Tested with multiple providers:

• ✅ Anthropic Claude 3.5 Sonnet
• ✅ DeepSeek Chat
• ✅ Google Gemini 2.0 Flash

Test Script

# First request
response1 = requests.post("/v1/responses", json={
    "model": "claude-3-5-sonnet",
    "input": "My name is Alice",
    "max_tokens": 100
})
response_id = response1.json()["id"]

# Second request with context
response2 = requests.post("/v1/responses", json={
    "model": "claude-3-5-sonnet",
    "input": "What's my name?",
    "previous_response_id": response_id,
    "max_tokens": 100
})
# Now correctly returns "Alice" instead of "I don't know"

Configuration Required

litellm_settings:
  cache: true  # Required
  cache_params:
    type: "redis"
    host: "localhost"
    port: 6379

Related Issues

• Fixes #12364 - Responses API conversation context timing issue
• Related to #12640 - Native OpenAI Responses API support

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Breaking change
• Documentation update

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have tested with multiple providers
• Changes are backward compatible
• Fails gracefully when Redis is not available

Additional Notes

This is implemented as a patch with clear comments indicating it's a temporary fix. The upstream team may want to integrate this more deeply into the core session handling logic. The patch:

• Only activates when Redis cache is configured
• Does not interfere with existing enterprise session handlers
• Adds minimal overhead (single Redis get/set operation)
• Uses 24-hour TTL for session data

Changed files

• REDIS_SESSION_PATCH.md (added, +255/-0)
• RESPONSES_API_TEST_README.md (added, +100/-0)
• litellm/responses/litellm_completion_transformation/handler.py (modified, +18/-0)
• litellm/responses/litellm_completion_transformation/streaming_iterator.py (modified, +80/-7)
• litellm/responses/litellm_completion_transformation/transformation.py (modified, +82/-0)
• responses_api_config.yaml (added, +36/-0)
• test_redis_session_patch.py (added, +92/-0)
• test_responses_api.py (added, +229/-0)