PR #26750: feat(responses): support cursor-based stream resume on retrieve

• Repository: BerriAI/litellm
• Author: chenyzan
• State: open | merged: False
• Link: https://github.com/BerriAI/litellm/pull/26750

Description (problem / solution / changelog)

Relevant issues

Related bug report: #26762

Follow-up to #26671, retargeted to the public OSS staging branch and updated to fix the synchronous retrieve streaming gap identified during review.

This PR adds support for OpenAI-style client.responses.retrieve(response_id, stream=True, starting_after=N) across LiteLLM's retrieve path.

Before this change:

• the proxy retrieve endpoint forwarded stream / starting_after query params for provider-backed GET requests
• the async retrieve path could open an SSE stream and return a ResponsesAPIStreamingIterator
• the sync get_responses(stream=True) path still issued a normal GET and then treated the SSE body like a non-streaming JSON response

This PR closes that gap and adds regression coverage for the sync and proxy paths.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link: after PR creation

• CI run for the last commit
  Link: after final push

• Merge / cherry-pick CI run
  Links: N/A (maintainer-owned)

Screenshots / Proof of Fix

Added regression coverage for:

• async retrieve streaming returning a ResponsesAPIStreamingIterator
• sync retrieve streaming returning a SyncResponsesAPIStreamingIterator
• sync HTTPHandler.get(..., stream=True) opening an actual streaming GET request
• proxy forwarding of stream / starting_after query params on GET /v1/responses/{response_id}
• proxy validation for non-integer starting_after

Local targeted verification:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .venv/bin/pytest -p pytest_asyncio.plugin \
  tests/test_litellm/responses/test_get_responses_stream_resume.py \
  tests/test_litellm/proxy/response_api_endpoints/test_endpoints.py \
  -x -vv

Result: 13 passed

Type

🆕 New Feature 🐛 Bug Fix

Changes

• add sync streaming GET support to HTTPHandler.get()
• return SyncResponsesAPIStreamingIterator from get_responses(..., stream=True) instead of treating the SSE body as a non-streaming response
• preserve the existing async retrieve streaming behavior
• add sync-path regression tests for responses retrieve stream resume
• add proxy endpoint tests for query-param forwarding and invalid starting_after handling

Changed files

• litellm/llms/azure/responses/transformation.py (modified, +15/-4)
• litellm/llms/base_llm/responses/transformation.py (modified, +11/-0)
• litellm/llms/custom_httpx/http_handler.py (modified, +81/-8)
• litellm/llms/custom_httpx/llm_http_handler.py (modified, +119/-5)
• litellm/llms/manus/responses/transformation.py (modified, +6/-0)
• litellm/llms/openai/responses/transformation.py (modified, +11/-1)
• litellm/llms/volcengine/responses/transformation.py (modified, +5/-0)
• litellm/proxy/response_api_endpoints/endpoints.py (modified, +22/-0)
• litellm/responses/main.py (modified, +35/-7)
• tests/test_litellm/llms/custom_httpx/test_credential_leak_prevention.py (modified, +64/-0)
• tests/test_litellm/proxy/response_api_endpoints/test_endpoints.py (modified, +81/-10)
• tests/test_litellm/responses/test_get_responses_stream_resume.py (added, +279/-0)