litellm - 💡(How to fix) Fix bug(streaming): fast_path in async_streaming_data_generator breaks tool-call continuation — client receives XML instead of text (introduced v1.87.0, PR #28289)

Summary

Since v1.87.0 (PR #28289), the fast_path optimization in async_streaming_data_generator causes tool-call continuation streaming to break when using /v1/chat/completions with Claude models via Bedrock.

Symptom: After a tool-call round-trip, the model's follow-up response is rendered as raw <tool_response> / <tool_use> XML in the client instead of a normal text response.

Affected versions: v1.87.0, v1.87.1, v1.88.0, v1.88.1
Working version: v1.86.2

Root Cause

PR #28289 introduced a fast_path in proxy/common_request_processing.py:

fast_path = (
    not caps.has_streaming_chunk_override
    and not caps.has_guardrail
    and not cost_injection_enabled
)

if fast_path:
    yield serialize_chunk(chunk)
    continue  # ← skips async_post_call_streaming_hook entirely

With a default LiteLLM config (no custom callbacks, no guardrails, cost injection disabled), fast_path is always True. The async_post_call_streaming_hook is never called.

In v1.86.2, this hook was always called (even if it returned early via its own fast-path check). Some side-effect of that call — likely related to str_so_far accumulation or streaming context bookkeeping — was required for correct tool-call continuation. Without it, the Bedrock/Anthropic streaming handler loses context and the model output is garbled.

Reproduction

Minimal version check

# Run inside LiteLLM container:
# docker run --rm --entrypoint /bin/sh litellm/litellm:v1.87.1 \
#   -c "/app/.venv/bin/python3 -c '...'"

import sys; sys.path.insert(0, "/app")
from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing
from litellm.proxy.utils import ProxyLogging
import inspect

src = inspect.getsource(ProxyBaseLLMRequestProcessing.async_streaming_data_generator)
has_fast_path = "fast_path" in src

caps = ProxyLogging._callback_capabilities()
fast_path_active = not caps.has_streaming_chunk_override and not caps.has_guardrail

print(f"fast_path present: {has_fast_path}")       # False on v1.86.2, True on v1.87+
print(f"fast_path active:  {fast_path_active}")    # True on all versions with default config

End-to-end

1. Start LiteLLM proxy v1.87+ with a Claude/Bedrock model
2. Send a chat completions request with tools that triggers a tool call
3. Include the role: "tool" result in the follow-up request
4. Expected: model returns normal text summarizing the tool result
5. Actual: model returns raw XML (<tool_response>...</tool_response> or <tool_use>...</tool_use>) as chat content

Version matrix

Config

litellm_settings:
  stream: true
  drop_params: true
  modify_params: true
  # No custom callbacks, no guardrails → fast_path is always True

Workaround

Downgrade to v1.86.2.

A potential fix: ensure that any state required for tool-call streaming continuation is preserved even when fast_path=True, or scope the fast path to explicitly exclude responses following a role: "tool" turn.

References

• PR #28289 (introduced fast_path): https://github.com/BerriAI/litellm/pull/28289
• Related perf PR #28720 (v1.88): https://github.com/BerriAI/litellm/pull/28720