PR #23748: fix: wire streaming SpendLog logging for /v1/messages beta endpoint

• Repository: BerriAI/litellm
• Author: weiguangli-io
• State: open | merged: False
• Link: https://github.com/BerriAI/litellm/pull/23748

Description (problem / solution / changelog)

Summary

Fixes #23150 — /v1/messages beta endpoint streaming doesn't log SpendLogs when websearch_interception callback is enabled.

Root Cause

When websearch_interception converts a streaming request to non-streaming (stream=True → stream=False) and no agentic loop runs (i.e., the LLM doesn't use the web search tool), the response is wrapped in FakeAnthropicMessagesStreamIterator. This iterator yields pre-built SSE chunks to the client but never triggers the streaming logging handler, so async_success_handler is never called and SpendLogs are missing.

Fix

Wrap FakeAnthropicMessagesStreamIterator with BaseAnthropicMessagesStreamingIterator.async_sse_wrapper(), which:

1. Iterates through all chunks from the fake stream
2. Collects them for logging
3. After the stream completes, calls _handle_streaming_logging → _route_streaming_logging_to_handler → async_success_handler

This reuses the exact same logging machinery that real streaming responses use (e.g., Bedrock's bedrock_sse_wrapper), ensuring SpendLogs are created with correct spend, tokens, model, and user information.

Context

• @cafonseca isolated the issue to the websearch_interception callback — removing it from callbacks restored SpendLog entries
• @JiwaniZakir was going to fix the main streaming path but stepped back
• The non-websearch streaming path already works correctly (the underlying chunk_processor/async_sse_wrapper handles logging)

Test plan

• Added tests/test_litellm/llms/custom_httpx/test_fake_stream_logging.py with 3 tests:
  • test_fake_stream_wrapped_with_logging_handler — verifies the fake stream is wrapped as an async generator (not raw FakeAnthropicMessagesStreamIterator)
  • test_fake_stream_logging_handler_called — verifies _handle_streaming_logging is called after stream consumption
  • test_no_websearch_conversion_returns_none — verifies no wrapping when websearch conversion didn't happen
• All existing test_llm_http_handler.py tests pass (4/4)
• All existing websearch_interception tests pass (37/37, 1 skipped, 1 unrelated arch failure)

Changed files

• litellm/llms/custom_httpx/llm_http_handler.py (modified, +15/-1)
• tests/test_litellm/llms/custom_httpx/test_fake_stream_logging.py (added, +173/-0)

---

PR #24135: fix(proxy): defer logging until post-call guardrails complete

• Repository: BerriAI/litellm
• Author: michelligabriele
• State: closed | merged: True
• Link: https://github.com/BerriAI/litellm/pull/24135

Description (problem / solution / changelog)

guardrail_information is None in StandardLoggingPayload because logging fires before post-call guardrails write to metadata.

Non-streaming: wrapper_async stores a closure instead of calling create_task immediately. The proxy fires it in a try/finally after post_call_success_hook so the SLP is built with guardrail info.

Streaming: a closure on logging_obj is called by CSW.anext at stream end. The closure runs only guardrail hooks (not all callbacks) on the assembled response, then fires both logging handlers. This avoids behavioral changes for non-guardrail callbacks on streaming.

Relevant issues

Replaces #23929

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🐛 Bug Fix

Changes

Problem

guardrail_information is always None in StandardLoggingPayload when post-call guardrails (e.g. OpenAI Moderation) are configured. This happens because:

• Non-streaming: asyncio.create_task in wrapper_async (utils.py) fires the logging task before post_call_success_hook runs in base_process_llm_request, so the SLP is built before guardrails write to metadata.
• Streaming: logging fires at stream exhaustion in CustomStreamWrapper.__anext__ without any guardrail data from the assembled response — post_call_success_hook is never called for streaming early-return routes.

Fix

Two deferral mechanisms — same concept (store a closure, call it at the right time), different execution points.

Non-streaming (utils.py + common_request_processing.py):

1. _has_post_call_guardrails() checks if any CustomGuardrail with post_call event hook is registered
2. If true and non-streaming: set logging_obj._defer_async_logging = True
3. wrapper_async sees the flag → stores closure on logging_obj._enqueue_deferred_logging instead of asyncio.create_task. Sync callbacks fire immediately (unchanged).
4. base_process_llm_request runs post_call_success_hook (guardrails write to metadata)
5. finally block calls the stored closure → create_task fires → SLP built with guardrail info

Streaming (common_request_processing.py + streaming_handler.py):

1. If _has_post_call_guardrails and response is CustomStreamWrapper: attach _on_deferred_stream_complete closure to logging_obj
2. The closure runs only guardrail hooks — iterates litellm.callbacks, filters for CustomGuardrail instances with post_call event hook, calls their async_post_call_success_hook. This is the same pattern ProxyLogging.post_call_success_hook uses internally, but filtered to guardrails only. Non-guardrail callbacks are not called (avoids behavioral changes for streaming).
3. CSW.__anext__ at stream end: checks for closure. If set, clears it and calls it via asyncio.create_task. If not set, fires logging directly (original behavior preserved).
4. Fallthrough safety: if code reaches the inline post_call_success_hook (no early return), the closure is cleared first to prevent double invocation.

Files

Tests (17 total)

Detection (7): _has_post_call_guardrails returns correct result for post_call, pre_call, event_hook=None, list event hooks, non-guardrail callbacks, empty callbacks

Non-streaming (3): deferred flag stores and executes closure, sync callbacks fire immediately, regression test without flag

Non-streaming exception (1): deferred logging fires even if guardrail raises HTTPException (try/finally)

Streaming (6): closure defers logging, regression without closure, closure runs only guardrail hooks (not all callbacks), guardrail-modified response flows to logging, exception resilience with guardrail_blocked, transient errors don't set guardrail_blocked, production closure integration test

Changed files

• docs/my-website/docs/proxy/guardrails/custom_guardrail.md (modified, +10/-2)
• docs/my-website/sidebars.js (modified, +14/-0)
• litellm/litellm_core_utils/litellm_logging.py (modified, +14/-0)
• litellm/litellm_core_utils/streaming_handler.py (modified, +25/-11)
• litellm/proxy/_new_secret_config.yaml (modified, +25/-34)
• litellm/proxy/common_request_processing.py (modified, +356/-104)
• litellm/utils.py (modified, +31/-8)
• poetry.lock (modified, +1/-1)
• tests/test_litellm/litellm_core_utils/test_litellm_logging.py (modified, +76/-0)
• tests/test_litellm/proxy/guardrails/test_deferred_guardrail_logging.py (added, +944/-0)

---

PR #26000: fix(proxy): handle non-standard SSE frames in Anthropic passthrough logging

• Repository: BerriAI/litellm
• Author: leikaiwei
• State: open | merged: False
• Link: https://github.com/BerriAI/litellm/pull/26000

Description (problem / solution / changelog)

Summary

Some third-party Anthropic-compatible API providers (e.g. AWS Bedrock proxies, custom gateways) emit non-standard SSE frames in their streaming responses:

1. OpenAI-style [DONE] sentinel frames mixed into Anthropic SSE streams
2. Non-JSON SSE lines (comments, keep-alive pings, debug output)

These cause json.JSONDecodeError in AnthropicPassthroughLoggingHandler._build_complete_streaming_response(), which breaks the entire logging pipeline — resulting in missing spend logs, incorrect token/cost accounting, and silent failures in the web UI.

Root Cause

In _build_complete_streaming_response(), the SSE event loop only catches StopIteration/StopAsyncIteration. When a non-standard frame arrives that isn't valid JSON, the unhandled JSONDecodeError propagates up and aborts logging for the entire streaming response.

Changes

• Skip SSE events containing [DONE] control frames before attempting JSON parse
• Catch json.JSONDecodeError for malformed SSE lines and continue to the next event

Both are minimal, defensive changes — valid Anthropic SSE events are unaffected.

Related Issues

• Partially addresses #17476 (pass-through streaming callback failures)
• Related to #23150 (spend logs not recorded for streaming requests)

Test Plan

• Added unit tests for [DONE] frame handling
• Added unit tests for non-JSON SSE line handling
• Added test for mixed valid/invalid frames to verify valid events still get processed
• All existing tests in test_anthropic_passthrough_logging_handler.py still pass

Changed files

• litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py (modified, +13/-3)
• tests/test_litellm/proxy/pass_through_endpoints/llm_provider_handlers/test_anthropic_passthrough_logging_handler.py (modified, +150/-65)