Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Bug]: BedrockGuardrail streaming hook drops HTTPException after HTTP 200, producing broken SSE stream [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#26387•Fetched 2026-04-24 10:36:36
View on GitHub
Comments
0
Participants
1
Timeline
4
Reactions
0
Author
orolegaorolega
Participants
orolegaorolega
Timeline (top)
labeled ×3cross-referenced ×1

When a Bedrock guardrail hard-blocks a streamed response, async_post_call_streaming_iterator_hook raises an HTTPException inside the async generator body. By the time the guardrail runs, the HTTP 200 header has already been written to the client — the exception cannot be converted to a 4xx response. Instead it propagates up, truncating the SSE stream mid-flight. The client receives a partial or malformed response with no indication of why.

Expected behaviour

The client receives a well-formed SSE error frame followed by [DONE]:

data: {"error": {"message": "Violated guardrail policy", "code": 400, "type": "guardrail_violation"}}

data: [DONE]

Actual behaviour

The HTTPException escapes the generator. The stream is truncated and the client has no structured error to parse.

Root cause

async_post_call_streaming_iterator_hook has no try/except around the make_bedrock_api_request / asyncio.gather calls. Both the parallel INPUT+OUTPUT path (should_validate_input=True) and the output-only path (should_validate_input=False) are affected.

GuardrailInterventionNormalStringError (soft-block / disable_exception_on_block=True) is not affected — it is already caught by inner try/except blocks.

Fix

Wrap the Bedrock validation block in a try/except HTTPException and yield two in-band SSE frames instead of letting the exception escape.

Error Message

header has already been written to the client — the exception cannot be The client receives a well-formed SSE error frame followed by [DONE]: data: {"error": {"message": "Violated guardrail policy", "code": 400, "type": "guardrail_violation"}} client has no structured error to parse. two in-band SSE frames instead of letting the exception escape. 4. Observe the SSE stream terminates abruptly without an error frame.

Root Cause

Root cause

Fix Action

Fixed

PR fix notes

PR #26388: fix: bedrock guardrail sse streaming exception

Description (problem / solution / changelog)

Relevant issues

Fixes #26387

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.

  • Branch creation CI run Link:

  • CI run for the last commit Link:

  • Merge / cherry-pick CI run Links:

Screenshots / Proof of Fix

Before fix: a streaming request blocked by a hard-block Bedrock policy (e.g. topic or content policy with action: BLOCKED) terminates the SSE stream with no error frame — the client receives a truncated response.

After fix: the client receives a well-formed error frame followed by [DONE]:

data: {"error": {"message": "Violated guardrail policy", "code": 400, "type": "guardrail_violation"}}

data: [DONE]

Type

🐛 Bug Fix ✅ Test

Changes

  • litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py Wrap the Bedrock validation block in async_post_call_streaming_iterator_hook with try/except HTTPException. When a hard-block exception would otherwise escape the generator after the HTTP 200 header has been written, it is caught and converted to two in-band SSE frames: a JSON error payload and [DONE]. Both the parallel INPUT+OUTPUT path (post_call hook) and the output-only path (during_call / pre_call hook) are covered. Soft-block handling via GuardrailInterventionNormalStringError is unchanged.

  • tests/test_litellm/proxy/guardrails/guardrail_hooks/test_bedrock_guardrails.py Add two regression tests:

    • test_streaming_hook_converts_http_exception_to_sse_error_frame_parallel_path — parallel INPUT+OUTPUT path (post_call, should_validate_input=True)
    • test_streaming_hook_converts_http_exception_to_sse_error_frame_output_only_path — output-only path (during_call, should_validate_input=False)
<img width="947" height="78" alt="image" src="https://github.com/user-attachments/assets/c8a1fdce-0236-475f-ae54-14ff4c6c7400" /> <img width="1240" height="90" alt="image" src="https://github.com/user-attachments/assets/ed439685-70f4-451f-9d32-50b4d08ba1de" /> <img width="832" height="155" alt="image" src="https://github.com/user-attachments/assets/d57bcd92-3da0-4622-968c-200a0c0fd30e" />

Changed files

  • litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py (modified, +94/-74)
  • tests/test_litellm/proxy/guardrails/guardrail_hooks/test_bedrock_guardrails.py (modified, +149/-0)
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Description

When a Bedrock guardrail hard-blocks a streamed response, async_post_call_streaming_iterator_hook raises an HTTPException inside the async generator body. By the time the guardrail runs, the HTTP 200 header has already been written to the client — the exception cannot be converted to a 4xx response. Instead it propagates up, truncating the SSE stream mid-flight. The client receives a partial or malformed response with no indication of why.

Expected behaviour

The client receives a well-formed SSE error frame followed by [DONE]:

data: {"error": {"message": "Violated guardrail policy", "code": 400, "type": "guardrail_violation"}}

data: [DONE]

Actual behaviour

The HTTPException escapes the generator. The stream is truncated and the client has no structured error to parse.

Root cause

async_post_call_streaming_iterator_hook has no try/except around the make_bedrock_api_request / asyncio.gather calls. Both the parallel INPUT+OUTPUT path (should_validate_input=True) and the output-only path (should_validate_input=False) are affected.

GuardrailInterventionNormalStringError (soft-block / disable_exception_on_block=True) is not affected — it is already caught by inner try/except blocks.

Fix

Wrap the Bedrock validation block in a try/except HTTPException and yield two in-band SSE frames instead of letting the exception escape.

Steps to Reproduce

  1. Configure a BedrockGuardrail with event_hook: post_call (or during_call).
  2. Set disable_exception_on_block to false (the default).
  3. Send a streaming request whose output triggers a hard-block policy (e.g. topic policy, content policy — anything that returns action: BLOCKED).
  4. Observe the SSE stream terminates abruptly without an error frame.

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.83.7

Twitter / LinkedIn details

No response

extent analysis

TL;DR

Wrap the Bedrock validation block in a try/except HTTPException to catch and handle the exception, yielding a well-formed SSE error frame instead of letting it escape.

Guidance

  • Identify the async_post_call_streaming_iterator_hook function and add a try/except block around the make_bedrock_api_request and asyncio.gather calls to catch HTTPException.
  • Yield two in-band SSE frames: one with the error details and another with [DONE] to indicate the end of the stream.
  • Verify that the client receives a well-formed SSE error frame followed by [DONE] when a hard-block policy is triggered.
  • Test both the parallel INPUT+OUTPUT path (should_validate_input=True) and the output-only path (should_validate_input=False) to ensure the fix works for both scenarios.

Example

try:
    # make_bedrock_api_request and asyncio.gather calls
except HTTPException as e:
    yield {"error": {"message": "Violated guardrail policy", "code": 400, "type": "guardrail_violation"}}
    yield "[DONE]"

Notes

This fix assumes that the HTTPException is the only exception that needs to be caught and handled. Additional error handling may be necessary depending on the specific requirements of the application.

Recommendation

Apply the workaround by wrapping the Bedrock validation block in a try/except HTTPException to catch and handle the exception, as this will provide a well-formed SSE error frame to the client instead of truncating the stream.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#batch processing#GPU compatibility#latency issue#model loading

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING