level : "ERROR", message : "litellm.proxy.proxy_server.async_data_generator(): Exception occured - litellm.BadRequestError: BedrockException - internalServerException {"message":"The system encountered an unexpected error during processing. Try your request again."}", stacktrace : "Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1915, in anext async for chunk in self.completion_stream: ...<71 lines>... return processed_chunk File "/usr/local/lib/python3.13/site-packages/litellm/llms/bedrock/chat/invoke_handler.py", line 1680, in aiter_bytes message = self._parse_message_from_event(event) File "/usr/local/lib/python3.13/site-packages/litellm/llms/bedrock/chat/invoke_handler.py", line 1699, in _parse_message_from_event raise BedrockError( ...<6 lines>... ) litellm.llms.bedrock.common_utils.BedrockError: internalServerException {"message":"The system encountered an unexpected error during processing. Try your request again."}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/litellm/proxy/proxy_server.py", line 5123, in async_data_generator async for chunk in proxy_logging_obj.async_post_call_streaming_iterator_hook( ...<32 lines>... yield f"data: {str(e)}\n\n" File "/usr/local/lib/python3.13/site-packages/litellm/proxy/utils.py", line 2131, in async_post_call_streaming_iterator_hook async for chunk in current_response: yield chunk File "/usr/local/lib/python3.13/site-packages/litellm/integrations/custom_logger.py", line 459, in async_post_call_streaming_iterator_hook async for item in response: yield item File "/usr/local/lib/python3.13/site-packages/litellm/integrations/custom_logger.py", line 459, in async_post_call_streaming_iterator_hook async for item in response: yield item File "/usr/local/lib/python3.13/site-packages/litellm/integrations/custom_logger.py", line 459, in async_post_call_streaming_iterator_hook async for item in response: yield item [Previous line repeated 1 more time] File "/usr/local/lib/python3.13/site-packages/litellm/proxy/hooks/responses_id_security.py", line 270, in async_post_call_streaming_iterator_hook async for chunk in response: ...<7 lines>... yield chunk File "/usr/local/lib/python3.13/site-packages/litellm/integrations/custom_logger.py", line 459, in async_post_call_streaming_iterator_hook async for item in response: yield item File "/usr/local/lib/python3.13/site-packages/litellm/integrations/custom_logger.py", line 459, in async_post_call_streaming_iterator_hook async for item in response: yield item File "/usr/local/lib/python3.13/site-packages/litellm/integrations/custom_logger.py", line 459, in async_post_call_streaming_iterator_hook async for item in response: yield item [Previous line repeated 5 more times] File "/usr/local/lib/python3.13/site-packages/litellm/router.py", line 1563, in anext return await self._async_generator.anext() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.13/site-packages/litellm/router.py", line 1568, in stream_with_fallbacks async for item in model_response: yield item File "/usr/local/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 2135, in anext raise mapped_exception File "/usr/local/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 2099, in anext mapped_exception = exception_type( model=self.model, ...<3 lines>... extra_kwargs={}, ) File "/usr/local/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2398, in exception_type raise e File "/usr/local/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 1080, in exception_type raise BadRequestError( ...<4 lines>... ) litellm.exceptions.BadRequestError: litellm.BadRequestError: BedrockException - internalServerException {"message":"The system encountered an unexpected error during processing. Try your request again."}",

Fix Plan

To fix the issue, we need to update the exception_mapping_utils.py file to correctly map the internalServerException to a retryable error.

Here are the steps:

• Update the exception_type function in exception_mapping_utils.py to check for internalServerException in the error message, regardless of the status code.
• Raise a ServiceUnavailableError instead of BadRequestError when internalServerException is encountered.

Example code:

def exception_type(e, model):
    # ... existing code ...
    if "internalServerException" in str(e):
        raise ServiceUnavailableError(model=model)
    # ... existing code ...

Additionally, consider updating the botocore version to the latest one, as the issue might be resolved in newer versions.

Verification

To verify the fix, repeat the steps to reproduce the issue and check that litellm.ServiceUnavailableError is raised instead of litellm.BadRequestError.

Extra Tips

• Make sure to test the fix thoroughly to ensure it doesn't introduce any regressions.
• Consider adding a retry mechanism for ServiceUnavailableError to handle temporary server-side errors.
• Keep the botocore version up to date to ensure you have the latest fixes and features.