Description

When a streaming request receives a 429 (RESOURCE_EXHAUSTED) error mid-stream and no fallbacks are configured, the litellm proxy process enters an infinite async generator loop, pinning CPU at 100% indefinitely. The process becomes completely unresponsive to new HTTP requests and must be killed manually.

This is a regression/edge case introduced by PR #22375, which correctly added the != 429 exemption to allow the router to handle 429 via MidStreamFallbackError. However, when fallbacks=None, there is no exit path — the error loops silently.

Environment

• litellm version: 1.83.9
• Provider: Vertex AI (vertex_ai/gemini-3.1-pro-preview, vertex_location: global)
• Deployment: litellm proxy via LaunchDaemon, num_retries: 0, no fallbacks configured

Steps to Reproduce

1. Configure litellm proxy with a Vertex AI model and no fallbacks:

   model_list:
     - model_name: gemini-pro
       litellm_params:
         model: vertex_ai/gemini-3.1-pro-preview
         vertex_project: my-project
         vertex_location: global
   
   router_settings:
     num_retries: 0
     allowed_fails: 0
     cooldown_time: 0
     # fallbacks: NOT configured

2. Send a long streaming request that triggers a mid-stream 429 from Vertex AI (e.g. hit QPM limit during active generation)
3. Observe: process CPU goes to 100% and stays there; all subsequent HTTP requests time out

Root Cause

In streaming_handler.py, _handle_stream_fallback_error wraps 429 in MidStreamFallbackError (intentionally, per PR #22375):

# streaming_handler.py ~L2303
if (
    mapped_status_code is not None
    and 400 <= mapped_status_code < 500
    and mapped_status_code != 429   # ← 429 is exempted from direct raise
):
    raise mapped_exception

raise MidStreamFallbackError(...)  # 429 ends up here

In router.py, stream_with_fallbacks catches MidStreamFallbackError and calls async_function_with_fallbacks_common_utils. With no fallbacks configured, this raises the error again as fallback_error:

# router.py ~L1875
except Exception as fallback_error:
    verbose_router_logger.error(f"Fallback also failed: {fallback_error}")
    raise fallback_error  # re-raises MidStreamFallbackError

This re-raised MidStreamFallbackError propagates back through the nested _wrap_streaming_iterator_with_enrichment / async_post_call_streaming_iterator_hook chain in proxy/utils.py. Each layer catches and re-raises, but the async generator is never properly exhausted or closed, causing the event loop to spin at 100% CPU.

Fix Applied (local workaround)

Removing the != 429 exemption so 429 is raised directly like other 4xx errors:

# streaming_handler.py
if (
    mapped_status_code is not None
    and 400 <= mapped_status_code < 500
    # removed: and mapped_status_code != 429
):
    raise mapped_exception

With this change, the 429 is raised immediately as a RateLimitError, the downstream consumer (OpenClaw in our case) receives the error correctly, applies its own cooldown logic, and retries — no CPU hang.

Trade-off: this disables mid-stream fallback on 429 for users who have fallbacks configured. A better fix might be: if fallbacks=None (or fallback also returns 429), raise directly instead of re-raising MidStreamFallbackError.

Related Issues

• PR #22375 — introduced the != 429 exemption (the root cause of this edge case)
• Issue #23707 — Vertex AI 429 silent failure during streaming (different symptom)
• Issue #20870 — Fallbacks on streaming with Gemini 429 (different scenario)