What happened?

When a Vertex AI (Gemini), Anthropic, or SageMaker streaming response arrives as fragmented SSE chunks, handle_accumulated_json_chunk concatenates every chunk into self.accumulated_json and calls json.loads() on the entire accumulated string after each chunk. If the JSON is incomplete, JSONDecodeError is caught and the next chunk triggers another full parse of the now-larger string.

This is O(n²) in total work for n bytes of streaming data. For large responses (long code generation, large tool-call payloads), the accumulated string grows to megabytes and json.loads — a single CPython C call that holds the GIL — blocks for seconds at a time.

In a uvicorn single-process async deployment (the default proxy configuration), this freezes the entire asyncio event loop. The liveness probe handler never gets a turn to run, the probe times out, and kubelet restarts the pod.

We confirmed this with 7 consecutive py-spy thread dumps spanning ~70 seconds, all showing the same state:

• MainThread (active+gil): stuck in json.loads → json.decoder.decode → json.decoder.raw_decode, called from handle_accumulated_json_chunk
• All 15 other threads: idle, waiting for work

Affected code paths

All three are copy-paste of the same pattern:

self.accumulated_json += message          # unbounded string growth
try:
    _data = json.loads(self.accumulated_json)  # re-parse entire blob every chunk
    self.accumulated_json = ""
    return self.chunk_parser(chunk=_data)
except json.JSONDecodeError:
    return None  # keep accumulating → next chunk re-parses even more

OpenAI is not affected — its iterator does a single json.loads per chunk with no accumulation.

Why it's dangerous

1. Unbounded accumulation — no cap on accumulated_json size
2. O(n²) total CPU — every chunk re-parses the entire buffer, not just the new bytes
3. GIL-blocking — json.loads is a single C call; it cannot yield to the event loop
4. Event loop starvation — in the default single-process uvicorn deployment, a multi-second json.loads call prevents all HTTP handlers (including health/liveness probes) from running

Steps to Reproduce

1. Deploy LiteLLM proxy with a Vertex AI Gemini model (or Anthropic/SageMaker)
2. Send a streaming request that produces a large response (e.g., long code generation, or a response with large tool-call JSON payloads — several MB total)
3. Ensure the SSE transport fragments the JSON across multiple chunks (this happens naturally at network boundaries; more likely with larger responses)
4. Observe: the proxy becomes unresponsive for 10+ seconds during the streaming response; liveness probes fail; under Kubernetes, the pod gets restarted

To confirm via profiling:

py-spy dump --pid <proxy_pid>

You'll see MainThread stuck in json.loads called from handle_accumulated_json_chunk.

Relevant log output

# 7 consecutive py-spy thread dumps (70 seconds), all identical:

Thread 1 (active+gil): "MainThread"
  json.loads (json/__init__.py)
  json.decoder.JSONDecoder.decode (json/decoder.py)
  json.decoder.JSONDecoder.raw_decode (json/decoder.py)
  handle_accumulated_json_chunk (vertex_and_google_ai_studio_gemini.py:3071)
  ...

# All 15 other threads: idle (waiting for work in ThreadPoolExecutor/AnyIO)

Related issues

• #13505 — Streaming CPU is 4-5x non-streaming (this bug is a significant contributor)
• #20268 — Sync next() blocking event loop during streaming (same symptom class)
• #24788 — Sync call blocking event loop → pod restarts (same symptom class)
• #16562 (closed) — Fix that expanded handle_accumulated_json_chunk usage, making this O(n²) path trigger more frequently

Possible fix directions

1. Collect chunks in a list, join only once — append to list[str], only "".join() + json.loads() when a heuristic suggests completeness (e.g., balanced braces, or trailing } / ])
2. Incremental JSON parser — use a library like ijson to parse as bytes arrive
3. Offload to a thread — run the parse in asyncio.to_thread() so it doesn't block the event loop (addresses the liveness probe issue but not the O(n²) CPU waste)
4. Cap accumulated buffer size — fail loudly rather than silently accumulating unbounded data

Options 1+3 combined would address both the CPU waste and the event loop starvation.

Component: SDK (litellm Python package) + Proxy (the proxy deployment is where liveness probe failures manifest)

LiteLLM version: v1.83.7 (also confirmed on litellm_internal_staging at v1.83.9-nightly)