Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

I have LiteLLM Proxy deployed (self-managed) and use it for streaming completions via /v1/messages endpoint. The client uses anthropic.AsyncAnthropic => proxy => Vertex. However, the client observes periodic large spikes of httpx.ReadTimeout errors. Timeouts spike every 3600s, which matches _DEFAULT_TTL_FOR_HTTPX_CLIENTS. When I monkeypatch this TTL in my proxy deployment to 5400s, the spikers shift to every 90m, conclusively implicating the httpx client TTL mechanism in this bug.

Spikes in client-experienced httpx.ReadTimeout with default settings (TTL is 60m) <img width="697" height="339" alt="Image" src="https://github.com/user-attachments/assets/39a2bc35-a524-410d-b19e-a985c6fc9bc3" />

Spikes in client-experienced httpx.ReadTimeout after TTL is monkeypatched to 90m <img width="699" height="338" alt="Image" src="https://github.com/user-attachments/assets/0c03fe24-278a-4d94-8dd9-0dda329412b7" />

Note that we confirmed that the center of the spikes is aligned with exactly the minute that a new revision of the proxy service was deployed. So if the revision was deployed at 11:45pm, we would see spikes at 12:45am, 1:45am, etc.

I also tried an experiment with setting router_settings.client_ttl to 5400 but this did NOT change the behavior, readtimeout was still experienced in intervals of 3600s, confirming that this is NOT the source of the issue.

Researching this bug, Claude and I identified this code snippet: https://github.com/BerriAI/litellm/blob/7e4e4545c5da303ed4586a3231ccb3369d9c63d4/litellm/llms/custom_httpx/http_handler.py#L725-L729 When the httpx client is evicted at the end of the TTL, the close() method is called explictly which propagates all the way into the underlying tcp connection. So in-flight responses have their connection cut out from under them. From the client's perspective, they are left waiting for additional bytes on the streaming response, but never observe any, eventually failing at the end of the read timeout. More discussion at https://gist.github.com/micahjsmith/c990aeb1505639466d50e14f88b09b82

The likely fix is to remove the custom __del__ behavior, which doesn't seem necessary, as all the underlying resources should be cleaned up by Python GC. The PR which introduces this code https://github.com/BerriAI/litellm/pull/2665 does not include any justification for the inclusion of the explicit __del__ handler.

A workaround is to monkeypatch _DEFAULT_TTL_FOR_HTTPX_CLIENTS to a very large number

Steps to Reproduce

1. Start litellm proxy
2. Stream completions from /v1/messages under high concurrency for a 3h period (for example). In the client, I use httpx connect/read/write/pool timeouts of 5/30/5/5.
3. Observe in the client that httpx.ReadTimeout is observed with spikes in occurences at the 1hr and 2hr marks.

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

1.81.12-stable.1

Twitter / LinkedIn details

No response