litellm - 💡(How to fix) Fix [Bug]: httpx.ConnectTimeout [3 comments, 3 participants]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

A bug happened!

litellm-1 | 10:21:23 - LiteLLM Proxy:ERROR: utils.py:2513 - LiteLLM Prisma Client Exception get_generic_data: All connection attempts failed litellm-1 | 10:21:27 - LiteLLM Proxy:ERROR: utils.py:2513 - LiteLLM Prisma Client Exception get_generic_data: All connection attempts failed litellm-1 | 10:21:28 - LiteLLM Proxy:WARNING: utils.py:4567 - Spend tracking - DB connection error writing spend logs, retry 1/3. logs_count=10, error=All connection attempts failed litellm-1 | Execution of job "update_spend (trigger: interval[0:00:13], next run at: 2026-03-12 10:21:28 CST)" skipped: maximum number of running instances reached (1) litellm-1 | 10:21:40 - LiteLLM Proxy:ERROR: utils.py:3955 - Prisma DB reconnect failed (3 consecutive). reason=db_health_watchdog_connection_error error= litellm-1 | Execution of job "update_spend (trigger: interval[0:00:13], next run at: 2026-03-12 10:21:41 CST)" skipped: maximum number of running instances reached (1) litellm-1 | Execution of job "ProxyConfig.add_deployment (trigger: interval[0:00:30], next run at: 2026-03-12 10:21:46 CST)" skipped: maximum number of running instances reached (1) litellm-1 | Execution of job "update_spend (trigger: interval[0:00:13], next run at: 2026-03-12 10:21:54 CST)" skipped: maximum number of running instances reached (1) litellm-1 | 10:22:06 - LiteLLM Proxy:ERROR: utils.py:4773 - Error in spend logs queue monitor: litellm-1 | Traceback (most recent call last): litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions litellm-1 | yield litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request litellm-1 | resp = await self._pool.handle_async_request(req) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request litellm-1 | raise exc from None litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request litellm-1 | response = await connection.handle_async_request( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | pool_request.request litellm-1 | ^^^^^^^^^^^^^^^^^^^^ litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request litellm-1 | raise exc litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request litellm-1 | stream = await self._connect(request) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection.py", line 124, in _connect litellm-1 | stream = await self._network_backend.connect_tcp(**kwargs) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp litellm-1 | return await self._backend.connect_tcp( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<5 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp litellm-1 | with map_exceptions(exc_map): litellm-1 | ~~~~~~~~~~~~~~^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/contextlib.py", line 162, in exit litellm-1 | self.gen.throw(value) litellm-1 | ~~~~~~~~~~~~~~^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions litellm-1 | raise to_exc(exc) from exc litellm-1 | httpcore.ConnectTimeout litellm-1 | litellm-1 | The above exception was the direct cause of the following exception: litellm-1 | litellm-1 | Traceback (most recent call last): litellm-1 | File "/usr/lib/python3.13/site-packages/litellm/proxy/utils.py", line 4760, in _monitor_spend_logs_queue litellm-1 | await update_spend_logs_job( litellm-1 | ...<3 lines>... litellm-1 | ) litellm-1 | File "/usr/lib/python3.13/site-packages/litellm/proxy/utils.py", line 4672, in update_spend_logs_job litellm-1 | await ProxyUpdateSpend.update_spend_logs( litellm-1 | ...<5 lines>... litellm-1 | ) litellm-1 | File "/usr/lib/python3.13/site-packages/litellm/proxy/utils.py", line 4581, in update_spend_logs litellm-1 | _raise_failed_update_spend_exception( litellm-1 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ litellm-1 | e=e, start_time=start_time, proxy_logging_obj=proxy_logging_obj litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/litellm/proxy/utils.py", line 4806, in _raise_failed_update_spend_exception litellm-1 | raise e litellm-1 | File "/usr/lib/python3.13/site-packages/litellm/proxy/utils.py", line 4548, in update_spend_logs litellm-1 | await prisma_client.db.litellm_spendlogs.create_many( litellm-1 | data=batch_with_dates, skip_duplicates=True litellm-1 | ) litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/actions.py", line 20795, in create_many litellm-1 | resp = await self._client._execute( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<7 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/client.py", line 651, in _execute litellm-1 | return await self._engine.query(builder.build(), tx_id=self._tx_id) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/engine/query.py", line 244, in query litellm-1 | return await self.request( litellm-1 | ^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<4 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/engine/http.py", line 119, in request litellm-1 | resp = await self.session.request(method, url, **kwargs) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/_async_http.py", line 28, in request litellm-1 | return Response(await self.session.request(method, url, **kwargs)) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1540, in request litellm-1 | return await self.send(request, auth=auth, follow_redirects=follow_redirects) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send litellm-1 | response = await self._send_handling_auth( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<4 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth litellm-1 | response = await self._send_handling_redirects( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<3 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects litellm-1 | response = await self._send_single_request(request) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request litellm-1 | response = await transport.handle_async_request(request) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 393, in handle_async_request litellm-1 | with map_httpcore_exceptions(): litellm-1 | ~~~~~~~~~~~~~~~~~~~~~~~^^ litellm-1 | File "/usr/lib/python3.13/contextlib.py", line 162, in exit litellm-1 | self.gen.throw(value) litellm-1 | ~~~~~~~~~~~~~~^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions litellm-1 | raise mapped_exc(message) from exc litellm-1 | httpx.ConnectTimeout litellm-1 | litellm-1 | Execution of job "update_spend (trigger: interval[0:00:13], next run at: 2026-03-12 10:22:07 CST)" skipped: maximum number of running instances reached (1) litellm-1 | Execution of job "ProxyConfig.add_deployment (trigger: interval[0:00:30], next run at: 2026-03-12 10:22:16 CST)" skipped: maximum number of running instances reached (1) litellm-1 | Execution of job "ProxyConfig.get_credentials (trigger: interval[0:00:30], next run at: 2026-03-12 10:22:17 CST)" skipped: maximum number of running instances reached (1) litellm-1 | Encountered exc 404: {"is_panic":false,"message":"Transaction API error: Transaction not found. Transaction ID is invalid, refers to an old closed transaction Prisma doesn't have information about anymore, or was obtained before disconnecting.","meta":{"error":"Transaction not found. Transaction ID is invalid, refers to an old closed transaction Prisma doesn't have information about anymore, or was obtained before disconnecting."},"error_code":"P2028"} while rolling back a transaction. Ignoring and raising original exception litellm-1 | Execution of job "update_spend (trigger: interval[0:00:13], next run at: 2026-03-12 10:22:20 CST)" skipped: maximum number of running instances reached (1) litellm-1 | 10:22:29 - LiteLLM Proxy:WARNING: utils.py:3934 - Escalating to heavy reconnect after 3 consecutive failures. reason=db_health_watchdog_connection_error litellm-1 | 10:22:29 - LiteLLM Proxy:WARNING: utils.py:3941 - Attempting Prisma DB reconnect. reason=db_health_watchdog_connection_error litellm-1 | 10:22:29 - LiteLLM Proxy:WARNING: utils.py:3876 - prisma-query-engine PID 0 is dead; reconnecting. litellm-1 | query-engine ac9d7041ed77bcc8a8dbd2ab6616b39013829574 litellm-1 | INFO: 192.168.52.26:51697 - "POST /v1/completions HTTP/1.1" 200 OK litellm-1 | INFO: 127.0.0.1:38602 - "GET /health/liveliness HTTP/1.1" 200 OK litellm-1 | 10:22:40 - LiteLLM Proxy:ERROR: proxy_server.py:5181 - litellm.proxy_server.py::get_credentials() - Error getting credentials from DB - litellm-1 | Traceback (most recent call last): litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions litellm-1 | yield litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request litellm-1 | resp = await self._pool.handle_async_request(req) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request litellm-1 | raise exc from None litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request litellm-1 | response = await connection.handle_async_request( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | pool_request.request litellm-1 | ^^^^^^^^^^^^^^^^^^^^ litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request litellm-1 | raise exc litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request litellm-1 | stream = await self._connect(request) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_async/connection.py", line 124, in _connect litellm-1 | stream = await self._network_backend.connect_tcp(**kwargs) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp litellm-1 | return await self._backend.connect_tcp( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<5 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp litellm-1 | with map_exceptions(exc_map): litellm-1 | ~~~~~~~~~~~~~~^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/contextlib.py", line 162, in exit litellm-1 | self.gen.throw(value) litellm-1 | ~~~~~~~~~~~~~~^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions litellm-1 | raise to_exc(exc) from exc litellm-1 | httpcore.ConnectTimeout litellm-1 | litellm-1 | The above exception was the direct cause of the following exception: litellm-1 | litellm-1 | Traceback (most recent call last): litellm-1 | File "/usr/lib/python3.13/site-packages/litellm/proxy/proxy_server.py", line 5172, in get_credentials litellm-1 | credentials = await prisma_client.db.litellm_credentialstable.find_many() litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/actions.py", line 1501, in find_many litellm-1 | resp = await self._client._execute( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<11 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/client.py", line 651, in _execute litellm-1 | return await self._engine.query(builder.build(), tx_id=self._tx_id) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/engine/query.py", line 244, in query litellm-1 | return await self.request( litellm-1 | ^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<4 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/engine/http.py", line 119, in request litellm-1 | resp = await self.session.request(method, url, **kwargs) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/prisma/_async_http.py", line 28, in request litellm-1 | return Response(await self.session.request(method, url, **kwargs)) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1540, in request litellm-1 | return await self.send(request, auth=auth, follow_redirects=follow_redirects) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send litellm-1 | response = await self._send_handling_auth( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<4 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth litellm-1 | response = await self._send_handling_redirects( litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | ...<3 lines>... litellm-1 | ) litellm-1 | ^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects litellm-1 | response = await self._send_single_request(request) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request litellm-1 | response = await transport.handle_async_request(request) litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 393, in handle_async_request litellm-1 | with map_httpcore_exceptions(): litellm-1 | ~~~~~~~~~~~~~~~~~~~~~~~^^ litellm-1 | File "/usr/lib/python3.13/contextlib.py", line 162, in exit litellm-1 | self.gen.throw(value) litellm-1 | ~~~~~~~~~~~~~~^^^^^^^ litellm-1 | File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions litellm-1 | raise mapped_exc(message) from exc litellm-1 | httpx.ConnectTimeout litellm-1 | 10:22:40 - LiteLLM Proxy:ERROR: auth_checks.py:93 - Budget lookup failed for user; cache will not be populated. Each request will hit the database. Error: . litellm-1 | Execution of job "update_spend (trigger: interval[0:00:13], next run at: 2026-03-12 10:22:33 CST)" skipped: maximum number of running instances reached (1)

Steps to Reproduce

Relevant log output

What part of LiteLLM is this about?

No response

What LiteLLM version are you on ?

1.82.1-nightly.1

Twitter / LinkedIn details

No response