litellm - 💡(How to fix) Fix [Bug]: Prisma query engine process dies after prolonged uptime, permanently breaking API key authentication

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Error Message

04:40:26 - LiteLLM Proxy:WARNING: utils.py:4176 - Escalating to heavy reconnect after 1485 consecutive failures. reason=auth_get_key_object_lookup_failure 04:40:26 - LiteLLM Proxy:WARNING: utils.py:4183 - Attempting Prisma DB reconnect. reason=auth_get_key_object_lookup_failure 04:40:26 - LiteLLM Proxy:WARNING: utils.py:4112 - prisma-query-engine PID 0 is dead; reconnecting. 04:40:26 - LiteLLM Proxy:ERROR: utils.py:4197 - Prisma DB reconnect failed (1486 consecutive). reason=auth_get_key_object_lookup_failure error=Client is not connected to the query engine, you must call connect() before attempting to query data. 04:40:26 - LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured - Client is not connected to the query engine, you must call connect() before attempting to query data.

Traceback (most recent call last): File ".../litellm/proxy/auth/user_api_key_auth.py", line 1194, in _user_api_key_auth_builder valid_token = await get_key_object( File ".../litellm/proxy/auth/auth_checks.py", line 2327, in get_key_object _valid_token: Optional[BaseModel] = await _fetch_key_object_from_db_with_reconnect( File ".../litellm/proxy/auth/auth_checks.py", line 2231, in _fetch_key_object_from_db_with_reconnect return await prisma_client.get_data( File ".../litellm/proxy/utils.py", line 3200, in get_data raise e File ".../litellm/proxy/utils.py", line 3112, in get_data response = await self._query_first_with_cached_plan_fallback( File ".../litellm/proxy/utils.py", line 2757, in _query_first_with_cached_plan_fallback return await self.db.query_first(sql_query, *args) File ".../prisma/client.py", line 524, in query_first results = await self.query_raw(query, *args) File ".../prisma/client.py", line 559, in query_raw resp = await self._execute( File ".../prisma/client.py", line 663, in _execute return await self._engine.query(builder.build(), tx_id=self._tx_id) File ".../prisma/client.py", line 701, in _engine raise errors.ClientNotConnectedError() prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call connect() before attempting to query data.

Code Example

Authentication Error, Client is not connected to the query engine, you must call `connect()` before attempting to query data.

---

curl -s http://localhost:8080/v1/chat/completions \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer sk-9xXw-abcde" \
     -d '{"model":"llm","messages":[{"role":"user","content":"test"}]}'

---

# config.yaml
general_settings:
  database_url: os.environ/DATABASE_URL
  master_key: os.environ/LITELLM_MASTER_KEY

router_settings:
  routing_strategy: least-busy
  redis_host: localhost
  redis_port: 6379

---

# .env
LITELLM_MASTER_KEY="ZSabcde="
DATABASE_URL="postgresql://litellm:<password>@localhost:5432/litellm"

---

04:40:26 - LiteLLM Proxy:WARNING: utils.py:4176 - Escalating to heavy reconnect after 1485 consecutive failures. reason=auth_get_key_object_lookup_failure
04:40:26 - LiteLLM Proxy:WARNING: utils.py:4183 - Attempting Prisma DB reconnect. reason=auth_get_key_object_lookup_failure
04:40:26 - LiteLLM Proxy:WARNING: utils.py:4112 - prisma-query-engine PID 0 is dead; reconnecting.
04:40:26 - LiteLLM Proxy:ERROR: utils.py:4197 - Prisma DB reconnect failed (1486 consecutive). reason=auth_get_key_object_lookup_failure error=Client is not connected to the query engine, you must call `connect()` before attempting to query data.
04:40:26 - LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured - Client is not connected to the query engine, you must call `connect()` before attempting to query data.

Traceback (most recent call last):
  File ".../litellm/proxy/auth/user_api_key_auth.py", line 1194, in _user_api_key_auth_builder
    valid_token = await get_key_object(
  File ".../litellm/proxy/auth/auth_checks.py", line 2327, in get_key_object
    _valid_token: Optional[BaseModel] = await _fetch_key_object_from_db_with_reconnect(
  File ".../litellm/proxy/auth/auth_checks.py", line 2231, in _fetch_key_object_from_db_with_reconnect
    return await prisma_client.get_data(
  File ".../litellm/proxy/utils.py", line 3200, in get_data
    raise e
  File ".../litellm/proxy/utils.py", line 3112, in get_data
    response = await self._query_first_with_cached_plan_fallback(
  File ".../litellm/proxy/utils.py", line 2757, in _query_first_with_cached_plan_fallback
    return await self.db.query_first(sql_query, *args)
  File ".../prisma/client.py", line 524, in query_first
    results = await self.query_raw(query, *args)
  File ".../prisma/client.py", line 559, in query_raw
    resp = await self._execute(
  File ".../prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
  File ".../prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
RAW_BUFFERClick to expand / collapse

[Bug]: Prisma query engine process dies after prolonged uptime, permanently breaking API key authentication

What happened?

The Prisma client inside the LiteLLM proxy process loses its connection to the query engine after prolonged uptime (~5 days in our case). Once this happens, all non-master-key authentication requests fail permanently with a 401 error and the message:

Authentication Error, Client is not connected to the query engine, you must call `connect()` before attempting to query data.

The proxy continues to accept requests authenticated with the master key (since that's validated from env vars, not the DB), but any virtual key stored in the database becomes unusable. The proxy does not self-heal — the consecutive failure counter reached 1486+ with no recovery, and all subsequent requests continue to fail until a full service restart.

Expected behavior: The proxy should either:

  • Automatically reconnect the Prisma client/query engine when the connection is lost, or
  • Have a health check that detects this state and triggers a restart/reconnect, or
  • At minimum, not silently degrade to a state where DB-backed auth is permanently broken with no path to recovery.

Steps to Reproduce

  1. Deploy LiteLLM proxy with PostgreSQL database and database_url configured. Create virtual API keys via the proxy.
  2. Run the proxy for an extended period (we observed failure after ~5 days of uptime).
  3. At some point, the Prisma query engine process dies (PID becomes 0 in logs).
  4. Send a request with a virtual key (non-master):
    curl -s http://localhost:8080/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer sk-9xXw-abcde" \
      -d '{"model":"llm","messages":[{"role":"user","content":"test"}]}'
  5. Observe 401 error. All subsequent requests with any virtual key also fail. Master key requests still succeed.
  6. Only recovery is sudo systemctl restart litellm-proxy

Config

# config.yaml
general_settings:
  database_url: os.environ/DATABASE_URL
  master_key: os.environ/LITELLM_MASTER_KEY

router_settings:
  routing_strategy: least-busy
  redis_host: localhost
  redis_port: 6379
# .env
LITELLM_MASTER_KEY="ZSabcde="
DATABASE_URL="postgresql://litellm:<password>@localhost:5432/litellm"

Relevant log output

04:40:26 - LiteLLM Proxy:WARNING: utils.py:4176 - Escalating to heavy reconnect after 1485 consecutive failures. reason=auth_get_key_object_lookup_failure
04:40:26 - LiteLLM Proxy:WARNING: utils.py:4183 - Attempting Prisma DB reconnect. reason=auth_get_key_object_lookup_failure
04:40:26 - LiteLLM Proxy:WARNING: utils.py:4112 - prisma-query-engine PID 0 is dead; reconnecting.
04:40:26 - LiteLLM Proxy:ERROR: utils.py:4197 - Prisma DB reconnect failed (1486 consecutive). reason=auth_get_key_object_lookup_failure error=Client is not connected to the query engine, you must call `connect()` before attempting to query data.
04:40:26 - LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured - Client is not connected to the query engine, you must call `connect()` before attempting to query data.

Traceback (most recent call last):
  File ".../litellm/proxy/auth/user_api_key_auth.py", line 1194, in _user_api_key_auth_builder
    valid_token = await get_key_object(
  File ".../litellm/proxy/auth/auth_checks.py", line 2327, in get_key_object
    _valid_token: Optional[BaseModel] = await _fetch_key_object_from_db_with_reconnect(
  File ".../litellm/proxy/auth/auth_checks.py", line 2231, in _fetch_key_object_from_db_with_reconnect
    return await prisma_client.get_data(
  File ".../litellm/proxy/utils.py", line 3200, in get_data
    raise e
  File ".../litellm/proxy/utils.py", line 3112, in get_data
    response = await self._query_first_with_cached_plan_fallback(
  File ".../litellm/proxy/utils.py", line 2757, in _query_first_with_cached_plan_fallback
    return await self.db.query_first(sql_query, *args)
  File ".../prisma/client.py", line 524, in query_first
    results = await self.query_raw(query, *args)
  File ".../prisma/client.py", line 559, in query_raw
    resp = await self._execute(
  File ".../prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
  File ".../prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on?

v1.83.14

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

litellm - 💡(How to fix) Fix [Bug]: Prisma query engine process dies after prolonged uptime, permanently breaking API key authentication