Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When the database becomes unreachable, the proxy's DB reconnect logic calls await self.db.disconnect() which ultimately invokes prisma-client-py's synchronous process.wait() (a blocking waitpid syscall) on the query engine subprocess. This freezes the entire asyncio event loop for 30-120+ seconds while the Rust query engine waits for TCP close operations to time out.

During this time, no coroutines can run, including the /health/liveliness endpoint. In Kubernetes, this causes liveness probe failures and pod restarts (exit code 137 / SIGKILL).

The asyncio.wait_for() wrapper around the disconnect call at utils.py:4134 cannot help because process.wait() is synchronous — it never yields back to the event loop, so the timeout never fires.

Two code paths trigger this:

1. _do_direct_reconnect (litellm/proxy/utils.py:4120-4134) — called when the engine process is alive but the DB is unreachable. Calls await self.db.disconnect() which hits the blocking process.wait().

2. recreate_prisma_client (litellm/proxy/db/prisma_client.py:217-236) — called from the heavy-reconnect path. Also calls await self._original_prisma.disconnect() as a first step before creating a new client.

Expected behavior: Reconnection should not block the event loop. Since the old Prisma client is being discarded anyway, the reconnect paths should kill the engine subprocess directly (SIGTERM → short grace → SIGKILL) and create a fresh client, avoiding the blocking close() path entirely.

Steps to Reproduce

1. Run a LiteLLM proxy connected to a PostgreSQL database
2. Make the database unreachable (e.g., network partition, firewall rule, or kill the DB process)
3. Wait for DB operations to start failing — the reconnect logic triggers _do_direct_reconnect
4. Observe that the /health/liveliness endpoint becomes unresponsive for 30-120+ seconds
5. In Kubernetes, the liveness probe fails and the kubelet kills the pod (exit code 137)

The issue is in prisma-client-py's generated QueryEngine.close() → self.process.wait(timeout=None), which is a synchronous blocking call. Even disconnect(timeout=timedelta(seconds=10)) would block the event loop for up to 10 seconds synchronously — long enough to fail a typical liveness probe.

Relevant log output

# Logs preceding crash show DB connectivity loss:
Prisma DB reconnect failed (2 consecutive). reason=engine_process_death error=Could not connect to the query engine
httpx.ConnectError: All connection attempts failed
Giving up get_data(...) after 3 tries (httpx.ConnectError: All connection attempts failed)

# Liveness probe fails with timeout (server completely unresponsive, not returning errors):
Liveness probe failed: context deadline exceeded

# Pod killed by kubelet:
Last State: Terminated, Reason: Error, Exit Code: 137

Suggested Fix

In both _do_direct_reconnect and recreate_prisma_client, replace await self.db.disconnect() with the existing _kill_engine_process helper (SIGTERM → 0.5s grace → SIGKILL) followed by creating a fresh Prisma client. This avoids the blocking close() path entirely.

For the PrismaClient.disconnect() wrapper used during graceful shutdown, offload the blocking call to a thread executor:

loop = asyncio.get_running_loop()
await asyncio.wait_for(
    loop.run_in_executor(None, self.db._original_prisma._engine.close),
    timeout=10,
)

Related issues

This is the same symptom class (synchronous calls blocking the asyncio event loop, causing liveness failures) as:

• #24788 — sync convert_url_to_base64() blocks event loop
• #26181 — O(n²) json.loads retry blocks event loop
• #20268 — sync next() on boto3 iterators blocks event loop

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on?

v1.83.7 (prisma-client-py 0.11.0)

Twitter / LinkedIn details

No response