litellm - 💡(How to fix) Fix [Bug]: Concurrent MCP tool calls fail in stateless mode — first request's lifespan teardown cancels all in-flight requests [1 comments, 1 participants]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When multiple MCP tools/call requests arrive concurrently at the /mcp/ endpoint (stateless mode), only the first to complete returns successfully. All other in-flight requests are cancelled. Sequential requests work fine.

Expected: All concurrent tool calls return successfully. Actual: 1 of N succeeds, the rest hang until the MCP client timeout (60s) and are cancelled.

Root Cause In stateless mode, each incoming request triggers Server.run() (MCP SDK lowlevel/server.py:373), which enters the LiteLLM-defined lifespan context manager. The lifespan's finally block calls shutdown_session_managers(), which tears down the shared StreamableHTTPSessionManager and its TaskGroup — killing all other concurrent requests.

litellm/proxy/_experimental/mcp_server/server.py:246-252:

@contextlib.asynccontextmanager
async def lifespan(app) -> AsyncIterator[None]:
    await initialize_session_managers()
    try:
        yield
    finally:
        await shutdown_session_managers()  # ← cancels all concurrent requests

The call chain:

1. Each stateless request spawns run_stateless_server into the shared _task_group (streamable_http_manager.py:182)
2. Each run_stateless_server calls self.app.run() → enters lifespan (server.py:246)
3. initialize_session_managers() is a no-op after the first call (guarded by _SESSION_MANAGERS_INITIALIZED, server.py:208)
4. The request is processed, the tool call completes on the upstream server
5. The first request's Server.run() exits → lifespan finally → shutdown_session_managers() (server.py:226)
6. shutdown_session_managers() calls _session_manager_cm.aexit() (server.py:235) → tg.cancel_scope.cancel() (streamable_http_manager.py:134)
7. All tasks in the shared TaskGroup are cancelled, including other in-flight requests The shutdown also sets _SESSION_MANAGERS_INITIALIZED = False (server.py:243), so subsequent requests have to re-initialize.

Steps to Reproduce

1. Configure LiteLLM with any MCP server (stateless Streamable HTTP):

mcp_servers:
  my_server:
    url: "http://my-mcp-server/mcp"
    transport: "http"
    allow_all_keys: true

1. Send 3 concurrent tools/call requests:

API_KEY="sk-your-key-here"
URL="http://localhost:4000/mcp/"
for i in 1 2 3; do
  (
    curl -s --max-time 65 -X POST "$URL" \
      -H "Authorization: Bearer $API_KEY" \
      -H "Content-Type: application/json" \
      -H "Accept: application/json, text/event-stream" \
      -d "{\"jsonrpc\":\"2.0\",\"id\":$i,\"method\":\"tools/call\",\"params\":{\"name\":\"my_tool\",\"arguments\":{\"query\":\"test $i\"}}}" \
      -o /dev/null -w "Call $i: HTTP %{http_code} in %{time_total}s\n" \
      || echo "Call $i: TIMED OUT"
  ) &
done
wait

1. Observe: 1 call returns HTTP 200 in ~400ms, the other 2 hang for 60s+ then time out.

Relevant log output

# Client output: 1 succeeds, 2 time out
Call 3: HTTP 200 in 0.134s
Call 1: TIMED OUT after 65s
Call 2: TIMED OUT after 65s
# Upstream MCP server logs: all 3 tool calls arrive and complete within 145ms
15:17:37.443 - CallTool "test longwait 1" → 15:17:37.588 Found results (145ms)
15:17:37.444 - CallTool "test longwait 2" → 15:17:37.587 Found results (143ms)
15:17:37.445 - CallTool "test longwait 3" → 15:17:37.524 Found results (79ms)
# LiteLLM proxy logs: cancellations at 15:17:39 (2s after backend completed)
# and 15:18:37 (60s MCP client timeout on the already-dead sessions)
15:17:39 - LiteLLM:WARNING: client.py:450 - MCP client tool call was cancelled
15:18:37 - LiteLLM:WARNING: client.py:450 - MCP client tool call was cancelled
15:18:37 - LiteLLM:WARNING: client.py:450 - MCP client tool call was cancelled

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

main-latest (ghcr.io/berriai/litellm:main-latest, pulled 2026-03-23)

Twitter / LinkedIn details

No response