litellm - 💡(How to fix) Fix [Bug]: MCP semantic tool filter crashloops at proxy startup with a large MCP server registry [1 participants]

build_router_from_mcp_registry() iterates every MCP server in the DB, fetches every tool from each, and constructs a single SemanticRouter(routes=[...], auto_sync="local") whose construction eagerly embeds every route's utterances in one shot. This is the only caller of this method in the codebase — grep confirms it runs exclusively at startup.

Two design gaps make this a hard crash-loop rather than "slow but eventually healthy":

1. No persistence / no incremental build. Every pod restart re-fetches every tool and re-embeds every route from scratch. If one boot OOMs, every subsequent boot OOMs identically. There is no on-disk cache, no Redis cache, no "resume from last successful embedding" — the work is entirely re-done.
2. Adding / removing MCP servers does not refresh the router. POST /v1/mcp/server (management_endpoints/mcp_management_endpoints.py:810-819) updates the DB and the global_mcp_server_manager registry but never calls build_router_from_mcp_registry() or any rebuild. So:
   • New tools are invisible to the semantic filter until the next proxy restart, which is also the next OOM.
   • A DB accumulated from previous registrations (dev/test leftovers, team onboarding, etc.) silently becomes a startup bomb.

Concrete numbers from my incident (GKE, 2 GiB container limit for litellm, qwen3-embedding-4b wrapped as embedding api):

• ~15 mock MCP servers × ~500 synthetic tools ≈ 7,500 routes
• Startup embedding pass exceeded the 2 GiB cgroup limit → exitCode: 137 within ~60s
• 24+ restarts in 90 minutes, readiness never passing
• Recovery required manually deleting LiteLLM_MCPServerTable / LiteLLM_ObjectPermissionTable rows so the registry was small enough to fit

Expected behavior

The semantic tool filter should be production-safe regardless of DB size:

1. Build the router lazily (on first request per tool) or incrementally (batched, with a memory budget), not eagerly-all-at-once at startup.
2. Persist embeddings (keyed by a hash of tool_name + description) so pod restarts are O(new-or-changed-tools), not O(all-registered-tools).
3. Rebuild / augment the index when POST|PUT|DELETE /v1/mcp/server fires, so the in-memory state stays in sync without requiring a restart.
4. Startup failure of the filter should degrade gracefully — return tools unfiltered and log — rather than blocking the proxy from ever becoming ready. (Currently initialize_from_config catches the exception, but the upstream OOM is at the OS level, so the except never runs.)

Steps to Reproduce

Steps to Reproduce

1. config.yaml with semantic filter enabled:

TL;DR

The likely fix involves modifying the build_router_from_mcp_registry method to build the router lazily or incrementally, persisting embeddings to avoid rebuilding on every restart, and updating the router when MCP servers are added or removed.

Guidance

• Modify the SemanticToolFilterHook.initialize_from_config method to build the router lazily, fetching tools only when needed, rather than eagerly at startup.
• Implement a persistence mechanism, such as a cache or database, to store embeddings and avoid rebuilding them on every restart.
• Update the build_router_from_mcp_registry method to rebuild or augment the index when MCP servers are added, removed, or updated, to keep the in-memory state in sync.
• Consider implementing a fallback mechanism to return tools unfiltered and log an error if the filter fails to initialize, rather than blocking the proxy from becoming ready.

Example

# Example of lazy building
async def build_router_from_mcp_registry(self):
    # Fetch tools only when needed
    tools = await self.fetch_tools_from_mcp_servers()
    # Build router incrementally
    for tool in tools:
        # Add tool to router
        self.semantic_filter.add_tool(tool)

# Example of persistence
async def build_router_from_mcp_registry(self):
    # Load persisted embeddings
    embeddings = await self.load_persisted_embeddings()
    # Build router from persisted embeddings
    self.semantic_filter.build_router(embeddings)

Notes

The provided solution is based on the assumption that the build_router_from_mcp_registry method is the primary cause of the OOM issue. However, further investigation may be necessary to confirm this and identify any other contributing factors.

Recommendation

Apply a workaround by modifying the build_router_from_mcp_registry method to build the router lazily or incrementally, and persisting embeddings to avoid rebuilding on every restart. This should help mitigate the OOM issue and allow the proxy to become ready.