litellm - 💡(How to fix) Fix [Feature]: Add an LLM as an orchestrator to choose which LLM to call in the gateway

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

The Feature

Add an LLM-based orchestration layer in the gateway that dynamically selects which downstream LLM provider/model to call based on the incoming request context. The orchestrator should analyze factors such as task type, latency requirements, cost constraints, model capabilities, token limits, and reliability, then route the request to the most suitable LLM automatically.

The feature could also support:

Fallback strategies when a provider fails Multi-model routing policies Context-aware model selection Load balancing between providers Configurable routing rules and priorities

Motivation, pitch

Currently, model selection must be handled manually or through static configuration, which makes it difficult to optimize for performance, cost, and reliability across different use cases.

I'm working on scenarios where different requests benefit from different models:

lightweight models for fast/simple tasks, reasoning-focused models for complex workflows, cheaper models for cost-sensitive operations, specialized models for coding, embeddings, or structured outputs.

Adding an orchestration LLM in the gateway would enable intelligent routing decisions automatically, reducing operational complexity and improving overall efficiency and resilience.

What part of LiteLLM is this about?

SDK (litellm Python package)

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

Yes

Twitter / LinkedIn details

https://www.linkedin.com/in/mohamed-aymen-bouyahia-23387a16a/