litellm - 💡(How to fix) Fix [Feature]: Expose router cooldown state via API endpoint for observability [1 participants]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

The Feature

When the LiteLLM router places a deployment in cooldown after repeated failures, that state is internal and not exposed via any API. Operators have no way to query which deployments are currently in cooldown, why they were cooled down, or when they will recover without reading application logs.

Proposal: add a /router/cooldowns endpoint (or extend /health) to surface current cooldown state per deployment:

• model_name, deployment_id, cooled_down_at, cooldown_expires_at, reason (e.g. RateLimitError), failure_count

Motivation, pitch

Cooldown is a core reliability mechanism, but without visibility:

1. Silent capacity reduction: when a deployment is cooled down, all traffic shifts to remaining deployments. Operators see latency increase but cannot tell why without grepping logs.
2. Debugging fallback chains is hard: there is no audit trail of which deployments were cooled down when a request fell back.
3. Alerting is impossible: monitoring cannot alert on "deployment X in cooldown > 10 minutes" without a queryable API.

The /health endpoint does live probes. A /router/cooldowns endpoint would complement it by showing current internal router state without triggering new requests. Related to #25457.

What part of LiteLLM is this about?

Proxy / Router

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

Twitter / LinkedIn details

No response