litellm - 💡(How to fix) Fix [Bug]: aembedding missing num_retries kwarg causes zero retries and no failover for embedding model groups [1 participants]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Bug Description

When using the LiteLLM proxy router with multiple deployments of the same embedding model_name across different hosts, failover never occurs when a host is unreachable. The router attempts the selected deployment once, fails, and returns an error — no retries, no failover to other deployments in the model group.

Root Cause

In litellm/router.py, the aembedding() method does not set num_retries in kwargs before calling async_function_with_fallbacks(). Compare aembedding to every other router method (acompletion, acreate_file, atext_completion, etc.) which all include this line:

kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)

aembedding (missing the line — bug):

async def aembedding(self, model, input, is_async=True, **kwargs):
    try:
        kwargs["model"] = model
        kwargs["input"] = input
        kwargs["original_function"] = self._aembedding          # ✓
        # kwargs["num_retries"] = ...                           # ✗ MISSING
        self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
        response = await self.async_function_with_fallbacks(**kwargs)

acreate_file (correct pattern — for comparison):

async def acreate_file(self, model, **kwargs):
    try:
        kwargs["model"] = model
        kwargs["original_function"] = self._acreate_file
        kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)  # ✓ present
        self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
        response = await self.async_function_with_fallbacks(**kwargs)

Note: _update_kwargs_before_fallbacks also sets num_retries (line 2234), but it uses kwargs.get("num_retries", self.num_retries) — so if num_retries is already in kwargs it won't be overwritten. The problem is that async_function_with_retries pops num_retries from kwargs (line 5564), so on the retry path the key is gone and _update_kwargs_before_fallbacks may not re-set it correctly. The result is num_retries evaluates to 0 inside the retry loop, causing immediate failure with no retries attempted.

Fix

Add the missing line to aembedding() in router.py:

async def aembedding(self, model, input, is_async=True, **kwargs):
    try:
        kwargs["model"] = model
        kwargs["input"] = input
        kwargs["original_function"] = self._aembedding
        kwargs["num_retries"] = kwargs.get("num_retries", self.num_retries)  # ADD THIS
        self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
        response = await self.async_function_with_fallbacks(**kwargs)

Steps to Reproduce

1. Configure LiteLLM proxy with multiple deployments of the same embedding model_name:

model_list:
  - model_name: mxbai-embed-large
    litellm_params:
      model: ollama/mxbai-embed-large:latest
      api_base: http://host1:11434

  - model_name: mxbai-embed-large
    litellm_params:
      model: ollama/mxbai-embed-large:latest
      api_base: http://host2:11434

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 3
  allowed_fails: 0
  cooldown_time: 60

1. Take host1 offline
2. Send an embedding request for mxbai-embed-large
3. Observe the request fails immediately with no retries — host2 is never tried
4. Repeat the same test with a chat completion model group — failover works correctly, confirming the bug is specific to aembedding

Additional Context

This bug was discovered while investigating embedding failover in a multi-host Ollama setup. Chat completion failover was working correctly (after separately fixing the APIConnectionError cooldown bug — see related issue #27362 ). Embedding failover was still broken despite identical config, and code inspection revealed this missing line as the cause.

All other async router methods set num_retries in kwargs before calling async_function_with_fallbacks. The aembedding method appears to have been overlooked when this pattern was established or when retries were added to other methods.

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.82.6 (also confirmed present in v1.82.1)

Twitter / LinkedIn details

No response