Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Bug]: async_completion_with_fallbacks mutates caller's fallback dicts via .pop() [3 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#28251•Fetched 2026-05-20 03:40:31
View on GitHub
Comments
0
Participants
1
Timeline
6
Reactions
1
Author
lorenzbaraldilorenzbaraldi
Participants
lorenzbaraldilorenzbaraldi
Timeline (top)
cross-referenced ×3labeled ×3

async_completion_with_fallbacks in litellm/litellm_core_utils/fallback_utils.py mutates the caller's fallback dictionary in-place by calling fallback.pop("model", original_model) (line 51). This permanently removes the "model" key from the original dict, causing all subsequent calls that reuse the same fallback configuration to lose the fallback model name.

Error Message

import asyncio import litellm

fallback_config = [{"model": "azure/example", "api_key": "xxx", "api_version": "version"}]

async def main(): # First call — works correctly try: await litellm.acompletion( model="openai/my-primary-model", messages=[{"role": "user", "content": "hi"}], fallbacks=fallback_config, api_key="invalid", # force primary to fail ) except Exception: pass

# After first call, the fallback dict is mutated:
print(fallback_config)
# [{api_key": "...", "api_version": "..."}]  ← "model" key is GONE!

# Second call — fallback uses the WRONG model (original_model instead of azure/gpt-4.1)
try:
    await litellm.acompletion(
        model="openai/my-primary-model",
        messages=[{"role": "user", "content": "hi"}],
        fallbacks=fallback_config,
        api_key="invalid",
    )
except Exception as e:
    print(f"Second call fails: {e}")

asyncio.run(main())

Root Cause

In fallback_utils.py:

for fallback in fallbacks:
    try:
        completion_kwargs = safe_deep_copy(base_kwargs)  # ✅ base_kwargs is copied
        if isinstance(fallback, dict):
            model = fallback.pop("model", original_model)  # ❌ mutates the ORIGINAL dict
            completion_kwargs.update(fallback)

safe_deep_copy correctly protects base_kwargs, but the fallback dict from the caller's list is mutated in-place. After .pop("model"), subsequent calls find no "model" key and default to original_model ( the primary model that already failed) but with the fallback's api_base/api_key, producing 404 errors or routing to the wrong provider.

Fix Action

Fixed

PR fix notes

PR #28253: Fix handling of dictionary fallback configurations in async completion

Description (problem / solution / changelog)

This pull request addresses an issue where fallback configuration dictionaries could be unintentionally mutated during fallback handling in the async_completion_with_fallbacks function. The fix ensures that fallback dictionaries remain unchanged, and a new test is added to verify this behavior.

Bug fix: Prevent mutation of fallback dictionaries

  • Updated async_completion_with_fallbacks in fallback_utils.py to copy the fallback dictionary before modifying it, ensuring the original dictionary is not mutated.

Testing:

  • Added test_fallback_dict_not_mutated in test_fallback_utils.py to confirm that fallback dictionaries are not changed after use, even when reused across multiple calls.

Relevant issues

Fixes https://github.com/BerriAI/litellm/issues/28251

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details

  • My PR passes all unit tests on make test-unit

  • My PR's scope is as isolated as possible, it only solves 1 specific problem

  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

  • Branch creation CI run
    Link:

  • CI run for the last commit
    Link:

  • Merge / cherry-pick CI run
    Links:

Type

🐛 Bug Fix

Changed files

  • litellm/litellm_core_utils/fallback_utils.py (modified, +3/-2)
  • tests/test_litellm/litellm_core_utils/test_fallback_utils.py (added, +43/-0)

PR #28291: fix: prevent mutation of fallback config dict in completion_with_fallbacks

Description (problem / solution / changelog)

Problem

async_completion_with_fallbacks calls .pop("model") directly on the fallback dict passed by the caller. This mutates the original config in-place, so subsequent calls using the same fallback config lose the model key and fall back to the primary (failed) model.

Fix

Copy the fallback dict before popping from it:

# Before
model = fallback.pop("model", original_model)

# After
fallback_copy = fallback.copy()
model = fallback_copy.pop("model", original_model)

Impact

Affects all applications that reuse fallback configs across multiple calls (common in agent frameworks and retry logic).

Fixes #28251

Changed files

  • litellm/litellm_core_utils/fallback_utils.py (modified, +3/-2)

PR #28295: fix(fallbacks): copy fallback dict before pop to avoid mutating caller config

Description (problem / solution / changelog)

Description

Fixes #28251

async_completion_with_fallbacks called fallback.pop("model", original_model) directly on the dict from the caller's fallbacks list, permanently removing the "model" key. On the second call with the same fallback configuration, the model name was gone, causing the fallback to silently use original_model for all subsequent fallbacks rather than the configured one.

Root Cause

# BEFORE (buggy) — mutates caller's dict
if isinstance(fallback, dict):
    model = fallback.pop("model", original_model)  # removes "model" from original
    completion_kwargs.update(fallback)

Fix

Copy the fallback dict before extracting the model key:

# AFTER (fixed) — caller's dict untouched
if isinstance(fallback, dict):
    fallback_copy = dict(fallback)          # shallow copy, caller's list unchanged
    model = fallback_copy.pop("model", original_model)
    completion_kwargs.update(fallback_copy)

Verification

import litellm

fallbacks = [{"model": "gpt-3.5-turbo", "api_key": "sk-test"}]

# First call: works correctly (fallback["model"] == "gpt-3.5-turbo")
# Second call: previously would silently fall back to original_model
#              because fallback["model"] was already popped
# After fix: both calls correctly use "gpt-3.5-turbo" from the fallback config

Changed files

  • litellm/litellm_core_utils/fallback_utils.py (modified, +3/-2)

Code Example

for fallback in fallbacks:
    try:
        completion_kwargs = safe_deep_copy(base_kwargs)  # ✅ base_kwargs is copied
        if isinstance(fallback, dict):
            model = fallback.pop("model", original_model)  # ❌ mutates the ORIGINAL dict
            completion_kwargs.update(fallback)

---

import asyncio
import litellm

fallback_config = [{"model": "azure/example", "api_key": "xxx", "api_version": "version"}]

async def main():
    # First call — works correctly
    try:
        await litellm.acompletion(
            model="openai/my-primary-model",
            messages=[{"role": "user", "content": "hi"}],
            fallbacks=fallback_config,
            api_key="invalid",  # force primary to fail
        )
    except Exception:
        pass

    # After first call, the fallback dict is mutated:
    print(fallback_config)
    # [{api_key": "...", "api_version": "..."}]  ← "model" key is GONE!

    # Second call — fallback uses the WRONG model (original_model instead of azure/gpt-4.1)
    try:
        await litellm.acompletion(
            model="openai/my-primary-model",
            messages=[{"role": "user", "content": "hi"}],
            fallbacks=fallback_config,
            api_key="invalid",
        )
    except Exception as e:
        print(f"Second call fails: {e}")

asyncio.run(main())

---
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Description

async_completion_with_fallbacks in litellm/litellm_core_utils/fallback_utils.py mutates the caller's fallback dictionary in-place by calling fallback.pop("model", original_model) (line 51). This permanently removes the "model" key from the original dict, causing all subsequent calls that reuse the same fallback configuration to lose the fallback model name.

Root Cause

In fallback_utils.py:

for fallback in fallbacks:
    try:
        completion_kwargs = safe_deep_copy(base_kwargs)  # ✅ base_kwargs is copied
        if isinstance(fallback, dict):
            model = fallback.pop("model", original_model)  # ❌ mutates the ORIGINAL dict
            completion_kwargs.update(fallback)

safe_deep_copy correctly protects base_kwargs, but the fallback dict from the caller's list is mutated in-place. After .pop("model"), subsequent calls find no "model" key and default to original_model ( the primary model that already failed) but with the fallback's api_base/api_key, producing 404 errors or routing to the wrong provider.

Impact

Any application that stores fallback configs and reuses them across multiple acompletion/completion calls (common in agent frameworks like Google ADK where a single LLM client instance serves many calls) will experience:

  • First fallback succeeds
  • All subsequent fallbacks fail with incorrect model/provider routing

Environment

  • litellm version: 1.86.0
  • Python: 3.12
  • File: litellm/litellm_core_utils/fallback_utils.py, line 51

Steps to Reproduce

import asyncio
import litellm

fallback_config = [{"model": "azure/example", "api_key": "xxx", "api_version": "version"}]

async def main():
   # First call — works correctly
   try:
       await litellm.acompletion(
           model="openai/my-primary-model",
           messages=[{"role": "user", "content": "hi"}],
           fallbacks=fallback_config,
           api_key="invalid",  # force primary to fail
       )
   except Exception:
       pass

   # After first call, the fallback dict is mutated:
   print(fallback_config)
   # [{api_key": "...", "api_version": "..."}]  ← "model" key is GONE!

   # Second call — fallback uses the WRONG model (original_model instead of azure/gpt-4.1)
   try:
       await litellm.acompletion(
           model="openai/my-primary-model",
           messages=[{"role": "user", "content": "hi"}],
           fallbacks=fallback_config,
           api_key="invalid",
       )
   except Exception as e:
       print(f"Second call fails: {e}")

asyncio.run(main())

Relevant log output

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

v1.86.0

Twitter / LinkedIn details

No response

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#memory leak#API versioning#request timeout

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

litellm - ✅(Solved) Fix [Bug]: async_completion_with_fallbacks mutates caller's fallback dicts via .pop() [3 pull requests, 1 participants]