Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When a Bedrock model times out and the router triggers a fallback to Azure OpenAI, the fallback request fails with:

openai.BadRequestError: 400 - 'tool_choice' is only allowed when 'tools' are specified

What happens:

1. A request arrives with both tools and tool_choice parameters
2. LiteLLM routes it to Bedrock (bedrock/converse/...)
3. During Bedrock request processing, converse_handler.py pops several keys from optional_params (e.g., stream, aws_access_key_id, etc.) — these .pop() calls mutate the shared kwargs dict
4. Bedrock times out (e.g., 120s × 3 retries)
5. run_async_fallback() in fallback_event_handlers.py reuses the same kwargs dict without making a deep copy
6. The fallback model (Azure OpenAI) receives kwargs in an inconsistent state — specifically, tool_choice is present but tools is missing
7. Azure OpenAI rejects the request with HTTP 400

Expected behavior:

The fallback should receive the original, unmodified kwargs so it can process the request with both tools and tool_choice intact, returning a structured tool_calls response identical to what the primary model would have returned.

Root cause analysis

In run_async_fallback() (fallback_event_handlers.py), the fallback loop iterates over fallback_model_group and reuses the same kwargs dict for each attempt:

for mg in fallback_model_group:
    # ...
    kwargs["model"] = mg
    response = await litellm_router.async_function_with_fallbacks(*args, **kwargs)

No deep copy of kwargs is made before each fallback attempt. If the previous provider's handler mutated kwargs (or any nested dict within it), the fallback model receives corrupted parameters.

Note on safe_deep_copy: While converse_transformation.py does use safe_deep_copy inside _prepare_request_params(), this only protects optional_params during the Bedrock-specific transformation. It does not protect the kwargs dict that is reused by run_async_fallback() across different providers. The converse_handler.py still performs destructive .pop() calls on the original optional_params dict before calling _async_transform_request() (for keys like stream, model_id, fake_stream, aws_access_key_id, etc. — see converse_handler.py lines 267–317).

Additionally, other parts of the call chain (e.g., completion() in main.py, _acompletion() in router.py) may also modify kwargs in ways that leave them in an inconsistent state for the fallback provider.

Proposed fix

Primary fix: make a deep copy of kwargs before each fallback attempt in run_async_fallback():

import copy

for mg in fallback_model_group:
    if mg == original_model_group:
        continue
    try:
        fallback_kwargs = copy.deepcopy(kwargs)
        fallback_kwargs = litellm_router.log_retry(kwargs=fallback_kwargs, e=original_exception)
        fallback_kwargs["model"] = mg
        # ... rest of the loop using fallback_kwargs instead of kwargs

Optional additional safeguard: a sanitization check to catch any remaining edge cases where this inconsistency could arise:

# If tool_choice is present but tools is not, remove tool_choice
if "tool_choice" in fallback_kwargs and "tools" not in fallback_kwargs:
    fallback_kwargs.pop("tool_choice", None)

Steps to Reproduce

1. Configure a Bedrock model as primary with Azure OpenAI models as default_fallbacks:

model_list:
  - model_name: my-model
    litellm_params:
      model: bedrock/openai.gpt-oss-20b-1:0
      # ... aws credentials

  - model_name: gpt-4o-mini-2024-07-18
    litellm_params:
      model: azure/gpt-4o-mini-2024-07-18
      # ... azure credentials

router_settings:
  timeout: 120
  num_retries: 2
  default_fallbacks: ["gpt-4o-mini-2024-07-18"]

1. Send a request with tools and tool_choice:

import litellm

response = await litellm.acompletion(
    model="my-model",
    messages=[{"role": "user", "content": "Classify this text"}],
    tools=[{"type": "function", "function": {"name": "text_classification", "parameters": {"type": "object", "properties": {"label": {"type": "string"}}}}}],
    tool_choice={"type": "function", "function": {"name": "text_classification"}},
)

1. If the Bedrock model times out, the fallback to Azure fails with:

openai.BadRequestError: 400 - 'tool_choice' is only allowed when 'tools' are specified

Relevant log output

Stack trace from the fallback attempt (Azure OpenAI, ~98.6ms):

litellm/main.py line 620 → acompletion
  llms/azure/azure.py line 491 → acompletion
    llms/azure/azure.py line 172 → azure_client.chat.completions.create(**data, timeout=timeout)
      → openai.BadRequestError: 400 - 'tool_choice' is only allowed when 'tools' are specified

The `data` dict that reaches Azure no longer contains `tools`, only `tool_choice`.

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.82.3-stable (also reproduced on v1.81.14-stable)

Twitter / LinkedIn details

No response