litellm - ✅(Solved) Fix [Bug]: Native Gemini/Vertex `generate_content` drops `toolConfig`, causing empty STOP on valid tool-calling requests [1 pull requests, 1 participants]

litellm2026-03-12 22:38:56

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

BerriAI/litellm#23491•Fetched 2026-04-08 00:44:02

View on GitHub

Comments

Participants

Timeline

Reactions

Author

emerzon

Participants

emerzon

Timeline (top)

labeled ×3closed ×1cross-referenced ×1

LiteLLM's native Google/Vertex generate_content path is dropping top-level toolConfig when forwarding Gemini requests. For certain valid tool-calling payloads, that changes provider behavior from a normal follow-up tool call into an empty response:

HTTP 200
one SSE event
candidates[0].content.parts[0].text == ""
finishReason == "STOP"
no follow-up functionCall

Root Cause

The incoming native request contains these top-level keys:

contents
generationConfig
systemInstruction
toolConfig
tools

On the LiteLLM native Vertex request path, contents are preserved, but toolConfig is dropped before the downstream Vertex request is sent.

I verified this locally by reconstructing the request body via LiteLLM's native Vertex transform.

Original payload keys:

['contents', 'generationConfig', 'systemInstruction', 'toolConfig', 'tools']

LiteLLM-transformed payload keys:

['contents', 'generationConfig', 'model', 'systemInstruction', 'tools']

The transformed payload has:

contents unchanged
toolConfig missing
generationConfig.thinkingConfig mapped to generationConfig.thinking_config

The thinkingConfig rename is not what breaks the request. The missing toolConfig is.

Fix Action

Fixed

Fixed by PR: fix(gemini): preserve toolConfig on native generate_content (https://github.com/BerriAI/litellm/pull/23493)

PR fix notes

PR #23493: fix(gemini): preserve toolConfig on native generate_content

Repository: BerriAI/litellm
Author: emerzon
State: closed | merged: True
Link: https://github.com/BerriAI/litellm/pull/23493

Description (problem / solution / changelog)

Relevant issues

Fixes #23491

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
My PR passes all unit tests on make test-unit
My PR's scope is as isolated as possible, it only solves 1 specific problem
I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

CI status guideline:

50-55 passing tests: main is stable with minor issues.

45-49 passing tests: acceptable but needs attention

<= 40 passing tests: unstable; be careful with your merges and assess the risk.

Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:

Type

🐛 Bug Fix ✅ Test

Changes

This fixes the native Google/Vertex generate_content path dropping top-level toolConfig.

Before this change, LiteLLM preserved contents but rebuilt the downstream native request body without toolConfig. For affected Gemini tool-calling requests, that changed provider behavior from a normal follow-up tool call into a single empty STOP event.

This PR:

threads toolConfig / tool_config through the native generate_content setup path
passes toolConfig through BaseLLMHTTPHandler.generate_content_handler() and async_generate_content_handler()
includes toolConfig in both Google-native and Vertex-native transform_generate_content_request() output
adds regression coverage to verify native request transforms preserve toolConfig
updates the proxy request test to assert the outbound native request body includes toolConfig

Changed files

litellm/google_genai/main.py (modified, +21/-0)
litellm/llms/base_llm/google_genai/transformation.py (modified, +2/-0)
litellm/llms/custom_httpx/llm_http_handler.py (modified, +5/-0)
litellm/llms/gemini/google_genai/transformation.py (modified, +3/-0)
litellm/llms/vertex_ai/google_genai/transformation.py (modified, +5/-1)
tests/proxy_unit_tests/test_google_gemini_proxy_request.py (modified, +3/-0)
tests/test_litellm/google_genai/test_google_genai_main.py (modified, +35/-18)
tests/test_litellm/google_genai/test_google_genai_transformation.py (modified, +26/-0)

Code Example

['contents', 'generationConfig', 'systemInstruction', 'toolConfig', 'tools']

---

['contents', 'generationConfig', 'model', 'systemInstruction', 'tools']

---

POST https://<litellm-host>/v1beta/models/gemini-3-flash-preview:streamGenerateContent?alt=sse
Content-Type: application/json
x-goog-api-key: <gateway-api-key>

---

{
  "contents": [
    {
      "role": "user",
      "parts": [
        {"text": "<task>list all my opened jira tickets</task>"},
        {"text": "<plan-mode / environment details omitted>"}
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "<thought omitted>",
          "thought": true,
          "thoughtSignature": "<base64-thought-signature>"
        },
        {
          "functionCall": {
            "name": "execute_command",
            "args": {
              "command": "env | grep -i jira",
              "requires_approval": false
            }
          },
          "thoughtSignature": "<base64-thought-signature>"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "opaque_tool_call_id_123",
            "response": {
              "result": "result missing"
            }
          }
        },
        {
          "text": "[execute_command for 'env | grep -i jira'] Result:\nCommand failed with exit code 1."
        },
        {
          "text": "<task resumption / environment details omitted>"
        }
      ]
    }
  ],
  "systemInstruction": {
    "parts": [
      {"text": "You are Cline, a software engineering AI."}
    ]
  },
  "tools": [
    {
      "functionDeclarations": [
        {
          "name": "execute_command",
          "description": "Request to execute a CLI command on the system.",
          "parameters": {
            "type": "OBJECT",
            "properties": {
              "command": {"type": "STRING"},
              "requires_approval": {"type": "BOOLEAN"}
            },
            "required": ["command", "requires_approval"]
          }
        }
      ]
    }
  ],
  "toolConfig": {
    "functionCallingConfig": {
      "mode": "ANY"
    }
  },
  "generationConfig": {
    "temperature": 1,
    "thinkingConfig": {
      "thinkingLevel": "HIGH",
      "includeThoughts": true
    }
  }
}

---

data: {"candidates": [{"content": {"role": "model", "parts": [{"text": ""}]}, "finishReason": "STOP"}], "usageMetadata": {"promptTokenCount": 18790, "totalTokenCount": 18790, "cachedContentTokenCount": 15523, "trafficType": "ON_DEMAND"}, "modelVersion": "gemini-3-flash-preview"}

---

POST https://aiplatform.googleapis.com/v1beta1/projects/<project-id>/locations/global/publishers/google/models/gemini-3-flash-preview:streamGenerateContent?alt=sse
Content-Type: application/json
Authorization: Bearer <access-token>

---

data: {"candidates": [{"content": {"role": "model", "parts": [{"text": "..."}]}}], ...}

data: {"candidates": [{"content": {"role": "model", "parts": [{"text": "..."}]}}], ...}

data: {"candidates": [{"content": {"role": "model", "parts": [{"functionCall": {"name": "execute_command", "args": {"command": "ls -F", "requires_approval": false}}, "thoughtSignature": "<provider-generated-signature>"}]}, "finishReason": "STOP"}], ...}

---

RAW_BUFFERClick to expand / collapse

Check for existing issues

I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Summary

HTTP 200
one SSE event
candidates[0].content.parts[0].text == ""
finishReason == "STOP"
no follow-up functionCall

Root Cause

The incoming native request contains these top-level keys:

contents
generationConfig
systemInstruction
toolConfig
tools

On the LiteLLM native Vertex request path, contents are preserved, but toolConfig is dropped before the downstream Vertex request is sent.

I verified this locally by reconstructing the request body via LiteLLM's native Vertex transform.

Original payload keys:

['contents', 'generationConfig', 'systemInstruction', 'toolConfig', 'tools']

LiteLLM-transformed payload keys:

['contents', 'generationConfig', 'model', 'systemInstruction', 'tools']

The transformed payload has:

contents unchanged
toolConfig missing
generationConfig.thinkingConfig mapped to generationConfig.thinking_config

The thinkingConfig rename is not what breaks the request. The missing toolConfig is.

Steps to Reproduce

1. LiteLLM native endpoint: affected behavior

Request to LiteLLM:

POST https://<litellm-host>/v1beta/models/gemini-3-flash-preview:streamGenerateContent?alt=sse
Content-Type: application/json
x-goog-api-key: <gateway-api-key>

{
  "contents": [
    {
      "role": "user",
      "parts": [
        {"text": "<task>list all my opened jira tickets</task>"},
        {"text": "<plan-mode / environment details omitted>"}
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "<thought omitted>",
          "thought": true,
          "thoughtSignature": "<base64-thought-signature>"
        },
        {
          "functionCall": {
            "name": "execute_command",
            "args": {
              "command": "env | grep -i jira",
              "requires_approval": false
            }
          },
          "thoughtSignature": "<base64-thought-signature>"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "opaque_tool_call_id_123",
            "response": {
              "result": "result missing"
            }
          }
        },
        {
          "text": "[execute_command for 'env | grep -i jira'] Result:\nCommand failed with exit code 1."
        },
        {
          "text": "<task resumption / environment details omitted>"
        }
      ]
    }
  ],
  "systemInstruction": {
    "parts": [
      {"text": "You are Cline, a software engineering AI."}
    ]
  },
  "tools": [
    {
      "functionDeclarations": [
        {
          "name": "execute_command",
          "description": "Request to execute a CLI command on the system.",
          "parameters": {
            "type": "OBJECT",
            "properties": {
              "command": {"type": "STRING"},
              "requires_approval": {"type": "BOOLEAN"}
            },
            "required": ["command", "requires_approval"]
          }
        }
      ]
    }
  ],
  "toolConfig": {
    "functionCallingConfig": {
      "mode": "ANY"
    }
  },
  "generationConfig": {
    "temperature": 1,
    "thinkingConfig": {
      "thinkingLevel": "HIGH",
      "includeThoughts": true
    }
  }
}

Observed response from the affected LiteLLM build:

data: {"candidates": [{"content": {"role": "model", "parts": [{"text": ""}]}, "finishReason": "STOP"}], "usageMetadata": {"promptTokenCount": 18790, "totalTokenCount": 18790, "cachedContentTokenCount": 15523, "trafficType": "ON_DEMAND"}, "modelVersion": "gemini-3-flash-preview"}

Important properties of the broken LiteLLM response:

exactly one SSE data: event
empty text part
finishReason: "STOP"
no follow-up functionCall

2. Direct Google / Vertex request: same semantic request succeeds

Request sent directly to Vertex:

POST https://aiplatform.googleapis.com/v1beta1/projects/<project-id>/locations/global/publishers/google/models/gemini-3-flash-preview:streamGenerateContent?alt=sse
Content-Type: application/json
Authorization: Bearer <access-token>

Observed response directly from Vertex:

data: {"candidates": [{"content": {"role": "model", "parts": [{"text": "..."}]}}], ...}

data: {"candidates": [{"content": {"role": "model", "parts": [{"text": "..."}]}}], ...}

data: {"candidates": [{"content": {"role": "model", "parts": [{"functionCall": {"name": "execute_command", "args": {"command": "ls -F", "requires_approval": false}}, "thoughtSignature": "<provider-generated-signature>"}]}, "finishReason": "STOP"}], ...}

Important properties of the direct Google / Vertex response:

multiple SSE data: events
non-empty content
final structured functionCall
finishReason: "STOP"
no empty text event

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.82.1

Twitter / LinkedIn details

No response

extent analysis

Fix Plan

To fix the issue of the toolConfig being dropped in the LiteLLM native Vertex request path, we need to modify the request transformation logic to preserve the toolConfig.

Here are the steps:

Modify the transform function in the LiteLLM proxy to include toolConfig in the transformed payload.
Update the generationConfig mapping to handle the rename of thinkingConfig to thinking_config.

Example code snippet:

def transform_payload(payload):
    # Preserve toolConfig
    transformed_payload = {
        'contents': payload['contents'],
        'generationConfig': {
            'temperature': payload['generationConfig']['temperature'],
            'thinking_config': payload['generationConfig']['thinkingConfig']  # Handle rename
        },
        'systemInstruction': payload['systemInstruction'],
        'tools': payload['tools'],
        'toolConfig': payload['toolConfig']  # Include toolConfig
    }
    return transformed_payload

Apply this transformation to the incoming request payload before sending it to the Vertex endpoint.

Verification

To verify that the fix worked, send a request to the LiteLLM native endpoint with the same payload as before and check the response. The response should now include the toolConfig and have a non-empty text part and a structured functionCall.

Example verification steps:

Send a request to the LiteLLM native endpoint with the original payload.
Check the response for the presence of toolConfig and a non-empty text part.
Verify that the response includes a structured functionCall with the expected name and args.

Extra Tips

Make sure to test the fix thoroughly to ensure that it does not introduce any new issues.
Consider adding logging or monitoring to track the transformed payload and response to ensure that the fix is working as expected.
Review the LiteLLM documentation and Vertex API documentation to ensure that the transformation is correct and compliant with the API requirements.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #device allocation #model download #tokenizer error #LLM response

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

litellm - ✅(Solved) Fix [Bug]: Native Gemini/Vertex `generate_content` drops `toolConfig`, causing empty STOP on valid tool-calling requests [1 pull requests, 1 participants]

Recommended Tools

GitHub issue graph ai analysis

Root Cause

Fix Action

Fixed

PR fix notes

PR #23493: fix(gemini): preserve toolConfig on native generate_content

Description (problem / solution / changelog)

Relevant issues

Pre-Submission checklist

CI (LiteLLM team)

Type

Changes

Changed files

Code Example

Check for existing issues

What happened?

Summary

Root Cause

Steps to Reproduce

1. LiteLLM native endpoint: affected behavior

2. Direct Google / Vertex request: same semantic request succeeds

Relevant log output

What part of LiteLLM is this about?

What LiteLLM version are you on ?

Twitter / LinkedIn details

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING