Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Bug]: LiteLLM Vertex AI Gemini Multimodal Data Drop and Schema Mismatch [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#24399•Fetched 2026-04-08 01:17:59
View on GitHub
Comments
0
Participants
1
Timeline
8
Reactions
0
Author
vivekvjnkvivekvjnk
Participants
vivekvjnkvivekvjnk
Timeline (top)
labeled ×3referenced ×3cross-referenced ×2

Root Cause

1. Issue Description

When using LiteLLM with Vertex AI Gemini models (vertex_ai/gemini-...), multimodal content such as images (passed via image_url) or tool results containing images were being dropped or ignored by the model.

Root Causes:

  1. Field Naming Mismatch: LiteLLM was sending field names in snake_case (e.g., inline_data, function_response, mime_type), but the Vertex AI REST API (generateContent endpoint) strictly requires camelCase for its structural fields (e.g., inlineData, functionResponse, mimeType).
  2. Schema Field Casing: A generic camelCase transformation attempted to fix structural fields but also incorrectly modified user-defined field names within tool arguments, responses, and JSON schemas (e.g., security_risk -> securityRisk), causing schema validation failures because the required list (strings) no longer matched the properties (modified keys).

Fix Action

Fixed

PR fix notes

PR #24400: fix(vertex_ai): fix multimodal field naming mismatch and preserve tool schema casing(fixes #24399)

Description (problem / solution / changelog)

…

Relevant issues

"Fixes #24399"

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.

  • Branch creation CI run
    Link:

  • CI run for the last commit
    Link:

  • Merge / cherry-pick CI run
    Links:

Type

<!-- Select the type of Pull Request --> <!-- Keep only the necessary ones -->

🆕 New Feature 🐛 Bug Fix 🧹 Refactoring 📖 Documentation 🚄 Infrastructure ✅ Test

Changes

Solution Overview

The solution introduces a recursive transformer _transform_part_to_httpx_format strictly within the Vertex AI implementation of LiteLLM.

Description

  • Fixed Multimodal Content: Corrected snake_case field names (e.g., inline_data -> inlineData) in Vertex AI payloads that caused images and other media to be dropped.
  • Preserved User Casing: Implemented selective recursive transformation to avoid modifying user-defined keys in tool schemas and arguments, preventing INVALID_ARGUMENT errors when schemas use snake_case.
  • Scope: All changes are strictly contained within litellm/llms/vertex_ai/ and target only Vertex AI providers.
  • Verified: Tested with multimodal payloads and complex tool schemas containing snake_case fields.

Key Features:

  • CamelCase Conversion: Structural Gemini fields are converted to camelCase to comply with the Vertex AI REST API.
  • Exception List: Fields containing user-defined data (args, response, properties, labels) are excluded from key-casing changes. For properties in JSON schemas, the values (nested schemas) are transformed while keys (field names) are preserved.
  • Minimal Footprint: The transformation is applied at the final stage of request body construction in _transform_request_body and context caching logic, ensuring no side effects on other providers like Google AI Studio.

I've added a comprehensive unit test in litellm/tests/litellm/llms/vertex_ai/gemini/test_transformation.py covering both the field naming fix and the schema preservation logic.

Changed files

  • litellm/llms/vertex_ai/context_caching/transformation.py (modified, +8/-1)
  • litellm/llms/vertex_ai/gemini/transformation.py (modified, +60/-0)
  • tests/litellm/llms/vertex_ai/gemini/test_transformation.py (modified, +77/-1)

Code Example

import sys
import json
# Assuming the fix is applied or testing the logic
from litellm.llms.vertex_ai.gemini.transformation import _transform_part_to_httpx_format
def reproduce_issue():
    # 1. Test Multimodal field naming
    part = {
        "inline_data": {
            "mime_type": "image/png",
            "data": "base64data"
        }
    }
    transformed = _transform_part_to_httpx_format(part)
    print("Multimodal Transformation:")
    print(f"  Input:  {list(part.keys())} -> {list(part['inline_data'].keys())}")
    print(f"  Output: {list(transformed.keys())} -> {list(transformed['inlineData'].keys())}")
    # Expected: 'inlineData' and 'mimeType'
    assert "inlineData" in transformed
    assert "mimeType" in transformed["inlineData"]
    # 2. Test User-defined Schema Preservation
    schema_part = {
        "properties": {
            "security_risk": {"type": "string"}
        },
        "required": ["security_risk"]
    }
    transformed_schema = _transform_part_to_httpx_format(schema_part)
    print("\nSchema Transformation:")
    print(f"  Input keys:  {list(schema_part['properties'].keys())}")
    print(f"  Output keys: {list(transformed_schema['properties'].keys())}")
    # Expected: 'security_risk' is preserved, NOT converted to 'securityRisk'
    assert "security_risk" in transformed_schema["properties"]
if __name__ == "__main__":
    reproduce_issue()

---
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

1. Issue Description

When using LiteLLM with Vertex AI Gemini models (vertex_ai/gemini-...), multimodal content such as images (passed via image_url) or tool results containing images were being dropped or ignored by the model.

Root Causes:

  1. Field Naming Mismatch: LiteLLM was sending field names in snake_case (e.g., inline_data, function_response, mime_type), but the Vertex AI REST API (generateContent endpoint) strictly requires camelCase for its structural fields (e.g., inlineData, functionResponse, mimeType).
  2. Schema Field Casing: A generic camelCase transformation attempted to fix structural fields but also incorrectly modified user-defined field names within tool arguments, responses, and JSON schemas (e.g., security_risk -> securityRisk), causing schema validation failures because the required list (strings) no longer matched the properties (modified keys).

Steps to Reproduce

2. Reproduction Script

This script demonstrates the transformation logic issue without making real LLM calls. It requires litellm to be installed.

import sys
import json
# Assuming the fix is applied or testing the logic
from litellm.llms.vertex_ai.gemini.transformation import _transform_part_to_httpx_format
def reproduce_issue():
    # 1. Test Multimodal field naming
    part = {
        "inline_data": {
            "mime_type": "image/png",
            "data": "base64data"
        }
    }
    transformed = _transform_part_to_httpx_format(part)
    print("Multimodal Transformation:")
    print(f"  Input:  {list(part.keys())} -> {list(part['inline_data'].keys())}")
    print(f"  Output: {list(transformed.keys())} -> {list(transformed['inlineData'].keys())}")
    # Expected: 'inlineData' and 'mimeType'
    assert "inlineData" in transformed
    assert "mimeType" in transformed["inlineData"]
    # 2. Test User-defined Schema Preservation
    schema_part = {
        "properties": {
            "security_risk": {"type": "string"}
        },
        "required": ["security_risk"]
    }
    transformed_schema = _transform_part_to_httpx_format(schema_part)
    print("\nSchema Transformation:")
    print(f"  Input keys:  {list(schema_part['properties'].keys())}")
    print(f"  Output keys: {list(transformed_schema['properties'].keys())}")
    # Expected: 'security_risk' is preserved, NOT converted to 'securityRisk'
    assert "security_risk" in transformed_schema["properties"]
if __name__ == "__main__":
    reproduce_issue()

Relevant log output

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

v1.82.6

Twitter / LinkedIn details

https://www.linkedin.com/in/vivek-vijayan-b0837b146/

extent analysis

Fix Plan

To address the field naming mismatch and schema field casing issues, we need to modify the _transform_part_to_httpx_format function in litellm.llms.vertex_ai.gemini.transformation.

Here are the steps:

  • Modify the function to only transform specific structural fields to camelCase, while preserving user-defined field names.
  • Use a whitelist approach to only transform known structural fields.

Code Changes

# In litellm.llms.vertex_ai.gemini.transformation
def _transform_part_to_httpx_format(part):
    # Define a whitelist of structural fields to transform
    structural_fields = {
        "inline_data": "inlineData",
        "mime_type": "mimeType",
        "function_response": "functionResponse"
    }
    
    transformed_part = {}
    for key, value in part.items():
        if key in structural_fields:
            transformed_part[structural_fields[key]] = value
        else:
            transformed_part[key] = value
            
        # Recursively transform nested dictionaries
        if isinstance(value, dict):
            transformed_part[transformed_part.keys()[-1]] = _transform_part_to_httpx_format(value)
    
    return transformed_part

Verification

Run the reproduction script again to verify that the transformation logic is correct. The output should show that the structural fields are transformed to camelCase, while user-defined field names are preserved.

Extra Tips

  • Make sure to test the changes thoroughly to ensure that they do not introduce any new issues.
  • Consider adding more fields to the whitelist as needed to support additional structural fields.
  • Review the code changes to ensure they align with the existing coding standards and best practices.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#LLM response#prompt template#agent execution#callback error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING