PR #24901: fix(anthropic): auto-detect image media_type from magic bytes to prevent API rejection

• Repository: BerriAI/litellm
• Author: xykong
• State: open | merged: False
• Link: https://github.com/BerriAI/litellm/pull/24901

Description (problem / solution / changelog)

Summary

Fixes #24900

Anthropic API strictly validates that the declared media_type in a base64 image source matches the actual image bytes. When they mismatch (e.g. a client sends a JPEG with a data:image/png;base64,... URI, or passes an Anthropic-native message with a wrong media_type), Anthropic returns a 400:

invalid_request_error: messages.92.content.0.tool_result.content.1.image.source.base64:
The image was specified using the image/png media type, but the image appears to be a image/jpeg image

Root Cause

There are two separate code paths that need fixing:

Path 1 — /chat/completions (OpenAI format)

convert_to_anthropic_image_obj() in factory.py extracted media_type directly from the data URI prefix and passed it through without byte-level verification:

# Before: trusted declared type blindly
media_type, base64_data = openai_image_url.split("data:")[1].split(";base64,")

Path 2 — /v1/messages (Anthropic native format)

When clients use the Anthropic-native /v1/messages API, images arrive already in Anthropic format:

{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "..."}}

These messages bypass factory.py entirely — anthropic_messages_handler() forwards them directly to Anthropic without any media_type validation.

Fix

Fix 1 — factory.py (convert_to_anthropic_image_obj)

After extracting the declared media_type, decode the first 128 bytes and run them through the existing get_image_type() magic-byte detector. Override the declared type if there's a mismatch:

try:
    from litellm.litellm_core_utils.token_counter import get_image_type
    image_bytes = base64.b64decode(base64_data[:128])
    detected = get_image_type(image_bytes)
    if detected is not None:
        detected_mime = "image/{}".format(detected)
        if detected_mime != media_type:
            media_type = detected_mime
except Exception:
    pass

Fix 2 — messages/handler.py (anthropic_messages_handler)

Add _fix_image_media_types_in_messages() called before the request is sent. It walks all content blocks (including nested tool_result content) and applies the same magic-byte correction:

def _fix_image_media_types_in_messages(messages):
    # Walk all blocks including tool_result nested content
    # Detect from magic bytes, correct media_type if mismatched
    ...

def anthropic_messages_handler(...):
    messages = _fix_image_media_types_in_messages(messages)
    ...

Why This Approach

• Zero overhead for correct images: only 128 bytes decoded for magic-byte check
• Reuses existing infrastructure: get_image_type() already used by BedrockImageProcessor and get_image_dimensions()
• Graceful degradation: detection wrapped in try/except; any failure preserves original behavior
• No breaking changes: only changes behavior when declared and actual types mismatch
• Covers both API formats: /chat/completions (OpenAI) and /v1/messages (Anthropic native)

Changed files

• litellm/litellm_core_utils/prompt_templates/factory.py (modified, +20/-0)
• litellm/llms/anthropic/experimental_pass_through/messages/handler.py (modified, +63/-0)