litellm - 💡(How to fix) Fix [Bug]: ResponseInputTextParam TypedDict missing cache_control field — Pydantic silently strips it [1 participants]

litellm2026-03-27 14:10:49

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

BerriAI/litellm#24684•Fetched 2026-04-08 01:42:11

View on GitHub

Comments

Participants

Timeline

Reactions

Author

gui-marra

Participants

gui-marra

Timeline (top)

closed ×1labeled ×1

Error Message

The LLM call succeeds but prompt caching is silently broken, with no error or warning.

Root Cause

The content TypedDicts don't include cache_control:

ResponseInputTextParam — only has type, text
ResponseInputFileParam — only has type, file_id, file_data, filename
ResponseInputImageParam — only has type, file_id, image_url, detail

But litellm's own is_cached_message() in litellm/utils.py checks for cache_control on these same content items:

def is_cached_message(message: AllMessageValues) -> bool:
    for content in message["content"]:
        if content.get("cache_control") is not None:
            return True

Fix Action

Workaround

Use pydantic.SkipValidation to prevent Pydantic from stripping the field:

from pydantic import BaseModel, SkipValidation
from litellm import ResponseInputParam

class MyModel(BaseModel):
    input_items: SkipValidation[ResponseInputParam]

Code Example

from pydantic import BaseModel
from litellm import ResponseInputParam

class MyModel(BaseModel):
    input_items: ResponseInputParam

msg = {
    "type": "message",
    "role": "user",
    "content": [
        {"type": "input_text", "text": "hello", "cache_control": {"type": "ephemeral"}},
    ],
}

result = MyModel(input_items=[msg])
print(result.input_items[0]["content"][0])
# Output: {'text': 'hello', 'type': 'input_text'}
# cache_control is silently gone!

---

"""Reproducer: Pydantic silently strips cache_control from litellm TypedDicts."""

from pydantic import BaseModel, SkipValidation
from litellm import ResponseInputParam

message_with_cache = {
    "type": "message",
    "role": "user",
    "content": [
        {
            "type": "input_text",
            "text": "Document context here...",
            "cache_control": {"type": "ephemeral"},
        },
        {
            "type": "input_text",
            "text": "What is the answer?",
        },
    ],
}

class BrokenModel(BaseModel):
    input_items: ResponseInputParam

class FixedModel(BaseModel):
    input_items: SkipValidation[ResponseInputParam]

print("=== Original message ===")
content = message_with_cache["content"]
for i, item in enumerate(content):
    print(f"  [{i}] cache_control: {'cache_control' in item}  keys: {list(item.keys())}")

print("\n=== BrokenModel (Pydantic validates against TypedDict) ===")
broken = BrokenModel(input_items=[message_with_cache])
content = broken.input_items[0]["content"]
for i, item in enumerate(content):
    print(f"  [{i}] cache_control: {'cache_control' in item}  keys: {list(item.keys())}")

print("\n=== FixedModel (SkipValidation preserves all fields) ===")
fixed = FixedModel(input_items=[message_with_cache])
content = fixed.input_items[0]["content"]
for i, item in enumerate(content):
    print(f"  [{i}] cache_control: {'cache_control' in item}  keys: {list(item.keys())}")

---

=== Original message ===
  [0] cache_control: True  keys: ['type', 'text', 'cache_control']
  [1] cache_control: False  keys: ['type', 'text']

=== BrokenModel (Pydantic validates against TypedDict) ===
  [0] cache_control: False  keys: ['text', 'type']
  [1] cache_control: False  keys: ['text', 'type']

=== FixedModel (SkipValidation preserves all fields) ===
  [0] cache_control: True  keys: ['type', 'text', 'cache_control']
  [1] cache_control: False  keys: ['type', 'text']

---

def is_cached_message(message: AllMessageValues) -> bool:
    for content in message["content"]:
        if content.get("cache_control") is not None:
            return True

---

# In openai/types or litellm's own types
class ResponseInputTextParam(TypedDict, total=False):
    type: Required[Literal["input_text"]]
    text: Required[str]
    cache_control: CacheControlParam  # {"type": "ephemeral"}

---

from pydantic import BaseModel, SkipValidation
from litellm import ResponseInputParam

class MyModel(BaseModel):
    input_items: SkipValidation[ResponseInputParam]

RAW_BUFFERClick to expand / collapse

What happened?

litellm supports cache_control: {"type": "ephemeral"} on message content items for Gemini and Anthropic explicit prompt caching. litellm's own context caching handler (vertex_ai/context_caching/) reads cache_control from content items via is_cached_message().

However, litellm's TypedDict definitions for content items (ResponseInputTextParam, ResponseInputFileParam, ResponseInputImageParam) don't include cache_control as a field.

This means any application that stores litellm messages in a Pydantic model with ResponseInputParam as a field type will have cache_control silently stripped — Pydantic validates content dicts against the TypedDict union and drops unknown keys.

The LLM call succeeds but prompt caching is silently broken, with no error or warning.

Reproducer

from pydantic import BaseModel
from litellm import ResponseInputParam

class MyModel(BaseModel):
    input_items: ResponseInputParam

msg = {
    "type": "message",
    "role": "user",
    "content": [
        {"type": "input_text", "text": "hello", "cache_control": {"type": "ephemeral"}},
    ],
}

result = MyModel(input_items=[msg])
print(result.input_items[0]["content"][0])
# Output: {'text': 'hello', 'type': 'input_text'}
# cache_control is silently gone!

Full reproducer script (copy-paste and run):

"""Reproducer: Pydantic silently strips cache_control from litellm TypedDicts."""

from pydantic import BaseModel, SkipValidation
from litellm import ResponseInputParam

message_with_cache = {
    "type": "message",
    "role": "user",
    "content": [
        {
            "type": "input_text",
            "text": "Document context here...",
            "cache_control": {"type": "ephemeral"},
        },
        {
            "type": "input_text",
            "text": "What is the answer?",
        },
    ],
}

class BrokenModel(BaseModel):
    input_items: ResponseInputParam

class FixedModel(BaseModel):
    input_items: SkipValidation[ResponseInputParam]

print("=== Original message ===")
content = message_with_cache["content"]
for i, item in enumerate(content):
    print(f"  [{i}] cache_control: {'cache_control' in item}  keys: {list(item.keys())}")

print("\n=== BrokenModel (Pydantic validates against TypedDict) ===")
broken = BrokenModel(input_items=[message_with_cache])
content = broken.input_items[0]["content"]
for i, item in enumerate(content):
    print(f"  [{i}] cache_control: {'cache_control' in item}  keys: {list(item.keys())}")

print("\n=== FixedModel (SkipValidation preserves all fields) ===")
fixed = FixedModel(input_items=[message_with_cache])
content = fixed.input_items[0]["content"]
for i, item in enumerate(content):
    print(f"  [{i}] cache_control: {'cache_control' in item}  keys: {list(item.keys())}")

Output:

=== Original message ===
  [0] cache_control: True  keys: ['type', 'text', 'cache_control']
  [1] cache_control: False  keys: ['type', 'text']

=== BrokenModel (Pydantic validates against TypedDict) ===
  [0] cache_control: False  keys: ['text', 'type']
  [1] cache_control: False  keys: ['text', 'type']

=== FixedModel (SkipValidation preserves all fields) ===
  [0] cache_control: True  keys: ['type', 'text', 'cache_control']
  [1] cache_control: False  keys: ['type', 'text']

Root cause

The content TypedDicts don't include cache_control:

ResponseInputTextParam — only has type, text
ResponseInputFileParam — only has type, file_id, file_data, filename
ResponseInputImageParam — only has type, file_id, image_url, detail

But litellm's own is_cached_message() in litellm/utils.py checks for cache_control on these same content items:

def is_cached_message(message: AllMessageValues) -> bool:
    for content in message["content"]:
        if content.get("cache_control") is not None:
            return True

Suggested fix

Add cache_control to the content TypedDicts:

# In openai/types or litellm's own types
class ResponseInputTextParam(TypedDict, total=False):
    type: Required[Literal["input_text"]]
    text: Required[str]
    cache_control: CacheControlParam  # {"type": "ephemeral"}

Same for ResponseInputFileParam and ResponseInputImageParam.

Relevant code

litellm/utils.py:is_cached_message() — reads cache_control from content items
litellm/llms/vertex_ai/context_caching/ — uses cache_control for Gemini cachedContents
litellm/integrations/anthropic_cache_control_hook.py — uses cache_control for Anthropic

Workaround

Use pydantic.SkipValidation to prevent Pydantic from stripping the field:

from pydantic import BaseModel, SkipValidation
from litellm import ResponseInputParam

class MyModel(BaseModel):
    input_items: SkipValidation[ResponseInputParam]

Environment

litellm version: latest (tested on 1.73+)
pydantic version: 2.x
Python: 3.13

Twitter / LinkedIn details

No response

extent analysis

Fix Plan

To resolve the issue, you can either update the TypedDict definitions to include the cache_control field or use pydantic.SkipValidation to prevent Pydantic from stripping the field.

Option 1: Update TypedDict definitions

Update the ResponseInputTextParam, ResponseInputFileParam, and ResponseInputImageParam TypedDicts to include the cache_control field:

class ResponseInputTextParam(TypedDict, total=False):
    type: Required[Literal["input_text"]]
    text: Required[str]
    cache_control: CacheControlParam  # {"type": "ephemeral"}

class ResponseInputFileParam(TypedDict, total=False):
    type: Required[Literal["input_file"]]
    file_id: Required[str]
    file_data: Required[str]
    filename: Required[str]
    cache_control: CacheControlParam  # {"type": "ephemeral"}

class ResponseInputImageParam(TypedDict, total=False):
    type: Required[Literal["input_image"]]
    file_id: Required[str]
    image_url: Required[str]
    detail: Required[str]
    cache_control: CacheControlParam  # {"type": "ephemeral"}

Option 2: Use `pydantic.SkipValidation`

Use pydantic.SkipValidation to prevent Pydantic from stripping the cache_control field:

from pydantic import BaseModel, SkipValidation
from litellm import ResponseInputParam

class MyModel(BaseModel):
    input_items: SkipValidation[ResponseInputParam]

Verification

To verify that the fix worked, you can create a test case that checks for the presence of the cache_control field in the input_items:

msg = {
    "type": "message",
    "role": "user",
    "content": [
        {"type": "input_text", "text": "hello", "cache_control": {"type": "ephemeral"}},
    ],
}

result = MyModel(input_items=[msg])
print(result.input_items[0]["content"][0].get("cache_control"))  # Should print: {"type": "ephemeral"}

Extra Tips

Make sure to update the TypedDict definitions in the correct location, which is likely in the litellm library.
If you choose to use pydantic.SkipValidation, be aware that this will disable validation for the entire input_items field, which may have unintended consequences.
Consider adding tests to ensure that the cache_control field is properly handled in different scenarios.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#model loading #dependency error #configuration error #environment variable #network issue

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

litellm - 💡(How to fix) Fix [Bug]: ResponseInputTextParam TypedDict missing cache_control field — Pydantic silently strips it [1 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Workaround

Code Example

What happened?

Reproducer

Root cause

Suggested fix

Relevant code

Workaround

Environment

Twitter / LinkedIn details

extent analysis

Fix Plan

Option 1: Update TypedDict definitions

Option 2: Use `pydantic.SkipValidation`

Verification

Extra Tips

Still need to ship something?

TRENDING

litellm - 💡(How to fix) Fix [Bug]: ResponseInputTextParam TypedDict missing cache_control field — Pydantic silently strips it [1 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Workaround

Code Example

What happened?

Reproducer

Root cause

Suggested fix

Relevant code

Workaround

Environment

Twitter / LinkedIn details

extent analysis

Fix Plan

Option 1: Update TypedDict definitions

Option 2: Use pydantic.SkipValidation

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING

Option 2: Use `pydantic.SkipValidation`