Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

This bug was previously reported in #17304 (2025-12-01, v1.80.7) and proposed the correct fix ("move those values to CachedContent from GenerateContent request"). That issue was auto-closed as not_planned on 2026-03-10 by the stale bot without any maintainer engagement. A community PR (#23986, 2026-03-18) attempted the fix but was closed because it over-scoped and introduced a TypeError. The correct 3-line change has never landed.

I'm re-filing because the bug is still present on the latest release (1.83.9) and is actively breaking production traffic for us on vertex_ai/gemini-3.1-pro-preview.

What happened?

When an OpenAI-format request contains cache_control: {"type": "ephemeral"} markers on system messages (Anthropic prompt-caching convention), LiteLLM's Vertex AI path translates those into Vertex's Context Cache API. But the generateContent request body is then assembled with both cachedContent AND system_instruction (+ tools / toolConfig), which Vertex explicitly rejects:

Vertex_aiException BadRequestError - {
  "error": {
    "code": 400,
    "message": "Tool config, tools and system instruction should not be set in the request when using cached content.",
    "status": "INVALID_ARGUMENT"
  }
}

Root cause (two interacting issues)

(1) separate_cached_messages only keeps the first continuous block of cache-marked messages. In litellm/llms/vertex_ai/context_caching/transformation.py:

last_continuous_block_idx = get_first_continuous_block_idx(filtered_messages)
if filtered_messages and last_continuous_block_idx is not None:
    first_cached_idx = filtered_messages[0][0]
    last_cached_idx = filtered_messages[last_continuous_block_idx][0]
    cached_messages = messages[first_cached_idx : last_cached_idx + 1]
    non_cached_messages = (
        messages[:first_cached_idx] + messages[last_cached_idx + 1 :]
    )

Any role=system message that has cache_control but sits after an uncached message gets dropped into non_cached_messages. Those leftover system messages then get turned into Vertex's system_instruction field. This is common in real-world agent frameworks that interleave cached and uncached system blocks (e.g. RAG contexts, skill catalogs, tool definitions).

(2) The generateContent body builder unconditionally adds system_instruction / tools / toolConfig even when cachedContent is set. In litellm/llms/vertex_ai/gemini/transformation.py (lines 749–765 on [email protected]):

data = RequestBody(contents=content)
if system_instructions is not None:
    data["system_instruction"] = system_instructions
if tools is not None:
    data["tools"] = tools
if tool_choice is not None:
    data["toolConfig"] = tool_choice
...
if cached_content is not None:
    data["cachedContent"] = cached_content   # <-- collision with the three above

Per Vertex AI docs, cachedContent is mutually exclusive with system_instruction, tools, and toolConfig in the same generateContent call — those must be baked into the cache at creation time, not re-sent.

Steps to Reproduce

Minimal reproduction using the SDK (works against the proxy too). The payload mimics what an agent loop produces: a large system block with cache_control, followed by a plain system block (the thing that breaks it), followed by another cache_control block.

import asyncio
import litellm

litellm.use_litellm_proxy = True  # optional, if routing through your proxy
MODEL = "vertex_ai/gemini-3.1-pro-preview"  # reproduces on gemini-2.5-pro etc. too

BIG = "AVAILABLE DATA CONTEXT\n" + ("glossary_entry_token " * 8000)   # > 32K tokens

def cached(text):
    return {
        "role": "system",
        "content": [
            {"type": "text", "text": text, "cache_control": {"type": "ephemeral"}}
        ],
    }

messages = [
    cached(BIG),                                                         # cached
    {"role": "system", "content": "AVAILABLE SKILLS\n- a\n- b"},         # breaks the block
    cached("TOOL DEFINITIONS\n" + ("tool_def_token " * 8000)),           # NOT cached anymore
    {"role": "user", "content": "Summarize."},
]

tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Get weather",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
    "cache_control": {"type": "ephemeral"},
}}]

async def main():
    try:
        resp = await litellm.acompletion(model=MODEL, messages=messages, tools=tools)
        print(resp.choices[0].message.content[:80])
    except Exception as e:
        print(type(e).__name__, str(e)[:600])

asyncio.run(main())

Observed:

BadRequestError litellm.BadRequestError: ... Vertex_aiException BadRequestError - {
  "error": { "code": 400,
    "message": "Tool config, tools and system instruction should not be set in the request when using cached content.",
    "status": "INVALID_ARGUMENT" } }

Confirmed on 1.81.8 (SDK), 1.81.0 / 1.83.3 / 1.83.9 (proxy source inspection — byte-identical generateContent body builder across all three).

Suggested fix (3 lines)

Guard the three conflicting fields in litellm/llms/vertex_ai/gemini/transformation.py so they are only added when cached_content is None. This was exactly the fix in the core diff of #23986 (which should have been merged with only this change):

 data = RequestBody(contents=content)
-if system_instructions is not None:
+if system_instructions is not None and cached_content is None:
     data["system_instruction"] = system_instructions
-if tools is not None:
+if tools is not None and cached_content is None:
     data["tools"] = tools
-if tool_choice is not None:
+if tool_choice is not None and cached_content is None:
     data["toolConfig"] = tool_choice
 ...
 if cached_content is not None:
     data["cachedContent"] = cached_content

This is the simplest correct fix — it matches Vertex AI's documented API contract (those three fields live in the cache, not in the call) and preserves all existing behavior when cached_content is not set.

Optional follow-on fix for the separator issue (#1 above): either include all cache-marked messages in cached_messages (not just the first continuous block), or raise a clear error when the caller mixes cache_control patterns in a way Vertex can't represent. The 3-line guard above is sufficient to stop the 400; the separator logic just becomes less useful (fewer messages get cached) rather than broken.

Relevant log output

litellm.BadRequestError: Error code: 400 - ... Vertex_aiException BadRequestError - {
  "error": {
    "code": 400,
    "message": "Tool config, tools and system instruction should not be set in the request when using cached content.",
    "status": "INVALID_ARGUMENT"
  }
}

What part of LiteLLM is this about?

Both SDK and Proxy (shared code path in llms/vertex_ai/).

What LiteLLM version are you on ?

Reproduced on 1.81.8 (SDK), confirmed unchanged source in 1.81.0 / 1.83.3 / 1.83.9.

Related

• #17304 — original bug report, closed as not_planned by stale bot without maintainer reply
• #23986 — community PR with the correct 3-line fix, closed because the PR also contained unrelated deletions

Twitter / LinkedIn details

No response