Check for existing issues

• I have searched existing issues — #15858 and #24369 are related but do not cover this case.

What happened?

When using Gemini's url_context tool (distinct from Grounding with Google Search), LiteLLM incorrectly charges the $0.035 web-search grounding fee per request. Root cause: the detection logic in tool_call_cost_tracking.py treats any url_citation annotation in the response as evidence of a web_search call — but Gemini's url_context tool also emits url_citation annotations (for per-claim grounding against the fetched URL).

Per Google's pricing documentation:

• url_context has no per-request surcharge; it is "Charged as input tokens per model pricing." Sources:
  • https://ai.google.dev/gemini-api/docs/url-context ("URL Context Tool … Charged as input tokens per model pricing")
  • https://ai.google.dev/gemini-api/docs/pricing (no url_context line item)
  • https://cloud.google.com/vertex-ai/generative-ai/pricing (grounding options listed are Google Search, Maps, Your Data — url_context is not a listed grounding product)
• Grounding with Google Search on Gemini 3.x is $14/1,000 queries, with 5,000 free/month. (Addressed separately by #24369 / PR #24448.)

So url_context requests are currently being over-billed in LiteLLM's cost ledger. On a 6-URL test probe, LiteLLM reported a total of $0.211 when Google's actual billing for pure url_context calls should be a few cents total (token cost only). This creates a systematic overcount ~1000× for url_context-heavy workloads.

Reproduction

import httpx

# Any Gemini Vertex model alias exposing url_context via the proxy
payload = {
    "model": "gemini-3-flash-preview-vertex",
    "messages": [{
        "role": "user",
        "content": (
            "Use the url_context tool to fetch "
            "https://pmc.ncbi.nlm.nih.gov/articles/PMC10018306/ "
            "and extract: 1) title, 2) publication date, 3) journal, "
            "4) a 2-sentence summary."
        ),
    }],
    "tools": [{"url_context": {}}],  # note: NOT web_search / googleSearch
}

r = httpx.post(
    "http://localhost:4000/v1/chat/completions",
    headers={"Authorization": "Bearer sk-..."},
    json=payload,
    timeout=180.0,
)
print("litellm cost:", r.headers.get("x-litellm-response-cost"))
print("vertex_ai_url_context_metadata populated:",
      bool(r.json().get("vertex_ai_url_context_metadata")))
print("vertex_ai_grounding_metadata populated:",
      bool(r.json().get("vertex_ai_grounding_metadata")))

Observed:

• x-litellm-response-cost: 0.0353 (token cost + $0.035 grounding surcharge)
• vertex_ai_url_context_metadata populated with URL_RETRIEVAL_STATUS_SUCCESS — proves url_context fired
• vertex_ai_grounding_metadata populated with groundingChunks / groundingSupports — these populate when the model grounds an answer, regardless of which tool sourced the grounding
• No vertex_ai_grounding_metadata[].webSearchQueries field — meaning web_search was not called
• message.annotations contains url_citation entries → this is what trips the false positive in the cost calculator

Expected:

• No grounding surcharge for url_context-only calls. Cost should equal prompt_tokens × input_rate + completion_tokens × output_rate per model tier.

Root cause

litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py, response_object_includes_web_search_call() at line ~316:

if isinstance(response_object, ModelResponse):
    # chat completions only include url_citation annotations when a web search call is made
    has_url_citations = (
        StandardBuiltInToolCostTracking.response_includes_annotation_type(
            response_object=response_object, annotation_type="url_citation"
        )
    )
    if has_url_citations:
        return True    # ← false positive: url_context also emits url_citation

The comment "chat completions only include url_citation annotations when a web search call is made" was correct when this code was written, but is no longer true now that Gemini's url_context tool emits the same annotation type for per-claim grounding against user-specified URLs.

Proposed fix

Use the vertex_ai_url_context_metadata field (already populated by LiteLLM's Gemini adapter at litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py:2475-2479) as a negative signal: if url_context metadata is present, the annotation-based shortcut should not fire; fall through to the structured web_search_requests check, which correctly distinguishes the two tools.

Diff sketch:

 if isinstance(response_object, ModelResponse):
-    # chat completions only include url_citation annotations when a web search call is made
     has_url_citations = (
         StandardBuiltInToolCostTracking.response_includes_annotation_type(
             response_object=response_object, annotation_type="url_citation"
         )
     )
-    if has_url_citations:
+    # Gemini's url_context tool also emits url_citation annotations, so annotation
+    # presence alone is not sufficient for Gemini/Vertex. Skip this shortcut when
+    # url_context metadata is present and rely on usage.prompt_tokens_details.web_search_requests
+    # below to detect actual web_search calls.
+    if has_url_citations and not StandardBuiltInToolCostTracking.response_object_includes_url_context_call(response_object):
         return True

…plus a small response_object_includes_url_context_call helper that reads vertex_ai_url_context_metadata off the response object (already attached by the Gemini adapter).

Will open a PR with the fix + regression tests shortly.

Related

• #15858 — earlier bug that added the annotation-based detection in the first place (the fix introduced this false positive).
• #24369 / PR #24448 — separate bug: hardcoded $0.035 rate should be $0.014 for Gemini 3.x per current pricing. These cover the rate; this issue covers the trigger.

What part of LiteLLM is this about?

Cost tracking / Vertex AI Gemini / url_context tool.

What LiteLLM version are you on?

main (reproduced on litellm/llms/vertex_ai/gemini/cost_calculator.py + tool_call_cost_tracking.py as of the latest commit on main at time of writing).