Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Bug]: vertex_count_tokens_location ignored in handler.py — Claude count_tokens always routes to us-central1 [4 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#23872•Fetched 2026-04-08 00:54:09
View on GitHub
Comments
1
Participants
2
Timeline
16
Reactions
0
Author
VivaciousDesireVivaciousDesire
Participants
AtharvaJaiswal005AtharvaJaiswal005VivaciousDesireVivaciousDesire
Timeline (top)
referenced ×5cross-referenced ×4labeled ×3subscribed ×2

Error Message

  1. Observe: request routes to us-central1 instead of europe-west1 → 500 error litellm.proxy.anthropic_endpoints.count_tokens(): Exception occurred - Server error '500 Internal Server Error' for url

Root Cause

Root Cause

Fix Action

Fix / Workaround

A workaround was added in common_utils.py (line 1064) to support vertex_count_tokens_location, but it is silently nullified by handler.py further down the call chain.

PR fix notes

PR #23874: fix(vertex_ai): respect vertex_count_tokens_location override in count_tokens handler

Description (problem / solution / changelog)

When using Claude models on Vertex AI with a non-us-central1 location, the count_tokens endpoint always routes to us-central1 regardless of vertex_count_tokens_location config. This causes 500 errors if the project does not have Claude count-tokens available in us-central1.

Root Cause

In vertex_ai_partner_models/count_tokens/handler.py, the code hardcodes us-central1 for Claude models without checking for the vertex_count_tokens_location override:

if not vertex_location or "claude" in model.lower():
    vertex_location = "us-central1"

Fix

Check for vertex_count_tokens_location in litellm_params before defaulting to us-central1:

if not vertex_location or "claude" in model.lower():
    vertex_location = litellm_params.get("vertex_count_tokens_location") or "us-central1"

Fixes #23872

Changed files

  • litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.py (modified, +3/-2)

PR #23875: fix: respect vertex_count_tokens_location in Claude count_tokens handler (#23872)

Description (problem / solution / changelog)

Summary

Respects vertex_count_tokens_location when routing Claude count_tokens requests on Vertex AI. Previously, the handler always overrode the location to us-central1 for Claude models, ignoring the explicit count-tokens location and causing 500 errors for projects without Claude count-tokens in us-central1.

Root Cause

In handler.py, the condition if not vertex_location or "claude" in model.lower() always overrode to us-central1 for any Claude model. vertex_count_tokens_location (resolved in common_utils.py) was never consulted in the handler.

Fix

  • Resolve vertex_count_tokens_location first into a local variable, then fall back to get_vertex_ai_location()
  • Only apply the us-central1 default for Claude when no explicit vertex_count_tokens_location is configured
  • Extract the location param to avoid redundant dict lookups

Testing

  • Python syntax verified
  • Logic preserves existing behavior when vertex_count_tokens_location is not set
  • Backwards-compatible: users without vertex_count_tokens_location still get us-central1 for Claude

Fixes #23872

Changed files

  • litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.py (modified, +12/-3)

PR #23907: fix(vertex): respect vertex_count_tokens_location for Claude count_tokens

Description (problem / solution / changelog)

Relevant issues

Fixes #23872

Pre-Submission checklist

  • I have Added testing in the tests/test_litellm/ directory
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai

Type

🐛 Bug Fix

Changes

The count_tokens handler for Vertex AI partner models unconditionally overrode vertex_location to us-central1 for Claude models (if not vertex_location or "claude" in model.lower()), ignoring the documented vertex_count_tokens_location parameter.

Additionally, us-central1 is no longer a supported region for Claude count_tokens — Google now supports us-east5, europe-west1, and asia-southeast1 (docs).

Fix: vertex_count_tokens_location takes precedence, vertex_location is used as fallback, and us-east5 is the default only when neither is set.

Changed files

  • litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.py (modified, +8/-4)
  • tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/__init__.py (added, +0/-0)
  • tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/test_count_tokens_location.py (added, +164/-0)

PR #24116: fix(vertex): respect vertex_count_tokens_location for Claude models

Description (problem / solution / changelog)

Summary

  • Check vertex_count_tokens_location from litellm_params before falling back to vertex_location
  • Only override to us-central1 when vertex_count_tokens_location is not explicitly set

Root Cause

handler.py unconditionally overrides vertex_location to us-central1 when "claude" is in the model name, ignoring any explicitly configured vertex_count_tokens_location. The fix in common_utils.py correctly resolves the location, but handler.py nullifies it downstream.

Changes

  • litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.py: Check vertex_count_tokens_location first, only fall back to us-central1 when not set

Fixes #23872

Changed files

  • litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.py (modified, +14/-18)

Code Example

# common_utils.py:1064
  vertex_location = (
      count_tokens_params_request.get("vertex_count_tokens_location")
      or vertex_location
  )
  # → correctly resolves to "europe-west1" ✅

  This value is passed to VertexAIPartnerModels.count_tokens() as vertex_location="europe-west1", which copies it into _litellm_params["vertex_location"] and calls
  handle_count_tokens_request().

  However, inside handler.py, the location is re-read from litellm_params and then unconditionally overridden because "claude" is always present in the model name:

  # handler.py:108-113 — BROKEN
  vertex_location = self.get_vertex_ai_location(litellm_params)
  # → reads "europe-west1"

  if not vertex_location or "claude" in model.lower():
      vertex_location = "us-central1"
  # → "claude" is ALWAYS in Claude model names → always overrides to "us-central1" ❌
  # → vertex_count_tokens_location is never consulted here

  ---
  Expected Behavior

  vertex_count_tokens_location should take precedence over the us-central1 default. The default override should only apply when no explicit count-tokens location is configured.

  ---
  Proposed Fix

  In litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.py, change lines 108–113:

  # Current (broken):
  vertex_location = self.get_vertex_ai_location(litellm_params)
  if not vertex_location or "claude" in model.lower():
      vertex_location = "us-central1"

  # Fixed:
  vertex_location = (
      litellm_params.get("vertex_count_tokens_location")
      or self.get_vertex_ai_location(litellm_params)
  )
  if not vertex_location or "claude" in model.lower():
      vertex_location = "us-central1"

  This ensures vertex_count_tokens_location is checked first. If set, the us-central1 override is bypassed. If not set and the model is Claude, it falls back to us-central1 as before.

  ---
  Steps to Reproduce

  1. Configure a Claude model on Vertex AI with vertex_location: europe-west1 and vertex_count_tokens_location: europe-west1
  2. Make a request that triggers token counting
  3. Observe: request routes to us-central1 instead of europe-west1 → 500 error

  LiteLLM config:

---

---
  Call Chain

---

### Relevant log output
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When using Claude models on Vertex AI with a non-us-central1 location (e.g., europe-west1), the count_tokens endpoint always routes to us-central1 regardless of vertex_location or vertex_count_tokens_location config. This causes 500 errors if the project doesn't have Claude count-tokens available in us-central1.

A workaround was added in common_utils.py (line 1064) to support vertex_count_tokens_location, but it is silently nullified by handler.py further down the call chain.

Root Cause

The fix in common_utils.py correctly resolves vertex_count_tokens_location:

# common_utils.py:1064
vertex_location = (
    count_tokens_params_request.get("vertex_count_tokens_location")
    or vertex_location
)
# → correctly resolves to "europe-west1" ✅

This value is passed to VertexAIPartnerModels.count_tokens() as vertex_location="europe-west1", which copies it into _litellm_params["vertex_location"] and calls
handle_count_tokens_request().

However, inside handler.py, the location is re-read from litellm_params and then unconditionally overridden because "claude" is always present in the model name:

# handler.py:108-113 — BROKEN
vertex_location = self.get_vertex_ai_location(litellm_params)
# → reads "europe-west1"

if not vertex_location or "claude" in model.lower():
    vertex_location = "us-central1"
# → "claude" is ALWAYS in Claude model names → always overrides to "us-central1" ❌
# → vertex_count_tokens_location is never consulted here

---
Expected Behavior

vertex_count_tokens_location should take precedence over the us-central1 default. The default override should only apply when no explicit count-tokens location is configured.

---
Proposed Fix

In litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.py, change lines 108–113:

# Current (broken):
vertex_location = self.get_vertex_ai_location(litellm_params)
if not vertex_location or "claude" in model.lower():
    vertex_location = "us-central1"

# Fixed:
vertex_location = (
    litellm_params.get("vertex_count_tokens_location")
    or self.get_vertex_ai_location(litellm_params)
)
if not vertex_location or "claude" in model.lower():
    vertex_location = "us-central1"

This ensures vertex_count_tokens_location is checked first. If set, the us-central1 override is bypassed. If not set and the model is Claude, it falls back to us-central1 as before.

---
Steps to Reproduce

1. Configure a Claude model on Vertex AI with vertex_location: europe-west1 and vertex_count_tokens_location: europe-west1
2. Make a request that triggers token counting
3. Observe: request routes to us-central1 instead of europe-west1 → 500 error

LiteLLM config:

model_list: - model_name: claude-sonnet-4-6 litellm_params: model: vertex_ai/claude-sonnet-4-6 vertex_project: my-gcp-project vertex_location: europe-west1 vertex_count_tokens_location: europe-west1 # ignored due to bug


  ---
  Call Chain

common_utils.py:1064 → resolves vertex_count_tokens_location = "europe-west1" ✅ main.py:323-327 → copies into _litellm_params["vertex_location"] = "europe-west1" ✅ handler.py:108 → re-reads vertex_location from litellm_params = "europe-west1" handler.py:112 → "claude" in model → overrides to "us-central1" ❌ handler.py:123-127 → builds URL with us-central1 → 500


### Relevant log output

```shell
litellm.proxy.anthropic_endpoints.count_tokens(): Exception occurred -
  Server error '500 Internal Server Error' for url
  'https://us-central1-aiplatform.googleapis.com/v1/projects/getmoss-ai/locations/us-central1/publishers/anthropic/models/count-tokens:rawPredict'

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

1.81.14

Twitter / LinkedIn details

https://www.linkedin.com/in/akanksha-singh-2989aa164/

extent analysis

Fix Plan

To resolve the issue, update the handler.py file as follows:

  • Replace the existing code block (lines 108-113) with the fixed version:
vertex_location = (
    litellm_params.get("vertex_count_tokens_location")
    or self.get_vertex_ai_location(litellm_params)
)
if not vertex_location or "claude" in model.lower():
    vertex_location = "us-central1"

This change ensures that vertex_count_tokens_location is checked first, and if set, the us-central1 override is bypassed.

Verification

To verify the fix:

  1. Update the handler.py file with the new code.
  2. Configure a Claude model on Vertex AI with vertex_location: europe-west1 and vertex_count_tokens_location: europe-west1.
  3. Make a request that triggers token counting.
  4. Check the logs to ensure the request routes to europe-west1 instead of us-central1.

Extra Tips

  • Make sure to test the fix with different model configurations to ensure the vertex_count_tokens_location is correctly prioritized.
  • Consider adding additional logging or monitoring to detect similar issues in the future.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#tokenizer error#prompt formatting#chain error#conversation history

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

litellm - ✅(Solved) Fix [Bug]: vertex_count_tokens_location ignored in handler.py — Claude count_tokens always routes to us-central1 [4 pull requests, 1 comments, 2 participants]