litellm - 💡(How to fix) Fix [Feature]: Support for mREP endpoint for vertex AI [1 comments, 2 participants]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

The Feature

Vertex AI Multi-Region Endpoints (mREPs) generate an incorrect URL when using api_base with vertex_ai/ models.

Google recently introduced for Vertex AI, which guarantee data residency within a jurisdiction. The mREP base URL format is https://aiplatform.us.rep.googleapis.com (note: no region prefix like us-central1-).

Expected URL format (per Google's docs):

https://aiplatform.us.rep.googleapis.com/v1/projects/{PROJECT}/locations/us/publishers/google/models/{MODEL}:generateContent

Actual URL generated by LiteLLM:

https://aiplatform.us.rep.googleapis.com:generateContent

Relevant code

In litellm/llms/vertex_ai/vertex_llm_base.py, the _check_custom_proxy method handles custom api_base for Vertex AI. When api_base is set and use_psc_endpoint_format is False (the default), it falls through to:

url = "{}:{}".format(api_base, endpoint)

This strips the entire /v1/projects/.../locations/.../publishers/google/models/... path and just appends :{endpoint} to the base URL.

The use_psc_endpoint_format=True path is also not correct for mREPs because it uses /endpoints/ instead of /publishers/google/models/:

url = "{}/{}/projects/{}/locations/{}/endpoints/{}:{}".format(
    api_base.rstrip("/"),
    version,
    vertex_project,
    vertex_location,
    model_for_url,
    endpoint,
)

Expected behavior

When api_base is set to an mREP endpoint like https://aiplatform.us.rep.googleapis.com, the generated URL should preserve the standard Vertex AI path structure:

{api_base}/v1/projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{model}:{endpoint}

How to reproduce

# litellm_config.yaml
model_list:
  - model_name: gemini-3-pro-preview
    litellm_params:
      model: vertex_ai/gemini-3.1-pro-preview
      vertex_project: my-project
      vertex_location: us
      api_base: https://aiplatform.us.rep.googleapis.com
      vertex_credentials: /path/to/creds.json

import litellm

response = litellm.completion(
    model="gemini-3.1-pro-preview",
    messages=[{"role": "user", "content": "hello"}],
)

The request will be sent to https://aiplatform.us.rep.googleapis.com:generateContent instead of https://aiplatform.us.rep.googleapis.com/v1/projects/my-project/locations/us/publishers/google/models/gemini-3.1-pro-preview:generateContent.

Current workaround

• Putting the whole URL as api_base https://aiplatform.us.rep.googleapis.com/v1/projects/my-project/locations/us/publishers/google/models/gemini-3.1-pro-preview so that the api:endpoint forms the correct endpoint.

LiteLLM version

1.80.8

Motivation, pitch

Google recently introduced for Vertex AI, which guarantee data residency within a jurisdiction. Given it is a key feature for a lot of org it would be nice to have it as feature out of the box.

What part of LiteLLM is this about?

Proxy

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

Twitter / LinkedIn details

https://www.linkedin.com/in/bharatsinha23/