Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Feature]: Support Grok models on Vertex AI [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#25668•Fetched 2026-04-14 05:38:18
View on GitHub
Comments
0
Participants
1
Timeline
3
Reactions
0
Author
lephuongbglephuongbg
Participants
lephuongbglephuongbg
Timeline (top)
labeled ×3

Error Message

litellm.APIConnectionError: Value frozenset({'MALFORMED_RESPONSE', 'TOO_MANY_TOOL_CALLS', 'FINISH_REASON_UNSPECIFIED', 'SAFETY', 'LANGUAGE', 'IMAGE_SAFETY', 'MAX_TOKENS', 'PROHIBITED_CONTENT', 'OTHER', 'RECITATION', 'STOP', 'MALFORMED_FUNCTION_CALL', 'IMAGE_PROHIBITED_CONTENT', 'SPII', 'BLOCKLIST'}) has unexpected type <class 'frozenset'> at Value._GEMINI_FINISH_REASON_KEYS. Received Model Group=xai/grok-4.1-fast-non-reasoning\nAvailable Model Group Fallbacks=None

PR fix notes

PR #25896: feat(vertex_ai): Add xAI Grok models support as Vertex AI partner models

Description (problem / solution / changelog)

Relevant issues

<!-- e.g. "Fixes #000" -->

Fixes #25668

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.

  • Branch creation CI run
    Link:

  • CI run for the last commit
    Link:

  • Merge / cherry-pick CI run
    Links:

Screenshots / Proof of Fix

<!-- Include screenshots, screen recordings, or log output demonstrating that your changes work as expected. For bug fixes: show reproduction before the fix and passing behavior after. For new features: show the feature working end-to-end. For UI changes: include before/after screenshots. -->

The original error reported in #25668 when calling any xai/grok-* model on Vertex AI:

litellm.APIConnectionError: Value frozenset({'MALFORMED_RESPONSE', 'TOO_MANY_TOOL_CALLS', 'FINISH_REASON_UNSPECIFIED', 'SAFETY', 'LANGUAGE', 'IMAGE_SAFETY', 'MAX_TOKENS', 'PROHIBITED_CONTENT', 'OTHER', 'RECITATION', 'STOP', 'MALFORMED_FUNCTION_CALL', 'IMAGE_PROHIBITED_CONTENT', 'SPII', 'BLOCKLIST'}) has unexpected type <class 'frozenset'> at Value._GEMINI_FINISH_REASON_KEYS. Received Model Group=xai/grok-4.1-fast-non-reasoning\nAvailable Model Group Fallbacks=None

Root cause: xai/grok-* models were not registered as Vertex AI partner models, so they fell through to the Gemini handler — which crashed trying to parse the OpenAI-format response using Gemini-specific logic (_GEMINI_FINISH_REASON_KEYS).

After fix: xai/grok-* models are correctly routed through the OpenAI-compatible partner model handler (same infrastructure as Llama 3 on Vertex AI), hitting the /endpoints/openapi/chat/completions endpoint. (as verified with unit tests added)

Type

<!-- Select the type of Pull Request --> <!-- Keep only the necessary ones -->

🆕 New Feature 🐛 Bug Fix

Changes

File Changes

FileChange
litellm/llms/vertex_ai/vertex_ai_partner_models/main.pyAdded "xai" prefix to PartnerModelPrefixes; added xai to is_vertex_partner_model and should_use_openai_handler
litellm/llms/vertex_ai/vertex_ai_partner_models/__init__.pyAdded "xai" to the get_vertex_ai_partner_model_config()
litellm/llms/vertex_ai/vertex_llm_base.pyAdded VertexPartnerProvider.xai case in create_vertex_url()
litellm/types/llms/vertex_ai.pyAdded "xai" to VertexPartnerProvider enum
litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.pyAdded xai publisher detection in _get_publisher_for_model()

Pricing entries

Added the 4 Grok models available on Vertex AI to pricing JSON file (model_prices_and_context_window_backup.json) (added in backup JSON file also): in original issue author wrote 3 models but verified on vertex-ai that there are 4 models available. see- https://docs.cloud.google.com/vertex-ai/generative-ai/docs/partner-models/grok

numbers were verified from the xAI website (https://docs.x.ai/developers/models)

  • vertex_ai/xai/grok-4.1-fast-non-reasoning
  • vertex_ai/xai/grok-4.1-fast-reasoning
  • vertex_ai/xai/grok-4.20-non-reasoning
  • vertex_ai/xai/grok-4.20-reasoning

Tests

Added tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/grok/test_vertex_partner_xai_routing.py covering:

  1. PartnerModelPrefixes enum registration
  2. is_vertex_partner_model() recognition for all 4 Grok models
  3. should_use_openai_handler() routing decision
  4. VertexPartnerProvider.xai enum membership
  5. create_vertex_url() produces correct /endpoints/openapi/chat/completions URL
  6. Pricing entries present in model_cost for all 4 models
  7. Correct supports_reasoning flags per model variant
  8. get_vertex_ai_partner_model_config() returns VertexAILlama3Config for xai spec
  9. Token counter publisher detection for xai models
  10. tests confirming other partner models are unaffected

Changed files

  • litellm/llms/vertex_ai/vertex_ai_partner_models/__init__.py (modified, +1/-0)
  • litellm/llms/vertex_ai/vertex_ai_partner_models/count_tokens/handler.py (modified, +3/-0)
  • litellm/llms/vertex_ai/vertex_ai_partner_models/main.py (modified, +7/-1)
  • litellm/llms/vertex_ai/vertex_llm_base.py (modified, +8/-2)
  • litellm/model_prices_and_context_window_backup.json (modified, +61/-1)
  • litellm/types/llms/vertex_ai.py (modified, +1/-0)
  • model_prices_and_context_window.json (modified, +84/-0)
  • tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/grok/test_vertex_partner_xai_routing.py (added, +239/-0)

Code Example

litellm.APIConnectionError: Value frozenset({'MALFORMED_RESPONSE', 'TOO_MANY_TOOL_CALLS', 'FINISH_REASON_UNSPECIFIED', 'SAFETY', 'LANGUAGE', 'IMAGE_SAFETY', 'MAX_TOKENS', 'PROHIBITED_CONTENT', 'OTHER', 'RECITATION', 'STOP', 'MALFORMED_FUNCTION_CALL', 'IMAGE_PROHIBITED_CONTENT', 'SPII', 'BLOCKLIST'}) has unexpected type <class 'frozenset'> at Value._GEMINI_FINISH_REASON_KEYS. Received Model Group=xai/grok-4.1-fast-non-reasoning\nAvailable Model Group Fallbacks=None
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

The Feature

3 Grok models have recently been made available on Vertex AI:

  • xai/grok-4.1-fast-non-reasoning
  • xai/grok-4.1-fast-reasoning
  • xai/grok-4.20-non-reasoning

Would be great if litellm can support it. Currently trying to access these models through LiteLLM will results in following error, might be due to lack of correct mapping:

litellm.APIConnectionError: Value frozenset({'MALFORMED_RESPONSE', 'TOO_MANY_TOOL_CALLS', 'FINISH_REASON_UNSPECIFIED', 'SAFETY', 'LANGUAGE', 'IMAGE_SAFETY', 'MAX_TOKENS', 'PROHIBITED_CONTENT', 'OTHER', 'RECITATION', 'STOP', 'MALFORMED_FUNCTION_CALL', 'IMAGE_PROHIBITED_CONTENT', 'SPII', 'BLOCKLIST'}) has unexpected type <class 'frozenset'> at Value._GEMINI_FINISH_REASON_KEYS. Received Model Group=xai/grok-4.1-fast-non-reasoning\nAvailable Model Group Fallbacks=None

Motivation, pitch

What part of LiteLLM is this about?

Proxy

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

Twitter / LinkedIn details

No response

extent analysis

TL;DR

The error is likely due to a missing or incorrect mapping for the new Grok models in LiteLLM, and a fix may involve updating the model mappings to support the new models.

Guidance

  • Verify that the new Grok models are correctly configured and registered in LiteLLM's model mappings.
  • Check the documentation for LiteLLM's APIConnectionError to see if there are any specific requirements or restrictions for model mappings.
  • Investigate the Value._GEMINI_FINISH_REASON_KEYS error to determine if it is related to the model mapping issue or a separate problem.
  • Consider reaching out to the LiteLLM development team or checking their documentation for guidance on supporting new models.

Notes

The error message suggests that there is an issue with the model mapping, but without more information about LiteLLM's internal workings, it is difficult to provide a more specific solution.

Recommendation

Apply workaround: Update the model mappings in LiteLLM to support the new Grok models, if possible, or wait for an official update from the LiteLLM development team.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#tensor shape#autograd error#model save/load#optimization

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING