Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Bug]: `/v1beta/models/{model}:generateContent` returns 500 instead of 400/404 for non-existent model [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#25466•Fetched 2026-04-11 06:14:11
View on GitHub
Comments
0
Participants
1
Timeline
4
Reactions
0
Author
mai-nakagawamai-nakagawa
Participants
mai-nakagawamai-nakagawa
Timeline (top)
labeled ×3cross-referenced ×1

Error Message

INFO: 127.0.0.1:60495 - "POST /v1beta/models/nonexistent-model%3AgenerateContent HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ self.scope, self.receive, self.send ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/applications.py", line 1139, in call await super().call(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/applications.py", line 107, in call await self.middleware_stack(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in call raise exc File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/Users/cont-m-nakagawa/vscode/litellm/litellm/proxy/middleware/in_flight_requests_middleware.py", line 46, in call await self.app(scope, receive, send) File "/Users/cont-m-nakagawa/vscode/litellm/litellm/proxy/middleware/prometheus_auth_middleware.py", line 37, in call await self.app(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/cors.py", line 85, in call await self.app(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call await self.app(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/routing.py", line 716, in call await self.middleware_stack(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/routing.py", line 736, in app await route.handle(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/routing.py", line 290, in handle await self.app(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 120, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 106, in app response = await f(request) ^^^^^^^^^^^^^^^^ File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 430, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... ) ^ File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 316, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/proxy/google_endpoints/endpoints.py", line 75, in google_generate_content response = await llm_router.agenerate_content(**data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5144, in async_wrapper return await self._ageneric_api_call_with_fallbacks( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... ) ^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3800, in _ageneric_api_call_with_fallbacks raise e File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3787, in _ageneric_api_call_with_fallbacks response = await self.async_function_with_fallbacks(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5613, in async_function_with_fallbacks return await self.async_function_with_fallbacks_common_utils( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<8 lines>... ) ^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5570, in async_function_with_fallbacks_common_utils raise original_exception File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5604, in async_function_with_fallbacks response = await self.async_function_with_retries(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5759, in async_function_with_retries self.should_retry_this_error( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ error=e, ^^^^^^^^ ...<4 lines>... content_policy_fallbacks=content_policy_fallbacks, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5963, in should_retry_this_error raise error File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5710, in async_function_with_retries response = await self.make_call(original_function, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5878, in make_call response = await response ^^^^^^^^^^^^^^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3927, in _ageneric_api_call_with_fallbacks_helper raise e File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3854, in _ageneric_api_call_with_fallbacks_helper raise e File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3845, in _ageneric_api_call_with_fallbacks_helper deployment = await self.async_get_available_deployment( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... ) ^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9461, in async_get_available_deployment raise e File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9346, in async_get_available_deployment healthy_deployments = await self.async_get_healthy_deployments( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<6 lines>... ) ^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9192, in async_get_healthy_deployments model, healthy_deployments = self._common_checks_available_deployment( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ model=model, ^^^^^^^^^^^^ ...<3 lines>... request_kwargs=request_kwargs, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) # type: ignore ^ File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9161, in _common_checks_available_deployment raise litellm.BadRequestError( ...<3 lines>... ) litellm.exceptions.BadRequestError: litellm.BadRequestError: You passed in model=nonexistent-model. There are no healthy deployments for this model. Received Model Group=nonexistent-model Available Model Group Fallbacks=None

Fix Action

Fixed

PR fix notes

PR #25469: fix(proxy): return 400 instead of 500 for non-existent model

Description (problem / solution / changelog)

Relevant issues

Fixes #25466

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.

  • Branch creation CI run
    Link:

  • CI run for the last commit
    Link:

  • Merge / cherry-pick CI run
    Links:

Type

<!-- Select the type of Pull Request --> <!-- Keep only the necessary ones -->

🐛 Bug Fix

Changes

Problem

/v1beta/models/{model}:generateContent and :streamGenerateContent returned 500 Internal Server Error when a non-existent model name was passed, instead of 400 Bad Request.

The root cause: the three handlers in litellm/proxy/google_endpoints/endpoints.py called llm_router.agenerate_content() (and the stream/countTokens equivalents) without a try/except. When the router raises litellm.BadRequestError (status_code=400) for an unknown model, the exception propagated unhandled to FastAPI's default handler, which returned 500.

This caused clients such as Gemini CLI to treat the error as a transient server failure and retry indefinitely.

The Anthropic-compatible endpoint (/v1/messages) and the interactions endpoints in the same file (/v1beta/interactions) were already handling this correctly via ProxyBaseLLMRequestProcessing.

Fix

Refactored google_generate_content, google_stream_generate_content, and google_count_tokens in litellm/proxy/google_endpoints/endpoints.py to use the same error-handling pattern already present in the interactions endpoints:

  • google_generate_content and google_stream_generate_content: replaced the manual add_litellm_data_to_request + function_setup + direct router call with ProxyBaseLLMRequestProcessing.base_process_llm_request() wrapped in try/except → _handle_llm_api_exception(). This also integrates pre/post-call hooks, guardrails, and cost tracking consistently with other endpoints.
  • google_count_tokens: added try/except around internal_token_counter() that converts exceptions to ProxyException with the upstream status code.
  • Removed unused imports (datetime, uuid, litellm, HTTPException, StreamingResponse) that were only needed by the old code path.

Tests

Rewrote tests/test_litellm/proxy/google_endpoints/test_google_api_endpoints.py:

  • Updated all existing tests to mock ProxyBaseLLMRequestProcessing.base_process_llm_request instead of the old internal implementation details.

  • Added two regression tests:

    • test_google_generate_content_returns_400_for_nonexistent_model
    • test_google_stream_generate_content_returns_400_for_nonexistent_model

    Both verify that a litellm.BadRequestError from the router results in a 400 response, not 500.

Changed files

  • litellm/proxy/google_endpoints/endpoints.py (modified, +84/-71)
  • tests/test_litellm/proxy/google_endpoints/test_google_api_endpoints.py (modified, +194/-433)

Code Example

curl -v \
  -X POST \
  -H "Content-Type: application/json" \
  -H "x-api-key: sk-test-1234" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "nonexistent-model", "messages": [{"role": "user", "content": "hello"}], "max_tokens": 1024}' \
  'http://localhost:4000/v1/messages'

---

{"error":{"message":"400: {'error': 'anthropic_messages: Invalid model name passed in model=nonexistent-model. Call `/v1/models` to view available models for your key.'}","type":"None","param":"None","code":"400"}}

---

make install-proxy-dev

---

model_list:
  - model_name: gemini-2.0-flash
    litellm_params:
      model: gemini/gemini-2.0-flash
      api_key: os.environ/GEMINI_API_KEY

general_settings:
  master_key: sk-test-1234

---

export GEMINI_API_KEY="<your-gemini-api-key>"
poetry run litellm --config test_config.yaml --port 4000

---

curl -v \
  -X POST \
  -H "Authorization: Bearer sk-test-1234" \
  -H "Content-Type: application/json" \
  -d '{"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}' \
  'http://localhost:4000/v1beta/models/nonexistent-model:generateContent'

---

curl -v \
  -X POST \
  -H "Content-Type: application/json" \
  -H "x-api-key: sk-test-1234" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "nonexistent-model", "messages": [{"role": "user", "content": "hello"}], "max_tokens": 1024}' \
  'http://localhost:4000/v1/messages'

---

INFO:     127.0.0.1:60495 - "POST /v1beta/models/nonexistent-model%3AgenerateContent HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/applications.py", line 1139, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/applications.py", line 107, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/proxy/middleware/in_flight_requests_middleware.py", line 46, in __call__
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/proxy/middleware/prometheus_auth_middleware.py", line 37, in __call__
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/routing.py", line 290, in handle
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 120, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 106, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 430, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 316, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/proxy/google_endpoints/endpoints.py", line 75, in google_generate_content
    response = await llm_router.agenerate_content(**data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5144, in async_wrapper
    return await self._ageneric_api_call_with_fallbacks(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3800, in _ageneric_api_call_with_fallbacks
    raise e
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3787, in _ageneric_api_call_with_fallbacks
    response = await self.async_function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5613, in async_function_with_fallbacks
    return await self.async_function_with_fallbacks_common_utils(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<8 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5570, in async_function_with_fallbacks_common_utils
    raise original_exception
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5604, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5759, in async_function_with_retries
    self.should_retry_this_error(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        error=e,
        ^^^^^^^^
    ...<4 lines>...
        content_policy_fallbacks=content_policy_fallbacks,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5963, in should_retry_this_error
    raise error
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5710, in async_function_with_retries
    response = await self.make_call(original_function, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5878, in make_call
    response = await response
               ^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3927, in _ageneric_api_call_with_fallbacks_helper
    raise e
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3854, in _ageneric_api_call_with_fallbacks_helper
    raise e
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3845, in _ageneric_api_call_with_fallbacks_helper
    deployment = await self.async_get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9461, in async_get_available_deployment
    raise e
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9346, in async_get_available_deployment
    healthy_deployments = await self.async_get_healthy_deployments(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9192, in async_get_healthy_deployments
    model, healthy_deployments = self._common_checks_available_deployment(
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<3 lines>...
        request_kwargs=request_kwargs,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )  # type: ignore
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9161, in _common_checks_available_deployment
    raise litellm.BadRequestError(
    ...<3 lines>...
    )
litellm.exceptions.BadRequestError: litellm.BadRequestError: You passed in model=nonexistent-model. There are no healthy deployments for this model. Received Model Group=nonexistent-model
Available Model Group Fallbacks=None
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When calling LiteLLM's Google AI-compatible endpoint (/v1beta/models/{model}:generateContent) with a non-existent model name, the server returns 500 Internal Server Error instead of an appropriate 400 Bad Request or 404 Not Found.

Expected: LiteLLM should return 400 Bad Request or 404 Not Found with a descriptive error message, since specifying a non-existent model is a client-side error.

For comparison, the Anthropic-compatible endpoint (/v1/messages) correctly returns 400 Bad Request when an invalid model is specified:

curl -v \
  -X POST \
  -H "Content-Type: application/json" \
  -H "x-api-key: sk-test-1234" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "nonexistent-model", "messages": [{"role": "user", "content": "hello"}], "max_tokens": 1024}' \
  'http://localhost:4000/v1/messages'

Response (correct behavior — 400 Bad Request):

{"error":{"message":"400: {'error': 'anthropic_messages: Invalid model name passed in model=nonexistent-model. Call `/v1/models` to view available models for your key.'}","type":"None","param":"None","code":"400"}}

This causes clients like Gemini CLI to treat the error as a transient server-side failure and retry indefinitely, since 5xx errors are typically considered retryable.

Steps to Reproduce

  1. Install proxy dependencies:
make install-proxy-dev
  1. Create test_config.yaml:
model_list:
  - model_name: gemini-2.0-flash
    litellm_params:
      model: gemini/gemini-2.0-flash
      api_key: os.environ/GEMINI_API_KEY

general_settings:
  master_key: sk-test-1234
  1. Start the proxy:
export GEMINI_API_KEY="<your-gemini-api-key>"
poetry run litellm --config test_config.yaml --port 4000
  1. Send a request with a non-existent model and observe the 500 error:
curl -v \
  -X POST \
  -H "Authorization: Bearer sk-test-1234" \
  -H "Content-Type: application/json" \
  -d '{"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}' \
  'http://localhost:4000/v1beta/models/nonexistent-model:generateContent'

For reference, the Anthropic-compatible endpoint (/v1/messages) returns 400 correctly under the same setup:

curl -v \
  -X POST \
  -H "Content-Type: application/json" \
  -H "x-api-key: sk-test-1234" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "nonexistent-model", "messages": [{"role": "user", "content": "hello"}], "max_tokens": 1024}' \
  'http://localhost:4000/v1/messages'

Response: 400 Bad Request

Relevant log output

INFO:     127.0.0.1:60495 - "POST /v1beta/models/nonexistent-model%3AgenerateContent HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/applications.py", line 1139, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/applications.py", line 107, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/proxy/middleware/in_flight_requests_middleware.py", line 46, in __call__
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/proxy/middleware/prometheus_auth_middleware.py", line 37, in __call__
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/routing.py", line 290, in handle
    await self.app(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 120, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 106, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 430, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/Library/Caches/pypoetry/virtualenvs/litellm--OoZ58KN-py3.13/lib/python3.13/site-packages/fastapi/routing.py", line 316, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/proxy/google_endpoints/endpoints.py", line 75, in google_generate_content
    response = await llm_router.agenerate_content(**data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5144, in async_wrapper
    return await self._ageneric_api_call_with_fallbacks(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3800, in _ageneric_api_call_with_fallbacks
    raise e
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3787, in _ageneric_api_call_with_fallbacks
    response = await self.async_function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5613, in async_function_with_fallbacks
    return await self.async_function_with_fallbacks_common_utils(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<8 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5570, in async_function_with_fallbacks_common_utils
    raise original_exception
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5604, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5759, in async_function_with_retries
    self.should_retry_this_error(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        error=e,
        ^^^^^^^^
    ...<4 lines>...
        content_policy_fallbacks=content_policy_fallbacks,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5963, in should_retry_this_error
    raise error
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5710, in async_function_with_retries
    response = await self.make_call(original_function, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 5878, in make_call
    response = await response
               ^^^^^^^^^^^^^^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3927, in _ageneric_api_call_with_fallbacks_helper
    raise e
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3854, in _ageneric_api_call_with_fallbacks_helper
    raise e
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 3845, in _ageneric_api_call_with_fallbacks_helper
    deployment = await self.async_get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9461, in async_get_available_deployment
    raise e
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9346, in async_get_available_deployment
    healthy_deployments = await self.async_get_healthy_deployments(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9192, in async_get_healthy_deployments
    model, healthy_deployments = self._common_checks_available_deployment(
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<3 lines>...
        request_kwargs=request_kwargs,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )  # type: ignore
    ^
  File "/Users/cont-m-nakagawa/vscode/litellm/litellm/router.py", line 9161, in _common_checks_available_deployment
    raise litellm.BadRequestError(
    ...<3 lines>...
    )
litellm.exceptions.BadRequestError: litellm.BadRequestError: You passed in model=nonexistent-model. There are no healthy deployments for this model. Received Model Group=nonexistent-model
Available Model Group Fallbacks=None

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.83.5

Twitter / LinkedIn details

https://www.linkedin.com/in/mainakagawa/

extent analysis

TL;DR

The most likely fix is to modify the LiteLLM proxy to return a 400 Bad Request or 404 Not Found error when a non-existent model is specified, instead of the current 500 Internal Server Error.

Guidance

  • Review the async_get_available_deployment function in router.py to ensure it correctly handles non-existent models and raises a BadRequestError with a descriptive error message.
  • Verify that the google_generate_content endpoint in endpoints.py properly catches and handles the BadRequestError exception, returning a 400 Bad Request response to the client.
  • Check the should_retry_this_error function in router.py to ensure it does not retry the request when a BadRequestError occurs, as this could lead to indefinite retries.
  • Test the modified proxy with a non-existent model to confirm it returns the expected 400 Bad Request or 404 Not Found error.

Example

No code example is provided, as the necessary modifications depend on the specific implementation details of the LiteLLM proxy.

Notes

The provided log output suggests that the BadRequestError exception is being raised, but not properly handled, leading to the 500 Internal Server Error response. Modifying the proxy to correctly handle this exception and return a 400 Bad Request or 404 Not Found error should resolve the issue.

Recommendation

Apply a workaround by modifying the LiteLLM proxy to return a 400 Bad Request or 404 Not Found error when a non-existent model is specified, as this will prevent clients like Gemini CLI from retrying indefinitely.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#authentication issue#prompt issue#agent setup

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING