litellm - 💡(How to fix) Fix when working with mlx_lm with gemma4, it will invoke /v1/response which mlx_lm serve not supported [1 participants]

on litellm it shows this error

TL;DR

The issue can likely be resolved by selecting a different model using the /model command, as the currently selected model claude-gemma-8bit may not exist or is inaccessible.

Guidance

• Verify the availability of the model claude-gemma-8bit and check if there are any access restrictions.
• Run the /model command to pick a different model, as suggested in the error message.
• Check the documentation or configuration of litellm and mlx_lm to see if there are any specific requirements or settings for model selection.
• If the issue persists, try checking the API endpoints and parameters to ensure they are correct and properly formatted.

Notes

The exact cause of the issue is unclear, but it appears to be related to model selection or availability. The error messages suggest a NotFoundError, which implies that the model may not exist or is not accessible.

Recommendation

Apply workaround: Run the /model command to select a different model, as this is the most straightforward solution suggested by the error messages.