litellm - 💡(How to fix) Fix [Feature]: add support for vLLM realtime endpoint [1 participants]

Fix Plan

To integrate the Voxtral Realtime model with LiteLLM, follow these steps:

• Setup a custom provider with the Voxtral Realtime model
• Configure the custom endpoint for the model
• Update the LiteLLM SDK to use the custom provider and endpoint

Code Changes

Example code snippet to setup the custom provider and endpoint:

import litellm

# Define the custom provider and endpoint
provider = litellm.Provider(
    name="Voxtral Realtime",
    model="mistralai/Voxtral-Mini-4B-Realtime-2602",
    endpoint="https://your-custom-endpoint.com/v1/inference"
)

# Update the LiteLLM SDK to use the custom provider and endpoint
litellm.setup_provider(provider)

Replace "https://your-custom-endpoint.com/v1/inference" with your actual custom endpoint URL.

Verification

To verify the fix, test the LiteLLM SDK with the custom provider and endpoint:

# Test the LiteLLM SDK with the custom provider and endpoint
response = litellm.query("Your test query", provider=provider)
print(response)

If the response is successful, the fix has worked.

Extra Tips

• Ensure the custom endpoint is correctly configured to handle the Voxtral Realtime model's requirements.
• Test the custom provider and endpoint thoroughly to avoid any regressions.