litellm - 💡(How to fix) Fix [Bug]: Under Anthropic style calls (with function-call), hosted-vllm backend responses tool-call with "invalid JSON" [1 participants]

Fix Plan

The fix involves modifying the config.yaml file to correctly configure the API connection.

Steps to Fix

• Update the api_base URL in config.yaml to include the correct protocol and port.
• Ensure the api_key is correctly set if authentication is required.

Example Code

model_list:
  - model_name: "*"
    litellm_params:
      model: hosted_vllm/GLM-4.7-Flash
      api_base: http://127.0.0.1:1024/v1
      api_key: "your_api_key_here"  # Replace with your actual API key
      drop_params: true

Verification

Start the service with the updated config.yaml file using the command litellm --config config.yaml --port 1025 and verify that the API connection is successful by checking the log output for any error messages related to the API connection.

Extra Tips

• Make sure to replace "your_api_key_here" with your actual API key if authentication is required.
• If you're still experiencing issues, try checking the API documentation for any specific requirements or restrictions on API connections.