litellm - 💡(How to fix) Fix [Bug]: Latest stable 1.81.14 fails on thinking and tools [1 participants]

litellm.BadRequestError: Hosted_vllmException - {"object":"error","message":"19 validation errors:\n {'type': 'string_type', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'str'), 'msg': 'Input should be a valid string', 'input': [{'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}]}\n {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentTextPart', 'type'), 'msg': "Input should be 'text'", 'input': 'thinking', 'ctx': {'expected': "'text'"}}\n {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentTextPart', 'text'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}\n {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentImagePart', 'type'), 'msg': "Input should be 'image_url'", 'input': 'thinking', 'ctx': {'expected': "'image_url'"}}\n {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentImagePart', 'image_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}\n {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentVideoPart', 'type'), 'msg': "Input should be 'video_url'", 'input': 'thinking', 'ctx': {'expected': "'video_url'"}}\n {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentVideoPart', 'video_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}\n {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentAudioPart', 'type'), 'msg': "Input should be 'audio_url'", 'input': 'thinking', 'ctx': {'expected': "'audio_url'"}}\n {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentAudioPart', 'audio_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}\n {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'role'), 'msg': "Input should be 'user'", 'input': 'assistant', 'ctx': {'expected': "'user'"}}\n {'type': 'string_type', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'str'), 'msg': 'Input should be a valid string', 'input': [{'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}]}\n {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentTextPart', 'type'), 'msg': "Input should be 'text'", 'input': 'thinking', 'ctx': {'expected': "'text'"}}\n {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentTextPart', 'text'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}\n {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentImagePart', 'type'), 'msg': "Input should be 'image_url'", 'input': 'thinking', 'ctx': {'expected': "'image_url'"}}\n {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentImagePart', 'image_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}\n {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentVideoPart', 'type'), 'msg': "Input should be 'video_url'", 'input': 'thinking', 'ctx': {'expected': "'video_url'"}}\n {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentVideoPart', 'video_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}\n {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentAudioPart', 'type'), 'msg': "Input should be 'audio_url'", 'input': 'thinking', 'ctx': {'expected': "'audio_url'"}}\n {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentAudioPart', 'audio_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}\n\n File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/http_server.py", line 1351, in openai_v1_chat_completions\n POST /v1/chat/completions [{'type': 'string_type', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'str'), 'msg': 'Input should be a valid string', 'input': [{'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}]}, {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentTextPart', 'type'), 'msg': "Input should be 'text'", 'input': 'thinking', 'ctx': {'expected': "'text'"}}, {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentTextPart', 'text'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}, {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentImagePart', 'type'), 'msg': "Input should be 'image_url'", 'input': 'thinking', 'ctx': {'expected': "'image_url'"}}, {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentImagePart', 'image_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}, {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentVideoPart', 'type'), 'msg': "Input should be 'video_url'", 'input': 'thinking', 'ctx': {'expected': "'video_url'"}}, {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentVideoPart', 'video_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}, {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentAudioPart', 'type'), 'msg': "Input should be 'audio_url'", 'input': 'thinking', 'ctx': {'expected': "'audio_url'"}}, {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageGenericParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentAudioPart', 'audio_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}, {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'role'), 'msg': "Input should be 'user'", 'input': 'assistant', 'ctx': {'expected': "'user'"}}, {'type': 'string_type', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'str'), 'msg': 'Input should be a valid string', 'input': [{'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}]}, {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentTextPart', 'type'), 'msg': "Input should be 'text'", 'input': 'thinking', 'ctx': {'expected': "'text'"}}, {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentTextPart', 'text'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}, {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentImagePart', 'type'), 'msg': "Input should be 'image_url'", 'input': 'thinking', 'ctx': {'expected': "'image_url'"}}, {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentImagePart', 'image_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}, {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentVideoPart', 'type'), 'msg': "Input should be 'video_url'", 'input': 'thinking', 'ctx': {'expected': "'video_url'"}}, {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentVideoPart', 'video_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}, {'type': 'literal_error', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentAudioPart', 'type'), 'msg': "Input should be 'audio_url'", 'input': 'thinking', 'ctx': {'expected': "'audio_url'"}}, {'type': 'missing', 'loc': ('body', 'messages', 2, 'ChatCompletionMessageUserParam', 'content', 'list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart,ChatCompletionMessageContentVideoPart,ChatCompletionMessageContentAudioPart]]', 0, 'ChatCompletionMessageContentAudioPart', 'audio_url'), 'msg': 'Field required', 'input': {'type': 'thinking', 'thinking': "The user wants to know what's in the mcpserver.py file. Let me read it for them. "}}]","type":"Bad Request","param":null,"code":400}. Received Model Group=kimi-k2.5 Available Model Group Fallbacks=None

Fix Plan

The issue seems to be related to the thinking type in the ChatCompletionMessageGenericParam and ChatCompletionMessageUserParam. To fix this, we need to ensure that the content field is a valid string and the type field is either 'text', 'image_url', 'video_url', or 'audio_url'.

Here are the steps to fix the issue:

• Check the input data for the ChatCompletionMessageGenericParam and ChatCompletionMessageUserParam to ensure it conforms to the expected format.
• Modify the code to handle the thinking type correctly. This may involve adding a check for the thinking type and converting it to a valid string or one of the expected types.

Example code:

if message['type'] == 'thinking':
    message['type'] = 'text'
    message['content'] = {'text': message['thinking']}

Alternatively, you can also add a fallback to handle the thinking type:

if message['type'] == 'thinking':
    message['type'] = 'text'
    message['content'] = {'text': 'Unknown content type'}

Verification

To verify that the fix worked, you can test the API endpoint with a sample request that includes the thinking type. Check the response to ensure that it is successful and the content is correctly formatted.

Extra Tips

• Make sure to handle any potential errors that may occur during the conversion process.
• Consider adding logging to track any issues that may arise during the conversion process.
• If you are using a library or framework to handle the API requests, check the documentation to see if there are any built-in features to handle the thinking type.