vllm - 💡(How to fix) Fix [Bug]: High concurrency or enabling eagle3 and streaming + function call can trigger accuracy issues on kimi k2.5 [1 comments, 1 participants]

TL;DR

The issue with the function call feature in version 0.18.0rc1 can likely be resolved by modifying the response handling to include the start or end identifier content.

Guidance

• Review the code responsible for generating the JSON response in the function call feature to ensure it includes the start and end identifier content.
• Verify that the eagle3 integration is correctly configured and that the issue is not related to the eagle3 setup.
• Check the documentation for any specific requirements or guidelines for handling function call responses in version 0.18.0rc1.
• Test the function call feature with a minimal example to isolate the issue and confirm the expected behavior.

Example

No specific code snippet can be provided without more context, but the fix likely involves modifying the response generation code to include the missing identifier content.

Notes

The issue seems to be specific to version 0.18.0rc1 and the eagle3 integration, so any fixes or workarounds may not apply to other versions or configurations.

Recommendation

Apply a workaround by modifying the response handling code to include the start and end identifier content, as this is likely a bug specific to version 0.18.0rc1.