vllm - 💡(How to fix) Fix [Bug]: 0.17.0rc1在A2部署GLM-4.7，开启MTP后工具调用异常 [1 comments, 1 participants]

Fix Plan

To address the JSON parsing errors when MTP is enabled, we need to modify the tool parser to handle the parallel prediction output correctly.

Step-by-Step Solution:

1. Update the partial_json_parser: Modify the parser to account for the parallel token generation when MTP is enabled. This can be achieved by buffering the output and reassembling the JSON objects.
2. Implement a retry mechanism: For the probabilistic errors, implement a retry mechanism with a limited number of attempts to handle transient errors.
3. Model precision adjustment: Consider adjusting the model precision or exploring alternative models to reduce the occurrence of probabilistic errors.

Example Code Snippet (Python):

import json

class MTPJsonParser:
    def __init__(self):
        self.buffer = []

    def parse(self, output):
        self.buffer.append(output)
        try:
            # Attempt to parse the buffered output as JSON
            json_output = json.loads(''.join(self.buffer))
            self.buffer = []
            return json_output
        except json.JSONDecodeError:
            # If parsing fails, continue buffering
            return None

    def retry_parse(self, output, max_retries=3):
        for _ in range(max_retries):
            parsed_output = self.parse(output)
            if parsed_output is not None:
                return parsed_output
        # If all retries fail, raise an error
        raise ValueError("Failed to parse JSON output after retries")

Verification

To verify the fix, test the tool calls with MTP enabled and disabled, ensuring that the JSON output is correctly parsed in both cases. Monitor the error rates and user feedback to confirm that the probabilistic errors are significantly reduced.

Extra Tips

• Regularly review and update the model to ensure the best possible precision and reduce errors.
• Consider implementing additional logging and monitoring to quickly identify and address any recurring issues.