vllm - 💡(How to fix) Fix [Feature]: Kimi K2.5 Speculative Decoding [1 comments, 2 participants]

Fix Plan

To support Kimi K2.5 with speculative decoding, we need to integrate the Eagle3 decoder into our existing model architecture.

Step-by-Step Solution

1. Install required dependencies: Ensure you have the transformers library installed, and install the eagle3 decoder library if not already installed.
2. Import necessary modules: Import the KimiK25 model and the Eagle3Decoder class.
3. Initialize the model and decoder: Initialize the KimiK25 model and the Eagle3Decoder instance.
4. Integrate the decoder: Modify the KimiK25 model to use the Eagle3Decoder for speculative decoding.

Example Code

from transformers import KimiK25Model
from eagle3_decoder import Eagle3Decoder

# Initialize the model and decoder
model = KimiK25Model.from_pretrained('AQ-MedAI/Kimi-K25-eagle3')
decoder = Eagle3Decoder()

# Integrate the decoder
class KimiK25WithEagle3(KimiK25Model):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.decoder = decoder

    def generate(self, *args, **kwargs):
        # Use the Eagle3 decoder for speculative decoding
        return self.decoder.decode(self, *args, **kwargs)

# Use the modified model
model = KimiK25WithEagle3.from_pretrained('AQ-MedAI/Kimi-K25-eagle3')

Verification

To verify that the fix worked, test the model with a sample input and check that the output is correct and the decoding speed is improved.

Extra Tips

• Ensure that the Eagle3Decoder is properly configured and optimized for your specific use case.
• Monitor the model's performance and adjust the decoder's parameters as needed to achieve the best results.