vllm - 💡(How to fix) Fix [Bug]: Endless generation [1 participants]

Fix Plan

To address the endless generation issue, we need to adjust the model's configuration to prevent it from including user inputs in the response and to stop generating new tokens when a certain limit is reached.

Steps to Fix

• Set a maximum sequence length to prevent endless generation:
  • Use the --max-sequence-length parameter when loading the model.
• Exclude user inputs from the response:
  • Use the --exclude-inputs parameter when generating chat completions.

Example Code

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

# Load the model and tokenizer
model = AutoModelForSeq2SeqLM.from_pretrained("devstral/small-2505")
tokenizer = AutoTokenizer.from_pretrained("devstral/small-2505")

# Set the maximum sequence length
max_sequence_length = 512

# Generate chat completions with the specified parameters
def generate_chat_completions(user_input, model, tokenizer, max_sequence_length):
    inputs = tokenizer(user_input, return_tensors="pt")
    outputs = model.generate(**inputs, max_length=max_sequence_length)
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response

# Test the function
user_input = "Hello, how are you?"
response = generate_chat_completions(user_input, model, tokenizer, max_sequence_length)
print(response)

Verification

To verify that the fix worked, test the generate_chat_completions function with different user inputs and check that the response does not include the user input and does not exceed the specified maximum sequence length.

Extra Tips

• Make sure to adjust the max_sequence_length parameter according to your specific use case to prevent the model from generating excessively long responses.
• Consider implementing additional logic to handle cases where the user input is longer than the maximum sequence length.