ollama - 💡(How to fix) Fix Longer thinking time settings? [2 comments, 2 participants]

Fix Plan

To allow longer thinking times, we can introduce a customizable timeout feature.

Implementation Steps

• Add a new configuration option for timeout multiplier (e.g., timeout_multiplier)
• Update the API to accept this new option
• Modify the model invocation to use the customized timeout

Example Code

# config.py
TIMEOUT_MULTIPLIER_OPTIONS = {
    'low': 1,
    'high': 5,
    'extreme': 10,
    'ridiculous': 50
}

# api.py
from config import TIMEOUT_MULTIPLIER_OPTIONS

def generate_text(prompt, model, timeout_multiplier='high'):
    timeout = TIMEOUT_MULTIPLIER_OPTIONS[timeout_multiplier] * BASE_TIMEOUT
    # invoke model with customized timeout
    response = model.generate_text(prompt, timeout=timeout)
    return response

# usage
generate_text('example prompt', 'gpt-oss:20b', timeout_multiplier='ridiculous')

Verification

Test the new feature by invoking the API with different timeout_multiplier values and verify that the model responds with improved results.

Extra Tips

• Consider adding input validation for timeout_multiplier to prevent excessive timeouts.
• Monitor the performance impact of longer timeouts on your system and adjust the configuration accordingly.