ollama - 💡(How to fix) Fix Cloud and local models produce no meaningful output on large-context generation tasks via OpenAI-compatible API [2 comments, 3 participants]

Fix Plan

To address the issue of models failing to produce meaningful output on large-context generation tasks, we will focus on adjusting the API request configuration and potentially updating the Ollama version.

Step-by-Step Solution:

1. Increase the response length: Adjust the max_tokens parameter in the API request to a higher value to allow for more substantial responses.
2. Specify the stop sequence: Define a stop sequence to prevent the model from regurgitating the input source code indefinitely.
3. Update Ollama version: Consider updating Ollama to the latest version, as the issue might be resolved in newer releases.

Example Code Snippet (Python):

import requests

# Set API endpoint and parameters
endpoint = "https://api.ollama.ai/v1/completions"
params = {
    "model": "glm-5:cloud",
    "prompt": "Review this Python file for bugs, logic issues, and improvements",
    "max_tokens": 2048,  # Increased response length
    "stop": "\n\n---\n\n"  # Stop sequence to prevent regurgitation
}

# Set the input file as context
with open("input_file.py", "r") as file:
    context = file.read()
params["context"] = context

# Send the request
response = requests.post(endpoint, json=params)

# Print the response
print(response.json()["completion"])

Verification

To verify that the fix worked, check the response from the API for a more substantial analysis of the input file, rather than just regurgitated source code.

Extra Tips

• Monitor the API response size and adjust the max_tokens parameter accordingly to avoid hitting response limits.
• Experiment with different stop sequences to find the most effective one for your specific use case.
• Keep an eye on Ollama version updates and consider updating to the latest version for potential bug fixes and improvements.