ollama - 💡(How to fix) Fix add the support for the token calculations and reduce the input tokens while answering [1 participants]

TL;DR

Implement a token tracking mechanism to monitor and store the usage of Ollama selected models, both locally and cloud-based, to optimize cost.

Guidance

• Identify the key components involved in tracking token usage, such as the Ollama models and the cache storage.
• Design a data structure to store the token usage information, including the model name, usage count, and timestamp.
• Develop a mechanism to update the token usage information in real-time, considering both local and cloud-based model usage.
• Consider implementing a caching strategy to store recently used models and their corresponding token usage.

Example

# Example of a simple token usage tracker
class TokenTracker:
    def __init__(self):
        self.usage_cache = {}

    def update_usage(self, model_name, token_count):
        if model_name in self.usage_cache:
            self.usage_cache[model_name] += token_count
        else:
            self.usage_cache[model_name] = token_count

    def get_usage(self, model_name):
        return self.usage_cache.get(model_name, 0)

Notes

The implementation details may vary depending on the specific requirements and the technology stack used. This example provides a basic idea of how to track token usage.

Recommendation

Apply workaround: Implement a custom token tracking mechanism, as described in the guidance section, to monitor and optimize token usage for Ollama selected models.