litellm - 💡(How to fix) Fix Got a liteLLM error while using Groq API Key [1 participants]

This error occurred because Hive uses a lot of tokens for creating a new Agent. The model used is moonshotai/kimi-k2-instruct-0905. The model fails even at creating simple Agents like an email checker. The rate limit is 1000 tokens, but the agent uses 10863 tokens for a 6-word message (Check my inbox for urgent emails).

Fix Plan

To resolve the token limit issue, we'll implement token-efficient agent creation.

Steps:

• Optimize Agent Creation: Reduce the number of tokens used by the agent.
• Token Budgeting: Implement a token budget to limit the number of tokens used.
• Model Configuration: Adjust the model configuration to use fewer tokens.

Example Code (Python):

import torch

# Define a function to create an agent with token budgeting
def create_agent(model, prompt, token_budget):
    # Initialize the agent with the given model and prompt
    agent = model(prompt)
    
    # Check if the agent exceeds the token budget
    if agent.num_tokens > token_budget:
        # If exceeded, truncate the agent to fit within the budget
        agent.truncate(token_budget)
    
    return agent

# Define the model and prompt
model = "moonshotai/kimi-k2-instruct-0905"
prompt = "Check my inbox for urgent emails"

# Define the token budget
token_budget = 1000

# Create the agent with token budgeting
agent = create_agent(model, prompt, token_budget)

# Use the created agent
print(agent)

Verification

Verify that the agent creation succeeds and the number of tokens used is within the limit. Check the agent.num_tokens value to ensure it's less than or equal to the token_budget.

Extra Tips

• Monitor token usage and adjust the token_budget as needed to avoid exceeding the rate limit.
• Consider using more efficient models or optimizing the prompt to reduce token usage.