Fix Plan

To handle cached tokens in the token cost calculator, we need to update the calculate_token_cost function in utils.py.

• Update the calculate_token_cost function to parse the cached_tokens from the VLLM response.
• Add a new variable to store the cached tokens and calculate the total token cost accordingly.

Example code:

def calculate_token_cost(vllm_response):
    # ...
    cached_tokens = vllm_response['usage']['prompt_tokens_details']['cached_tokens']
    total_tokens = vllm_response['usage']['total_tokens']
    cache_read_tokens = cached_tokens
    cache_creation_tokens = total_tokens - cache_read_tokens
    # ...
    return {
        # ...
        'cache_read_tokens': cache_read_tokens,
        'cache_creation_tokens': cache_creation_tokens,
    }

• Update the logging to display the correct cache read and creation tokens.

Verification

To verify the fix, send a request with the VLLM provider and check the LiteLLM logs for the correct cache read and creation tokens.

Extra Tips

Make sure to test the updated function with different VLLM responses to ensure it handles all possible cases correctly. Additionally, consider adding error handling for cases where the cached_tokens field is missing or null.