litellm - 💡(How to fix) Fix LiteLLM spend meric issue (litellm_spend_metric_total)

I’m experiencing an issue where litellm_spend_metric_total resets to zero whenever LiteLLM restarts or experiences downtime. This causes incorrect spend calculations in Grafana when using increase() or rate() functions.

Current Query:

promql
sum by (model) (
  clamp_min(increase(litellm_spend_metric_total[$__range]), 0)
)

Problem:

• When LiteLLM restarts, the counter metric resets to 0
• This causes increase() to calculate negative values (new value - old value)
• I’m using clamp_min(..., 0) as a workaround to filter out negative spikes, but this doesn’t solve the underlying issue
• For teams/groups with a single model, the calculation works correctly
• For models that belong to multiple teams/groups, the spend is significantly under reported (e.g., showing 25 instead of 250)
• The spend data becomes inaccurate and doesn’t reflect actual cumulative costs

Expected Behavior:

Counter metrics like litellm_spend_metric_total should either:

Persist across restarts (loaded from database on startup), OR Be documented as non-persistent with recommended query patterns for handling resets

Environment:

LiteLLM version: 1.80.8
Deployment: Docker
Metrics backend: Prometheus