litellm - 💡(How to fix) Fix [Bug]: Phantom Redis spend counter inflation on v1.83.14-stable (3-6× real spend), independent of #27639's budget reservation feature [1 participants]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

user_api_key_auth raises Budget has been exceeded! Current cost: X, Max budget: Y where X (the value read from the Redis-backed cross-pod spend counter) is 3–6× the real key spend stored in Postgres. Keys that are nowhere near their max_budget get rejected with HTTP 400 in bursts; retries from a quieter window succeed.

This looks adjacent to #27639, but it is not that bug:

• #27639 is about reserve_budget_for_request() in litellm/proxy/spend_tracking/budget_reservation.py (added by #26845, merged 2026-05-01, first shipped in v1.84.0).
• v1.83.14-stable was tagged 2026-04-26 and does not contain budget_reservation.py at all. _estimate_output_tokens, _estimate_image_generation_cost, and the entire reserve → finalize pipeline are absent.
• A diff of v1.83.10-stable vs v1.83.14-stable shows zero changes to auth_checks.py, proxy_server.py, user_api_key_auth.py, or proxy_track_cost_callback.py — so the symptom is also present unchanged in v1.83.10-stable (the version recommended by the CVE-2026-42208 security advisory).
• The most likely source is the cross-pod Redis spend counter introduced by #24682 (merged 2026-03-27, first shipped in stable in v1.83.3). #27509's fix targets reserve_budget_for_request and does not apply to v1.83.x.

Symptom evidence

Two affected keys (aliases only, values redacted) on the same v1.83.14-stable deployment:

For both keys the parent team has max_budget: null — so enforcement is strictly at the key level. The phantom value is what get_current_spend() returns from the Redis counter, not real ledger spend.

A 7-day Datadog aggregation across all affected keys on the deployment shows a striking pattern: the reported current_cost for a given key is a clean integer multiple of an apparent base value, repeated identically across thousands of log lines.

The "1×, 2×, 3×, 4× base" pattern strongly suggests an in-flight reservation/increment that is not being decremented on the failure path, then compounding across concurrent requests or replicas.

Onset

• Bug onset: ~2026-05-13, immediately after bumping to v1.83.14-stable (for the CVE-2026-42208 / GHSA-r75f-5x8p-qvmc fix).
• Previously running v1.81.0-stable (which predates #24682's cross-pod counter, and predates the CVE fix). No prior reports of this symptom on v1.81.0.
• Reproducible on both teams whenever concurrency rises; quieter periods recover (counter drifts back down somewhat but stays inflated).

Steps to Reproduce

1. Deploy LiteLLM v1.83.14-stable on a multi-replica Fargate cluster (2+ ECS tasks behind an ALB) backed by a shared Redis cache. Config: litellm_settings.cache_params.type=redis, router_settings.redis_host/port/password set, no use_redis_transaction_buffer, no max_end_user_budget_id.
2. Generate a virtual key with key-level max_budget=3000 and team-level max_budget=null. Send concurrent chat-completion traffic from a service that does NOT explicitly pass max_tokens.
3. Observe real key.spend (via GET /key/info) tracking ~$1,000 while user_api_key_auth begins raising BudgetExceededError with current_cost=$6,000+ in bursts, even though the key is at ~35% of budget.
4. Retries from a quieter window succeed. The Redis counter does not monotonically reconcile back to real spend.

Relevant log output

litellm.proxy.proxy_server.user_api_key_auth(): Exception occured - Budget has been exceeded! Current cost: 6145.880053130025, Max budget: 3000.0
Requester IP Address: <internal>

(The identical current_cost value — to 13 decimal places — appears ~307 times in a 7-day window for one key, and 8,915 times for another key at current_cost: 10151.04896475005, indicating a stable Redis-side value being read repeatedly rather than naturally accumulating costs.)

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.83.14-stable (also v1.83.10-stable based on diff; cross-pod counter present since v1.83.3-stable)

Suggested investigation areas

1. litellm/proxy/proxy_server.py — increment_spend_counters, get_current_spend, _init_and_increment_spend_counter (all added by #24682).
2. Failure-path handling: when an LLM call errors after admission, is the counter ever decremented? In v1.83.14 there is no budget_reservation.py to clean up after failed requests.
3. Whether increment_spend_counters is being invoked more than once per request under certain code paths (retries, fallbacks, streaming chunks).
4. Whether the Redis counter ever reseeds from authoritative DB spend on v1.83.x (the reseed file litellm/proxy/db/spend_counter_reseed.py is present in v1.83.14 but may not be wired into the read path the same way it is in v1.84.0).

Twitter / LinkedIn details

No response