litellm - ✅(Solved) Fix Bug: _set_virtual_key_rate_limit_metrics crashes with ValueError: Incorrect label count when include_labels is configured [1 pull requests, 1 participants]

moshekan · 2026-03-29T12:28:58Z

[litellm] PR 25201: fix prometheus : use prometheus label factory in set virtual key rate limit metrics - Repository: BerriAI/litellm - Author: tom-onyx - Stat… # PR #25201: fix(prometheus): use prometheus_label_factory in _set_virtual_key_rate_limit_metrics - Repository: BerriAI/litellm - Author: tom-onyx - State: open | merged: False - Link: https://github.com/BerriAI/litellm/pull/25201 ## Description (problem / solution / changelog) ## Summary Fixes #24760 `_set_virtual_key_rate_limit_metrics` uses hardcoded positional args to `.labels()`, which crashes with `ValueError: Incorrect label count` when `custom_prometheus_metadata_labels` is configured. The Gauge is created with labels from `get_labels_for_metric()` which appends custom metadata labels, but the hardcoded call never provides values for them. This switches to `prometheus_label_factory` + `enum_values`, consistent with every other metric in the callback. The function signature is updated to accept `enum_values` from the caller instead of individual parameters. **Impact**: On our production deployment (v1.81.9-stable), this crashes on **100% of requests** (~10K/hr per pod). The crash happens after spend/token metrics are recorded but **before** latency, deployment success, cache, and streaming request count metrics — silently dropping all of these downstream metrics. ## Changes - `litellm/integrations/prometheus.py`: Updated `_set_virtual_key_rate_limit_metrics` to use `prometheus_label_factory` for both `litellm_remaining_api_key_requests_for_model` and `litellm_remaining_api_key_tokens_for_model` gauges - `tests/test_litellm/integrations/test_prometheus_virtual_key_rate_limit.py`: Added 4 unit tests covering default labels, custom metadata labels, None values, and remaining value propagation ## Test plan - [x] Added `test_prometheus_virtual_key_rate_limit.py` with 4 tests - [x] `test_set_virtual_key_rate_limit_metrics_basic` — default label configuration - [x] `test_set_virtual_key_rate_limit_metrics_with_custom_labels` — reproduces the exact crash from #24760 (custom_prometheus_metadata_labels configured) - [x] `test_set_virtual_key_rate_limit_metrics_with_none_values` — graceful handling of None metadata - [x] `test_set_virtual_key_rate_limit_metrics_sets_remaining_values` — verifies remaining request/token counts are propagated from metadata - [x] All existing prometheus tests pass (4 pre-existing failures unrelated to this change) ## Pre-submission checklist - [x] At least 1 test added to `tests/test_litellm/` - [x] Scope is isolated to fixing the label count mismatch 🤖 Generated with [Claude Code](https://claude.com/claude-code) @greptileai review ## Changed files - `litellm/integrations/prometheus.py` (modified, +20/-18) - `tests/test_litellm/integrations/test_prometheus_virtual_key_rate_limit.py` (added, +183/-0) ## Bug Description `_set_virtual_key_rate_limit_metrics` in `litellm/integrations/prometheus.py` crashes with `ValueError: Incorrect label count` on every invocation when `prometheus_metrics_config` with `include_labels` is configured. The method hardcodes 4 positional arguments to `.labels()`, but the Gauge `litellm_remaining_api_key_requests_for_model` may have fewer labels after `get_labels_for_metric()` filters them via `include_labels`. ## Stack Trace ``` File "litellm/litellm_core_utils/litellm_logging.py", line 2590, in async_success_handler await callback.async_log_success_event(...) File "litellm/integrations/prometheus.py", line 1022, in async_log_success_event self._set_virtual_key_rate_limit_metrics(...) File "litellm/integrations/prometheus.py", line 1278, in _set_virtual_key_rate_limit_metrics self.litellm_remaining_api_key_requests_for_model.labels( user_api_key, user_api_key_alias, model_group, model_id ).set(remaining_requests) File "prometheus_client/metrics.py", line 199, in labels raise ValueError('Incorrect label count') ValueError: Incorrect label count ``` ## Root Cause `_set_virtual_key_rate_limit_metrics` passes 4 hardcoded positional args to `.labels()`: ```python self.litellm_remaining_api_key_requests_for_model.labels( user_api_key, user_api_key_alias, model_group, model_id ).set(remaining_requests) ``` But the Gauge is created with `get_labels_for_metric()` which may return fewer labels when `include_labels` is configured: ```python self.litellm_remaining_api_key_requests_for_model = self._gauge_factory( "litellm_remaining_api_key_requests_for_model", ..., labelnames=self.get_labels_for_metric("litellm_remaining_api_key_requests_for_model"), ) ``` The same issue affects `litellm_remaining_api_key_tokens_for_model` in the same method. ## Impact - The crash aborts `async_log_success_event` after spend/token metrics are recorded but **before** latency, deployment success, cache, and streaming request count metrics - **All of these downstream metrics are silently dropped** for every request where this crashes - On our production deployment (v1.81.9-stable), this fires **2.13M times over 7 days** (~50-60

litellm2026-03-29 12:28:58

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

BerriAI/litellm#24760•Fetched 2026-04-08 01:49:22

View on GitHub

Comments

Participants

Timeline

Reactions

Author

moshekan

Participants

moshekan

Timeline (top)

cross-referenced ×1referenced ×1

Error Message

File "litellm/litellm_core_utils/litellm_logging.py", line 2590, in async_success_handler
    await callback.async_log_success_event(...)
File "litellm/integrations/prometheus.py", line 1022, in async_log_success_event
    self._set_virtual_key_rate_limit_metrics(...)
File "litellm/integrations/prometheus.py", line 1278, in _set_virtual_key_rate_limit_metrics
    self.litellm_remaining_api_key_requests_for_model.labels(
        user_api_key, user_api_key_alias, model_group, model_id
    ).set(remaining_requests)
File "prometheus_client/metrics.py", line 199, in labels
    raise ValueError('Incorrect label count')
ValueError: Incorrect label count

Root Cause

_set_virtual_key_rate_limit_metrics passes 4 hardcoded positional args to .labels():

self.litellm_remaining_api_key_requests_for_model.labels(
    user_api_key, user_api_key_alias, model_group, model_id
).set(remaining_requests)

But the Gauge is created with get_labels_for_metric() which may return fewer labels when include_labels is configured:

self.litellm_remaining_api_key_requests_for_model = self._gauge_factory(
    "litellm_remaining_api_key_requests_for_model",
    ...,
    labelnames=self.get_labels_for_metric("litellm_remaining_api_key_requests_for_model"),
)

The same issue affects litellm_remaining_api_key_tokens_for_model in the same method.

PR fix notes

PR #25201: fix(prometheus): use prometheus_label_factory in _set_virtual_key_rate_limit_metrics

Repository: BerriAI/litellm
Author: tom-onyx
State: open | merged: False
Link: https://github.com/BerriAI/litellm/pull/25201

Description (problem / solution / changelog)

Summary

Fixes #24760

_set_virtual_key_rate_limit_metrics uses hardcoded positional args to .labels(), which crashes with ValueError: Incorrect label count when custom_prometheus_metadata_labels is configured. The Gauge is created with labels from get_labels_for_metric() which appends custom metadata labels, but the hardcoded call never provides values for them.

This switches to prometheus_label_factory + enum_values, consistent with every other metric in the callback. The function signature is updated to accept enum_values from the caller instead of individual parameters.

Impact: On our production deployment (v1.81.9-stable), this crashes on 100% of requests (~10K/hr per pod). The crash happens after spend/token metrics are recorded but before latency, deployment success, cache, and streaming request count metrics — silently dropping all of these downstream metrics.

Changes

litellm/integrations/prometheus.py: Updated _set_virtual_key_rate_limit_metrics to use prometheus_label_factory for both litellm_remaining_api_key_requests_for_model and litellm_remaining_api_key_tokens_for_model gauges
tests/test_litellm/integrations/test_prometheus_virtual_key_rate_limit.py: Added 4 unit tests covering default labels, custom metadata labels, None values, and remaining value propagation

Test plan

Added test_prometheus_virtual_key_rate_limit.py with 4 tests
test_set_virtual_key_rate_limit_metrics_basic — default label configuration
test_set_virtual_key_rate_limit_metrics_with_custom_labels — reproduces the exact crash from #24760 (custom_prometheus_metadata_labels configured)
test_set_virtual_key_rate_limit_metrics_with_none_values — graceful handling of None metadata
test_set_virtual_key_rate_limit_metrics_sets_remaining_values — verifies remaining request/token counts are propagated from metadata
All existing prometheus tests pass (4 pre-existing failures unrelated to this change)

Pre-submission checklist

At least 1 test added to tests/test_litellm/
Scope is isolated to fixing the label count mismatch

🤖 Generated with Claude Code

@greptileai review

Changed files

litellm/integrations/prometheus.py (modified, +20/-18)
tests/test_litellm/integrations/test_prometheus_virtual_key_rate_limit.py (added, +183/-0)

Code Example

File "litellm/litellm_core_utils/litellm_logging.py", line 2590, in async_success_handler
    await callback.async_log_success_event(...)
File "litellm/integrations/prometheus.py", line 1022, in async_log_success_event
    self._set_virtual_key_rate_limit_metrics(...)
File "litellm/integrations/prometheus.py", line 1278, in _set_virtual_key_rate_limit_metrics
    self.litellm_remaining_api_key_requests_for_model.labels(
        user_api_key, user_api_key_alias, model_group, model_id
    ).set(remaining_requests)
File "prometheus_client/metrics.py", line 199, in labels
    raise ValueError('Incorrect label count')
ValueError: Incorrect label count

---

self.litellm_remaining_api_key_requests_for_model.labels(
    user_api_key, user_api_key_alias, model_group, model_id
).set(remaining_requests)

---

self.litellm_remaining_api_key_requests_for_model = self._gauge_factory(
    "litellm_remaining_api_key_requests_for_model",
    ...,
    labelnames=self.get_labels_for_metric("litellm_remaining_api_key_requests_for_model"),
)

---

_labels = prometheus_label_factory(
    supported_enum_labels=self.get_labels_for_metric("litellm_remaining_api_key_requests_for_model"),
    enum_values=enum_values,
)
self.litellm_remaining_api_key_requests_for_model.labels(**_labels).set(remaining_requests)

RAW_BUFFERClick to expand / collapse

Bug Description

_set_virtual_key_rate_limit_metrics in litellm/integrations/prometheus.py crashes with ValueError: Incorrect label count on every invocation when prometheus_metrics_config with include_labels is configured.

The method hardcodes 4 positional arguments to .labels(), but the Gauge litellm_remaining_api_key_requests_for_model may have fewer labels after get_labels_for_metric() filters them via include_labels.

Stack Trace

File "litellm/litellm_core_utils/litellm_logging.py", line 2590, in async_success_handler
    await callback.async_log_success_event(...)
File "litellm/integrations/prometheus.py", line 1022, in async_log_success_event
    self._set_virtual_key_rate_limit_metrics(...)
File "litellm/integrations/prometheus.py", line 1278, in _set_virtual_key_rate_limit_metrics
    self.litellm_remaining_api_key_requests_for_model.labels(
        user_api_key, user_api_key_alias, model_group, model_id
    ).set(remaining_requests)
File "prometheus_client/metrics.py", line 199, in labels
    raise ValueError('Incorrect label count')
ValueError: Incorrect label count

Root Cause

_set_virtual_key_rate_limit_metrics passes 4 hardcoded positional args to .labels():

self.litellm_remaining_api_key_requests_for_model.labels(
    user_api_key, user_api_key_alias, model_group, model_id
).set(remaining_requests)

But the Gauge is created with get_labels_for_metric() which may return fewer labels when include_labels is configured:

self.litellm_remaining_api_key_requests_for_model = self._gauge_factory(
    "litellm_remaining_api_key_requests_for_model",
    ...,
    labelnames=self.get_labels_for_metric("litellm_remaining_api_key_requests_for_model"),
)

The same issue affects litellm_remaining_api_key_tokens_for_model in the same method.

Impact

The crash aborts async_log_success_event after spend/token metrics are recorded but before latency, deployment success, cache, and streaming request count metrics
All of these downstream metrics are silently dropped for every request where this crashes
On our production deployment (v1.81.9-stable), this fires 2.13M times over 7 days (~50-60% of all requests)
litellm_remaining_api_key_requests_for_model never successfully emits any data
Latency metrics (_set_latency_metrics), deployment success metrics, and cache metrics are all broken

Expected Fix

Use prometheus_label_factory() (same pattern as litellm_proxy_total_requests_metric at the end of async_log_success_event) instead of hardcoded positional arguments:

_labels = prometheus_label_factory(
    supported_enum_labels=self.get_labels_for_metric("litellm_remaining_api_key_requests_for_model"),
    enum_values=enum_values,
)
self.litellm_remaining_api_key_requests_for_model.labels(**_labels).set(remaining_requests)

Related Issues

#7611 — Same error class in prometheus_services.py (fixed)
#16773 — Same error in increment_deployment_cooled_down (fixed via #17847)
#22580 — Same error in _set_latency_metrics (still open)

Environment

LiteLLM version: v1.81.9-stable
Image: ghcr.io/berriai/litellm-database:main-v1.81.9-stable
Python: 3.13
Helm chart: litellm-helm-0.1.837

extent analysis

Fix Plan

To fix the issue, we need to replace the hardcoded positional arguments with a dynamic approach using prometheus_label_factory(). Here are the steps:

Replace the hardcoded .labels() call with a dynamic one:

_labels = prometheus_label_factory(
    supported_enum_labels=self.get_labels_for_metric("litellm_remaining_api_key_requests_for_model"),
    enum_values={
        "user_api_key": user_api_key,
        "user_api_key_alias": user_api_key_alias,
        "model_group": model_group,
        "model_id": model_id,
    },
)
self.litellm_remaining_api_key_requests_for_model.labels(**_labels).set(remaining_requests)

Apply the same fix to litellm_remaining_api_key_tokens_for_model:

_labels = prometheus_label_factory(
    supported_enum_labels=self.get_labels_for_metric("litellm_remaining_api_key_tokens_for_model"),
    enum_values={
        "user_api_key": user_api_key,
        "user_api_key_alias": user_api_key_alias,
        "model_group": model_group,
        "model_id": model_id,
    },
)
self.litellm_remaining_api_key_tokens_for_model.labels(**_labels).set(remaining_tokens)

Verification

To verify that the fix worked, check the following:

The ValueError: Incorrect label count exception should no longer occur.
The litellm_remaining_api_key_requests_for_model and litellm_remaining_api_key_tokens_for_model metrics should be successfully emitted.
The downstream metrics (latency, deployment success, cache, and streaming request count) should no longer be silently dropped.

Extra Tips

Make sure to test the fix thoroughly to ensure that it works as expected.
Consider adding additional logging or monitoring to detect similar issues in the future.
Review the related issues (#7611, #16773, #22580) to ensure that similar fixes are applied consistently throughout the codebase.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #output truncation #response parsing #generation error #database connection

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

litellm - ✅(Solved) Fix Bug: _set_virtual_key_rate_limit_metrics crashes with ValueError: Incorrect label count when include_labels is configured [1 pull requests, 1 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

PR fix notes

PR #25201: fix(prometheus): use prometheus_label_factory in _set_virtual_key_rate_limit_metrics

Description (problem / solution / changelog)

Summary

Changes

Test plan

Pre-submission checklist

Changed files

Code Example

Bug Description

Stack Trace

Root Cause

Impact

Expected Fix

Related Issues

Environment

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

TRENDING

litellm - ✅(Solved) Fix Bug: _set_virtual_key_rate_limit_metrics crashes with ValueError: Incorrect label count when include_labels is configured [1 pull requests, 1 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

PR fix notes

PR #25201: fix(prometheus): use prometheus_label_factory in _set_virtual_key_rate_limit_metrics

Description (problem / solution / changelog)

Summary

Changes

Test plan

Pre-submission checklist

Changed files

Code Example

Bug Description

Stack Trace

Root Cause

Impact

Expected Fix

Related Issues

Environment

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING