vllm - 💡(How to fix) Fix fix(metrics): Prometheus counter crash on negative prompt tokens with external KV transfer [1 participants]

Error: vllm/v1/metrics/loggers.py:1316 — ValueError: Negative prompt token count Clamp prompt token count to max(0, ...) in the metrics logger, or track externally-received prompt tokens separately via the KV connector metadata. The negative value is a bookkeeping artifact of disaggregated serving, not an actual error.

TL;DR

Clamp the prompt token count to a non-negative value in the metrics logger to prevent crashes.

Guidance

• Identify the line of code in vllm/v1/metrics/loggers.py:1316 where the ValueError occurs and modify it to clamp the prompt token count to max(0, ...) to prevent negative values.
• Consider tracking externally-received prompt tokens separately via the KV connector metadata to improve accuracy in metrics logging.
• Verify that the decode engine no longer crashes under sustained concurrent load with disaggregated inference after applying the fix.
• Test the modified metrics logger with various input scenarios to ensure it handles different token counts correctly.

Example

# Example of clamping prompt token count to non-negative value
prompt_tokens = max(0, total_tokens - completion_tokens)

Notes

This fix assumes that the negative prompt token count is a bookkeeping artifact and not an actual error. If the negative count indicates a genuine issue, further investigation may be necessary.

Recommendation

Apply the workaround by clamping the prompt token count to a non-negative value, as this is a production blocker for deployments using NixlConnector with Prometheus metrics enabled.