Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Bug Description

When using LiteLLM proxy with gemini_api: true (to serve Gemini-compatible endpoints for clients like gemini-cli) forwarding to a vLLM backend, the usage / usageMetadata field is missing from the response received by the client.

Root Cause

vLLM delivers usage in a separate, trailing SSE chunk that has an empty choices array, sent after the chunk that carries finish_reason: stop: data: {"choices":[{"finish_reason":"stop","delta":{"content":""},...}], ...} ← chunk N-1 data: {"choices":[], "usage":{"prompt_tokens":26,"completion_tokens":242,"total_tokens":268}} ← chunk N data: [DONE] LiteLLM's stream consumer stops processing after it sees finish_reason in chunk N-1, so chunk N (which contains the usage) is never read. As a result:

The usage is not captured internally by LiteLLM. The Gemini translation layer has nothing to convert into usageMetadata, so the field is absent from the response forwarded to the client.

Environment

LiteLLM version: latest (reproduced on recent releases) Backend: vLLM (OpenAI-compatible endpoint) Client: gemini-cli (expects Gemini-format usageMetadata) Config: gemini_api: true, stream_options.include_usage: true

Expected Behavior

LiteLLM should continue reading the stream until data: [DONE] rather than stopping at the first finish_reason chunk. The trailing usage chunk should be consumed, merged, and correctly translated into usageMetadata when the Gemini compatibility layer is active.

Actual Behavior

LiteLLM stops consuming the stream upon receiving finish_reason: stop. The subsequent usage-only chunk is discarded, and no usage information reaches the client.

Workaround

A lightweight reverse proxy can be placed between LiteLLM and vLLM to merge the two trailing chunks before LiteLLM sees them: gemini-cli → LiteLLM (:4000) → merge-proxy (:8082) → vLLM (:8081) The merge proxy buffers the finish_reason chunk and, if the next chunk is a usage-only chunk (choices: []), injects the usage field into the buffered chunk before forwarding. This is a workaround, not a fix — ideally LiteLLM handles this natively. Alternatively, setting fake_stream: true on the model avoids the issue entirely by using a non-streaming request to vLLM, at the cost of increased time-to-first-token.

Suggested Fix

In the SSE stream consumer, do not close the stream on finish_reason. Instead, keep reading until data: [DONE] is received, and collect any usage data that arrives in subsequent chunks.

Steps to Reproduce

Start a vLLM server and confirm it returns a trailing usage chunk (curl example below). Configure LiteLLM proxy with gemini_api: true pointing to the vLLM backend. Send a streaming request via gemini-cli (or any Gemini-compatible client). Observe that the response contains no usageMetadata.

Verify vLLM trailing-chunk behavior directly: curl http://localhost:8081/v1beta/models/GLM-4.7:streamGenerateContent \ -H "Content-Type: application/json" \ -d '{ "contents": [ { "parts": [{"text": "你好，帮我写一个快速排序，保存在main.py中"}] } ] }' The last two data chunks before [DONE] will look exactly like the example above.

Relevant log output

What part of LiteLLM is this about?

No response

What LiteLLM version are you on ?

v1.83.3

Twitter / LinkedIn details

No response