ollama - 💡(How to fix) Fix Feature Request: Include usage metrics in streaming chat responses [1 comments, 2 participants]

Problem Description

Currently, the OpenAPI spec defines ChatStreamEvent without the usage/metrics fields that are available in the non-streaming ChatResponse. This creates an inconsistency where developers cannot access important performance metrics (token counts, timing information) during streaming responses.

Proposed Solution

Add the following fields to ChatStreamEvent in the OpenAPI spec to match ChatResponse:

• total_duration - Total request duration in nanoseconds
• load_duration - Model loading duration in nanoseconds
• prompt_eval_count - Number of tokens in the prompt
• prompt_eval_duration - Time spent evaluating the prompt in nanoseconds
• eval_count - Number of tokens in the response
• eval_duration - Time spent generating response in nanoseconds

Additionally, add logprobs to ChatStreamEvent since it's present in GenerateStreamEvent but missing from the chat streaming spec.

Use Case

Developers need real-time token usage and timing data during streaming to:

• Display token counts to users in real-time
• Monitor model performance and latency
• Implement rate limiting based on actual usage
• Debug performance issues

Implementation Suggestion

The Go implementation already uses a single ChatResponse struct for both streaming and non-streaming responses (see api/types.go). The fix primarily requires updating the OpenAPI spec in docs/api.md to include the metrics fields in ChatStreamEvent.

Reference: A similar pattern is already implemented for GenerateStreamEvent which correctly includes these metrics.

Additional Context

This would also resolve the inconsistency noted in issue #14680 regarding the mismatch between streaming and non-streaming response schemas.