Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Bug Description

Time to First Token (TTFT) is never correctly captured for requests routed through the /v1/messages endpoint when using providers that have a native BaseAnthropicMessagesConfig implementation (Anthropic, Bedrock, Vertex AI Claude, Azure AI Claude, Minimax).

The result is:

1. Usage logs UI: TTFT column shows - (null) or displays a value equal to the total request duration — neither is the real first-token time.

   <img width="1484" height="418" alt="Image" src="https://github.com/user-attachments/assets/261e67c6-92e7-4451-8993-44d8ecc0f305" />

2. litellm_llm_api_time_to_first_token_metric (Prometheus): Emitted for streaming requests, but the value equals the full API call duration, not the actual TTFT.

   <img width="2860" height="1034" alt="Image" src="https://github.com/user-attachments/assets/ef06a2fe-ee12-49f7-94ae-22f4ee0f48ef" />

Expected Behaviour

completion_start_time should be set to the timestamp when the first non-empty chunk is received from the upstream provider, consistent with how the standard /chat/completions streaming path works.

Actual Behaviour

completion_start_time is set to end_time (after the last chunk), making reported TTFT equal to total duration.

Affected Providers

Any provider that returns a non-None value from ProviderConfigManager.get_provider_anthropic_messages_config() uses the pass-through path and is affected:

Providers not affected: any provider that returns None from get_provider_anthropic_messages_config (e.g. Mistral, Cohere, Groq) falls through to LiteLLMMessagesToCompletionTransformationHandler, which uses the standard CustomStreamWrapper and correctly captures completion_start_time on the first streaming chunk.

Suspected Root Cause

The /v1/messages endpoint for the affected providers routes through a pass-through streaming pipeline (BaseAnthropicMessagesStreamingIterator.async_sse_wrapper) that collects all chunks and fires a single logging call after the stream is complete:

# litellm/llms/anthropic/experimental_pass_through/messages/streaming_iterator.py
async def async_sse_wrapper(self, completion_stream):
    collected_chunks = []
    async for chunk in completion_stream:
        collected_chunks.append(self._convert_chunk_to_sse_format(chunk))
        yield chunk                                    # chunks go to client here

    await self._handle_streaming_logging(collected_chunks)  # logging fires after last chunk

async def _handle_streaming_logging(self, collected_chunks):
    end_time = datetime.now()                          # captured AFTER all chunks sent
    asyncio.create_task(
        PassThroughStreamingHandler._route_streaming_logging_to_handler(
            ...
            end_time=end_time,
        )
    )

Because the logging call only receives end_time (after streaming completes), completion_start_time is never set during the stream. Inside async_success_handler, the fallback triggers:

# litellm/litellm_core_utils/litellm_logging.py
if self.completion_start_time is None:
    self.completion_start_time = end_time   # falls back to end_time, not first-chunk time
    self.model_call_details["completion_start_time"] = self.completion_start_time

completion_start_time ends up equal to end_time, so:

• In the DB: completionStartTime ≈ endTime → TTFT ≈ Duration or "-" (UI filters it)
• In Prometheus: litellm_llm_api_time_to_first_token_metric = completion_start_time − api_call_start_time = full API call duration

Contrast with the standard /chat/completions streaming path, where _update_completion_start_time() is called on the first chunk:

# litellm/litellm_core_utils/streaming_handler.py
if self.logging_obj.completion_start_time is None:
    self.logging_obj._update_completion_start_time(
        completion_start_time=datetime.datetime.now()
    )

Steps to Reproduce

1. Configure LiteLLM proxy with a Bedrock Claude model (or any other affected provider above).
2. Send a streaming request to /v1/messages.
3. Observe the usage logs UI: the TTFT column is either - or shows a value equal to the Duration column.
4. Observe Prometheus: litellm_llm_api_time_to_first_token_metric value matches litellm_llm_api_latency_metric rather than being a fraction of it.

Related

• PR #9688 / Issue #9210 — fixed the same completion_start_time = end_time fallback for /chat/completions streaming, but the /v1/messages pass-through path was not covered.

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.82.3-stable

Twitter / LinkedIn details

No response