vllm - 💡(How to fix) Fix [Feature]: Expose Word-Level Timestamps in `/v1/realtime` API for Voxtral Realtime [1 participants]

Summary

The vLLM Realtime API (/v1/realtime) currently returns only text in transcription.delta and transcription.done events. However, the Voxtral Realtime model already has enough internal information to derive per-word timestamps via [STREAMING_WORD] token positions and the known transcription_delay. Exposing these timestamps in the WebSocket protocol would unlock critical downstream use cases without any model changes.

Motivation

Current behavior

{"type": "transcription.delta", "delta": "Hello world"}
{"type": "transcription.done", "text": "Hello world."}

Clients receive text but have no timing information. Applications that need to know when a word was spoken must resort to coarse heuristics (e.g., wall-clock arrival time of deltas) or run a separate forced-alignment model.

Why this matters

Technical Background

The Voxtral Realtime model uses a [STREAMING_WORD] token as a word boundary marker in its token vocabulary. The position (index) of each [STREAMING_WORD] token in the generated sequence is directly tied to the audio frame timeline.

From tekken.json:

"audio": {
  "sampling_rate": 16000,
  "frame_rate": 12.5,
  "audio_encoding_config": {
    "num_mel_bins": 128,
    "hop_length": 160,
    "window_size": 400
  },
  "transcription_delay_ms": 480,
  "streaming_look_ahead_ms": 2.5,
  "streaming_look_back_ms": 52.5,
  "streaming_n_left_pad_tokens": 32,
  "transcription_format": "streaming"
}

Key parameters:

• Each text token corresponds to 80ms of audio
• transcription_delay_ms = 480 → delay in tokens: 480 / 80 = 6 tokens
• frame_rate = 12.5 frames/sec → 1 frame = 80ms

Deriving a word's end timestamp

end_frame_idx = streaming_word_token_idx - transcription_delay_in_tokens
end_time_sec  = end_frame_idx / frame_rate
end_time_ms   = end_time_sec * 1000

For example, if [STREAMING_WORD] appears at token index 25:

end_frame_idx = 25 - 6 = 19
end_time_ms   = (19 / 12.5) * 1000 = 1520 ms

This computation is trivial and can be performed in the serving layer without any model modifications.

Proposed API Change

Option A: Add end_time_ms to delta events (minimal)

{
  "type": "transcription.delta",
  "delta": "world",
  "end_time_ms": 1520
}

A single timestamp per emission group — the end time of the last word in the delta. Simple, low overhead, sufficient for most use cases.

Option B: Add word_timestamps array (detailed)

{
  "type": "transcription.delta",
  "delta": "Hello world",
  "word_timestamps": [
    {"word": "Hello", "end_ms": 960},
    {"word": "world", "end_ms": 1520}
  ]
}

Backward compatibility

Both options are fully backward-compatible — new fields are additive. Existing clients that ignore unknown fields will continue to work unchanged. Optionally, timestamp inclusion could be gated behind a session parameter (e.g., "include_timestamps": true in session.update).

Implementation Notes

The change is localized to vLLM's realtime endpoint handler where generated tokens are decoded and streamed to the client:

1. During token-by-token generation, detect [STREAMING_WORD] token IDs
2. Record their position in the sequence
3. Apply the delay offset: position - transcription_delay_tokens
4. Convert to milliseconds: adjusted_position * 80
5. Attach to the outgoing WebSocket message

No changes to the model, tokenizer, or audio encoder are required.

References

• Model: mistralai/Voxtral-Mini-4B-Realtime-2602
• Technical report: arxiv:2602.11298
• vLLM Realtime API docs: vLLM Realtime
• Related HF discussion: #2 — transcription_delay param
• vLLM blog on streaming input: [blog.vllm.ai](https://blog.vllm.ai/2026/01/31/streaming-r