litellm - 💡(How to fix) Fix [Bug]: Streaming SSE output differs from upstream for /v1/messages and /v1/responses

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  store_model_in_db: true
litellm_settings:
  model_list: []
  request_timeout: 6000
router_settings:
  routing_strategy: simple-shuffle
  model_group_alias: {}
  fallbacks: []

---

curl -sS -N http://upstream/v1/messages \
  -H 'Content-Type: application/json' \
  -d '{"model":"nvidia/nemotron-3-nano-4b","messages":[{"role":"user","content":"hi"}],"max_tokens":32,"stream":true}'

curl -sS -N http://upstream/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{"model":"nvidia/nemotron-3-nano-4b","input":"hi","max_output_tokens":32,"stream":true}'

---

docker compose exec -T gateway python - <<'PY'
import os, httpx
BASE='http://litellm:4000'
MASTER=os.environ['LITELLM_MASTER_KEY']
headers={'Authorization': f'Bearer {MASTER}', 'Content-Type':'application/json'}

with httpx.stream('POST', f'{BASE}/v1/messages', headers=headers, json={
  "model":"nemotron-3-nano-4b",
  "messages":[{"role":"user","content":"hi"}],
  "max_tokens":32,
  "stream":True
}, timeout=20) as s:
  for line in s.iter_lines():
    print(line)

with httpx.stream('POST', f'{BASE}/v1/responses', headers=headers, json={
  "model":"nemotron-3-nano-4b",
  "input":"hi",
  "max_output_tokens":32,
  "stream":True
}, timeout=20) as s:
  for line in s.iter_lines():
    print(line)
PY

---

Upstream: `POST /v1/messages` (`stream=true`) (trimmed)

event: message_start
data: {"type":"message_start",...}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}
...


LiteLLM: `POST /v1/messages` (`stream=true`) (trimmed)

event: message_start
data: {"type": "message_start", ...}

event: content_block_delta
data: {"type": "content_block_delta", "index": -1, "delta": {"type": "text_delta", "text": "\n\nHel"}}
...


Upstream: `POST /v1/responses` (`stream=true`) (trimmed)

event: response.created
data: {"type":"response.created",...}

event: response.output_item.added
data: {"type":"response.output_item.added",...}

event: response.content_part.added
data: {"type":"response.content_part.added",...}
...


LiteLLM: `POST /v1/responses` (`stream=true`) (trimmed)

data: {"type":"response.output_text.delta","item_id":"resp_...","output_index":0,"content_index":0,"delta":"\nHell","model":"nemotron-3-nano-4b"}
...

data: {"type":"response.completed",...}
data: [DONE]