litellm - 💡(How to fix) Fix [Critical Hotfix Request]: No stable LiteLLM path for Claude Code → vLLM

Summary

I am opening this as a critical hotfix request.

The Claude Code → LiteLLM → vLLM path currently has no stable production-ready routing option.

Both practical LiteLLM paths are broken in different ways:

1. hosted_vllm can corrupt Anthropic Messages streaming.
2. anthropic provider preserves generation semantics, but /v1/messages/count_tokens is broken for custom backends.

As a result, teams using Claude Code with self-hosted vLLM do not currently have a reliable LiteLLM path.

This should be treated as a hotfix-level compatibility issue, not as a normal backlog item.

Affected production path

Claude Code
  → LiteLLM
  → vLLM

This path depends on the following Anthropic-compatible behaviors:

• /v1/messages
• streaming
• content blocks
• tool use blocks
• tool result blocks
• thinking / reasoning blocks
• /v1/messages/count_tokens
• accurate context tracking
• reliable auto compaction behavior

For Claude Code-style coding-agent workflows, these are not optional features. They are part of the core runtime path.

Problem 1: hosted_vllm can corrupt Anthropic Messages streaming

Related issue:

https://github.com/BerriAI/litellm/issues/30014

When LiteLLM is configured with hosted_vllm, Anthropic /v1/messages requests are converted into OpenAI-compatible /v1/chat/completions requests.

That adapter path can drop the first non-empty text delta around content block transitions.

This is a critical issue for Claude Code-style clients.

Dropping even one token can corrupt:

• command output
• code blocks
• tool responses
• structured agent messages
• streaming state transitions

For coding-agent workloads, this is not a cosmetic streaming bug. It can cause sessions to fail, hang, or produce invalid output.

Problem 2: anthropic provider has broken token counting for custom backends

Related issue:

https://github.com/BerriAI/litellm/issues/29764

The anthropic provider is the correct path for preserving native Anthropic Messages behavior with vLLM.

However, /v1/messages/count_tokens currently does not reliably route to the configured Anthropic-compatible backend.

Instead of using the configured api_base, LiteLLM may call Anthropic's public API endpoint.

That is invalid for:

• self-hosted deployments
• private deployments
• firewall-restricted environments
• air-gapped environments
• vLLM-backed Anthropic-compatible deployments

Even worse, when token counting fails or falls back to a local tokenizer, Claude Code auto compaction can become unreliable because the token count no longer matches the backend's actual tokenizer behavior.

For Claude Code, inaccurate token counting directly affects context management and auto compaction. This is also a critical runtime-path issue.

Why both issues must be fixed

The two issues break different parts of the same critical path.

hosted_vllm:
  /v1/messages generation path is unsafe
  because Anthropic streaming semantics can be corrupted

anthropic provider:
  /v1/messages generation path is correct
  but /v1/messages/count_tokens is broken for custom backends

So the practical result is:

Claude Code → LiteLLM → vLLM

has no fully reliable production path today.

This is the core issue.

One path breaks streaming generation.

The other path breaks token counting and auto compaction.

Both are critical for Claude Code.

Additional validation against another gateway

We also tested the same Claude Code-compatible workflow against another LLM gateway, Bifrost, using the same vLLM backend.

In that setup, both critical paths worked correctly:

Claude Code → Bifrost → vLLM /v1/messages
Claude Code → Bifrost → vLLM /v1/messages/count_tokens

Streaming behavior was stable, and token counting was routed to the configured backend as expected.

This is not meant as a product comparison. The point is that the same client and the same vLLM backend can work correctly through a gateway layer.

That makes this look like a LiteLLM compatibility issue, not a Claude Code issue and not a vLLM issue.

Expected behavior

hosted_vllm path

Claude Code
  → LiteLLM /v1/messages
  → vLLM-compatible backend

This path must preserve Anthropic Messages streaming semantics correctly.

It must not:

• drop the first non-empty text delta
• corrupt content block transitions
• break tool-use streaming
• break Claude Code-compatible streaming behavior

anthropic provider path

Claude Code
  → LiteLLM /v1/messages
  → configured vLLM /v1/messages

Claude Code
  → LiteLLM /v1/messages/count_tokens
  → configured vLLM /v1/messages/count_tokens

This path must respect the configured api_base consistently.

If a custom Anthropic-compatible backend is configured, LiteLLM should not call api.anthropic.com for token counting.

Requested hotfix actions

1. Fix hosted_vllm streaming compatibility

For issue #30014:

• ensure the first non-empty text delta is never dropped
• preserve Anthropic content block transition semantics
• add regression tests for empty initial deltas followed by non-empty deltas
• add tests for Claude Code-style streaming payloads if possible

2. Fix anthropic provider token counting

For issue #29764 and PR #29765:

• merge the token counting fix urgently
• ensure /v1/messages/count_tokens respects model-level api_base
• ensure it routes to the configured Anthropic-compatible backend
• ensure it does not call api.anthropic.com when a custom backend is configured
• cover both environment-based and model-config-based base URL settings

3. Release a patch version

Please include these fixes in a patch release.

This should not wait for a normal release cycle. The affected path is a core production path for Claude Code + LiteLLM + self-hosted vLLM users.

Why this should be prioritized

This is not an edge case.

Claude Code → LiteLLM → vLLM is one of the most important deployment combinations for teams using Claude Code-compatible workflows with self-hosted models.

LiteLLM is positioned as a production LLM gateway. For that role, this compatibility path needs to be reliable.

Right now:

• one path risks corrupting the generation stream
• the other path breaks token counting and auto compaction

Both are critical for Claude Code.

Please prioritize these as hotfix-level issues.