litellm - 💡(How to fix) Fix [Bug]: `POST /v1/messages/count_tokens` silently falls back to local tokeniser for Anthropic-shape content blocks on Bedrock

Body

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When the LiteLLM proxy is configured with a Bedrock-backed Anthropic Claude model and a client POSTs to /v1/messages/count_tokens with a typical Anthropic Messages body (content blocks with {"type": "text"|"tool_use"|"tool_result"}), the request:

1. Routes through BedrockTokenCounter.
2. Hits _detect_input_type → "converse" because messages is a list.
3. _transform_to_converse_format copies the Anthropic content blocks verbatim into a Converse body. Anthropic blocks ({"type": "text", ...}) and Converse blocks ({"text": ...}) are different shapes; Bedrock rejects with HTTP 400 messages.N.content: Field required.
4. _try_provider_token_count catches the error and logs:

   Provider token counting failed (400): {"message":"messages.N.content: Field required"}. Falling back to local tokenizer.

5. Local tiktoken returns a count that does not match what Bedrock would charge for the same body. On a representative tool-heavy ~110k-token Anthropic payload I measured a ~50% under-count (39,326 from the local fallback vs 85,780 from bedrock-runtime:CountTokens directly).

There's a second related bug in the same file: _transform_to_invoke_model_format ships the body as a raw JSON string, but the Bedrock CountTokens API spec describes invokeModel.body as a Base64-encoded blob. Even when InvokeModel mode is selected, Bedrock returns 400 Unable to parse request body for non-trivial payloads.

Steps to reproduce

1. Deploy LiteLLM v1.83.14 (or any recent version since CountTokens was introduced) with a Bedrock-backed Anthropic model in model_list:

   model_list:
     - model_name: claude-sonnet
       litellm_params:
         model: bedrock/anthropic.claude-sonnet-4-6
         aws_region_name: ap-southeast-2

2. Ensure the LiteLLM task IAM role has bedrock:CountTokens (a separate gotcha — without this the symptom is identical because LiteLLM falls back to local on the 403 too).
3. POST /v1/messages/count_tokens with Anthropic-shape content blocks:

   curl -sS http://localhost:4000/v1/messages/count_tokens \
     -H "Authorization: Bearer $KEY" \
     -H 'Content-Type: application/json' \
     -d '{
       "model": "claude-sonnet",
       "messages": [
         {"role": "user", "content": [{"type": "text", "text": "Hello"}]}
       ]
     }'

4. The response succeeds with {"input_tokens": <N>} but <N> is the local-tiktoken count, not Bedrock's. The proxy logs (with LITELLM_LOG=DEBUG) show Provider token counting failed. For tool-using bodies the discrepancy can be ~50%.

Expected behaviour

/v1/messages/count_tokens for Anthropic-shape content should reach bedrock-runtime:CountTokens and return the same number AWS would charge for sending the same body to bedrock-runtime:InvokeModel.

Why this matters

Anyone using the proxy as a count-only check (cost estimation, prompt-budget enforcement, optimisation pipelines that need to compare pre/post token counts) is getting silently incorrect numbers on Bedrock + Anthropic, which is the most common Bedrock model family. The fallback is silent in normal logs (only DEBUG/WARN); production users won't realise their token counts are wrong.

Relevant log output

Provider token counting failed (400): {"message":"messages.1.content: Field required"}. Falling back to local tokenizer.

Root cause

Two bugs in litellm/llms/bedrock/count_tokens/transformation.py:

1. _detect_input_type — returns "converse" whenever messages is a list, without checking whether the content blocks are Anthropic-shape ({"type": ...}) or Converse-shape ({"text": ...} / {"toolUse": ...} / {"toolResult": ...}). The Converse transform doesn't translate Anthropic blocks; it copies them through verbatim:

   elif isinstance(content, list):
       transformed_message["content"] = content

2. _transform_to_invoke_model_format — wraps body as json.dumps(body_data) (a raw JSON string). The Bedrock CountTokens API spec requires invokeModel.body to be Base64-encoded.

Suggested fix

Detect Anthropic-shape content blocks and route them to InvokeModel (which preserves the body verbatim, matching exactly what Bedrock's tokeniser scores), and Base64-encode the InvokeModel body. Draft PR: https://github.com/BerriAI/litellm/pull/27626.

What part of LiteLLM is this about?

Proxy

LiteLLM version

v1.83.14-stable.patch.3 (also reproducible on main at the time of writing)