Issue Station
Sign In
Tools hereToolbox

vllm - ✅(Solved) Fix [Bug] TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]] on embedding endpoint (v0.19.0) [1 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#40292•Fetched 2026-04-20 11:59:31
View on GitHub
Comments
1
Participants
2
Timeline
4
Reactions
0
Author
jsboigejsboige
Participants
DarkLight1337DarkLight1337jsboigejsboige
Timeline (top)
commented ×1cross-referenced ×1mentioned ×1subscribed ×1

We observe intermittent TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]] errors on the /v1/embeddings endpoint. The error rate is ~0.04% (276 errors out of 714,905 requests over 24 hours). The errors occur in bursts (108 burst errors with <50 OK requests between them, 168 isolated errors), suggesting a specific batch operation triggers the condition.

Error Message

File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/embed/api_router.py", line 42, in create_embedding
    return await handler(request, raw_request)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/base/serving.py", line 99, in __call__
    await self.io_processor.pre_process_online_async(ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/base/io_processor.py", line 82, in pre_process_online_async
    self.pre_process_online(ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/embed/io_processor.py", line 63, in pre_process_online
    super().pre_process_online(ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/base/io_processor.py", line 151, in _preprocess_completion_online
    return renderer.render_cmpl(...)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 837, in render_cmpl
    tok_prompts = self.tokenize_prompts(dict_prompts, tok_params)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 521, in tokenize_prompts
    return [self.tokenize_prompt(prompt, params) for prompt in prompts]
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 514, in tokenize_prompt
    return self._tokenize_singleton_prompt(prompt, params)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 425, in _tokenize_singleton_prompt
    prompt = self._tokenize_prompt(prompt, params)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 372, in _tokenize_prompt
    prompt_token_ids = tokenizer.encode(prompt, ...)
File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 2872, in encode
    encoded_inputs = self.encode_plus(...)
File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_fast.py", line 553, in _batch_encode_plus
    encodings = self._tokenizer.encode_batch(...)
TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

Root Cause

The error occurs in _tokenize_singleton_prompt (line 418-425 of renderers/base.py):

if "prompt_token_ids" not in prompt and "prompt_embeds" not in prompt:
    prompt = params.apply_pre_tokenization(self.tokenizer, prompt)
    prompt = self._tokenize_prompt(prompt, params)  # <-- crashes here

The guard checks for "prompt_token_ids" and "prompt_embeds" keys, but does not validate that prompt["prompt"] is a str before passing it to tokenizer.encode(). The HuggingFace fast tokenizer's encode_batch() only accepts text (strings), not list[int].

The parse_dec_only_prompt function in vllm/renderers/inputs/preprocess.py passes dicts through without runtime type validation:

if isinstance(prompt, dict):
    if ("prompt" in prompt or "prompt_token_ids" in prompt or "prompt_embeds" in prompt):
        return prompt  # passes through as-is, no type checking

Since Python TypedDict annotations are not enforced at runtime, a dict with {"prompt": list[int]} bypasses type checks and reaches the tokenizer.

PR fix notes

PR #40339: [Bugfix] Normalize malformed dict prompts that carry token IDs in prompt

Description (problem / solution / changelog)

Summary

This change hardens prompt preprocessing for renderer-based tokenization paths for #40292

Some requests can currently pass malformed dict prompts such as {"prompt": [1, 2, 3]}. Because TypedDict annotations are not enforced at runtime, these inputs can bypass prompt parsing and later reach Hugging Face tokenization as non-text inputs, causing low-level tokenizer errors.

This patch normalizes that shape at the preprocessing boundary by rewriting:

  • {"prompt": [1, 2, 3]} -> {"prompt_token_ids": [1, 2, 3]}

It also keeps a defensive check in the renderer tokenization path so invalid non-string prompt values fail with a clearer error message instead of surfacing a tokenizer-internal exception.

Files changed

  • vllm/renderers/inputs/preprocess.py
  • vllm/renderers/base.py
  • tests/renderers/inputs/test_preprocess.py

Testing

Added preprocessing tests for:

  • normal text dict prompts
  • dict prompts with token IDs stored in prompt
  • invalid mixed-type prompt lists
  • nested encoder/decoder prompt normalization

GitHub CI is intended to validate the full test result for this patch.

Tips

Since the PR is a rubustness fix, if the issue still lacks minimal example that triggers the error, please close the PR. PTAL.

Changed files

  • tests/renderers/inputs/test_preprocess.py (modified, +34/-1)
  • vllm/renderers/base.py (modified, +10/-0)
  • vllm/renderers/inputs/preprocess.py (modified, +35/-0)

Code Example

File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/embed/api_router.py", line 42, in create_embedding
    return await handler(request, raw_request)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/base/serving.py", line 99, in __call__
    await self.io_processor.pre_process_online_async(ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/base/io_processor.py", line 82, in pre_process_online_async
    self.pre_process_online(ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/embed/io_processor.py", line 63, in pre_process_online
    super().pre_process_online(ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/base/io_processor.py", line 151, in _preprocess_completion_online
    return renderer.render_cmpl(...)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 837, in render_cmpl
    tok_prompts = self.tokenize_prompts(dict_prompts, tok_params)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 521, in tokenize_prompts
    return [self.tokenize_prompt(prompt, params) for prompt in prompts]
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 514, in tokenize_prompt
    return self._tokenize_singleton_prompt(prompt, params)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 425, in _tokenize_singleton_prompt
    prompt = self._tokenize_prompt(prompt, params)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 372, in _tokenize_prompt
    prompt_token_ids = tokenizer.encode(prompt, ...)
File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 2872, in encode
    encoded_inputs = self.encode_plus(...)
File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_fast.py", line 553, in _batch_encode_plus
    encodings = self._tokenizer.encode_batch(...)
TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

---

if "prompt_token_ids" not in prompt and "prompt_embeds" not in prompt:
    prompt = params.apply_pre_tokenization(self.tokenizer, prompt)
    prompt = self._tokenize_prompt(prompt, params)  # <-- crashes here

---

if isinstance(prompt, dict):
    if ("prompt" in prompt or "prompt_token_ids" in prompt or "prompt_embeds" in prompt):
        return prompt  # passes through as-is, no type checking

---

if "prompt_token_ids" not in prompt and "prompt_embeds" not in prompt:
    p = prompt.get("prompt")
    if not isinstance(p, str):
        # If prompt is already token IDs, convert to prompt_token_ids
        if isinstance(p, list) and all(isinstance(x, int) for x in p):
            prompt["prompt_token_ids"] = p
            del prompt["prompt"]
        else:
            raise TypeError(
                f"Expected prompt['prompt'] to be str, got {type(p).__name__}"
            )
    prompt = params.apply_pre_tokenization(self.tokenizer, prompt)
    prompt = self._tokenize_prompt(prompt, params)
RAW_BUFFERClick to expand / collapse

Bug Report

Environment

  • vLLM version: v0.19.0
  • Model: Qwen3-4B-AWQ (compressed-tensors W4A16)
  • Runner: --runner pooling (embedding task)
  • GPU: NVIDIA RTX 3080 Laptop 16GB
  • Docker image: vllm/vllm-openai:v0.19.0

Description

We observe intermittent TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]] errors on the /v1/embeddings endpoint. The error rate is ~0.04% (276 errors out of 714,905 requests over 24 hours). The errors occur in bursts (108 burst errors with <50 OK requests between them, 168 isolated errors), suggesting a specific batch operation triggers the condition.

Stack Trace

File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/embed/api_router.py", line 42, in create_embedding
    return await handler(request, raw_request)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/base/serving.py", line 99, in __call__
    await self.io_processor.pre_process_online_async(ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/base/io_processor.py", line 82, in pre_process_online_async
    self.pre_process_online(ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/embed/io_processor.py", line 63, in pre_process_online
    super().pre_process_online(ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/pooling/base/io_processor.py", line 151, in _preprocess_completion_online
    return renderer.render_cmpl(...)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 837, in render_cmpl
    tok_prompts = self.tokenize_prompts(dict_prompts, tok_params)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 521, in tokenize_prompts
    return [self.tokenize_prompt(prompt, params) for prompt in prompts]
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 514, in tokenize_prompt
    return self._tokenize_singleton_prompt(prompt, params)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 425, in _tokenize_singleton_prompt
    prompt = self._tokenize_prompt(prompt, params)
File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/base.py", line 372, in _tokenize_prompt
    prompt_token_ids = tokenizer.encode(prompt, ...)
File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 2872, in encode
    encoded_inputs = self.encode_plus(...)
File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_fast.py", line 553, in _batch_encode_plus
    encodings = self._tokenizer.encode_batch(...)
TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

Root Cause Analysis

The error occurs in _tokenize_singleton_prompt (line 418-425 of renderers/base.py):

if "prompt_token_ids" not in prompt and "prompt_embeds" not in prompt:
    prompt = params.apply_pre_tokenization(self.tokenizer, prompt)
    prompt = self._tokenize_prompt(prompt, params)  # <-- crashes here

The guard checks for "prompt_token_ids" and "prompt_embeds" keys, but does not validate that prompt["prompt"] is a str before passing it to tokenizer.encode(). The HuggingFace fast tokenizer's encode_batch() only accepts text (strings), not list[int].

The parse_dec_only_prompt function in vllm/renderers/inputs/preprocess.py passes dicts through without runtime type validation:

if isinstance(prompt, dict):
    if ("prompt" in prompt or "prompt_token_ids" in prompt or "prompt_embeds" in prompt):
        return prompt  # passes through as-is, no type checking

Since Python TypedDict annotations are not enforced at runtime, a dict with {"prompt": list[int]} bypasses type checks and reaches the tokenizer.

Suggested Fix

Add a type guard in _tokenize_singleton_prompt before calling _tokenize_prompt:

if "prompt_token_ids" not in prompt and "prompt_embeds" not in prompt:
    p = prompt.get("prompt")
    if not isinstance(p, str):
        # If prompt is already token IDs, convert to prompt_token_ids
        if isinstance(p, list) and all(isinstance(x, int) for x in p):
            prompt["prompt_token_ids"] = p
            del prompt["prompt"]
        else:
            raise TypeError(
                f"Expected prompt['prompt'] to be str, got {type(p).__name__}"
            )
    prompt = params.apply_pre_tokenization(self.tokenizer, prompt)
    prompt = self._tokenize_prompt(prompt, params)

Impact

  • Error rate: ~0.04% of all embedding requests
  • Client impact: 400 Bad Request returned; client must retry
  • Service health: Container remains healthy, no memory leaks or crashes

How to Reproduce

Running vLLM v0.19.0 with an embedding model, the error appears intermittently under heavy indexing workloads (700K+ requests/day). To capture the exact triggering request, enabling debug logging (VLLM_LOGGING_LEVEL=debug) would be needed.

extent analysis

TL;DR

Add a type guard in _tokenize_singleton_prompt to ensure prompt["prompt"] is a string before passing it to the tokenizer.

Guidance

  • Verify the type of prompt["prompt"] before calling tokenizer.encode() to prevent passing non-string values.
  • Implement the suggested fix in _tokenize_singleton_prompt to add a type check and handle cases where prompt["prompt"] is already token IDs.
  • Test the updated code with a mix of string and non-string inputs for prompt["prompt"] to ensure the type guard is effective.
  • Monitor the error rate after applying the fix to confirm its effectiveness.

Example

if "prompt_token_ids" not in prompt and "prompt_embeds" not in prompt:
    p = prompt.get("prompt")
    if not isinstance(p, str):
        # If prompt is already token IDs, convert to prompt_token_ids
        if isinstance(p, list) and all(isinstance(x, int) for x in p):
            prompt["prompt_token_ids"] = p
            del prompt["prompt"]
        else:
            raise TypeError(
                f"Expected prompt['prompt'] to be str, got {type(p).__name__}"
            )

Notes

The suggested fix assumes that the prompt["prompt"] value is either a string or a list of integers representing token IDs. If other types of values are possible, additional handling may be necessary.

Recommendation

Apply the suggested fix to add a type guard in _tokenize_singleton_prompt to prevent the TypeError and ensure correct handling of different input types.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#conversation history#tool integration#LLM response#memory leak

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING