litellm - 💡(How to fix) Fix EmbeddingResponse(**response_json) fails with `must be a mapping, not list` when using openai_like/lm_studio provider with llama.cpp server [2 pull requests]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Error Message

lm_studio/ and hosted_vllm/:

File "litellm/llms/openai_like/embedding/handler.py", line 75, in aembedding
    return EmbeddingResponse(**response_json)
TypeError: litellm.types.utils.EmbeddingResponse() argument after ** must be a mapping, not list

openai/ — same root cause, different layer:

File "litellm/llms/openai/openai.py", line 1269, in aembedding
    stringified_response = response.model_dump()
AttributeError: 'list' object has no attribute 'model_dump'

Root Cause

When using lm_studio/ (or hosted_vllm/, openai/) as the provider for an OpenAI-compatible embedding server (llama.cpp), LiteLLM raises a TypeError because response_json passed to EmbeddingResponse(**response_json) is a list instead of a dict.

Fix Action

Fixed

Code Example

File "litellm/llms/openai_like/embedding/handler.py", line 75, in aembedding
    return EmbeddingResponse(**response_json)
TypeError: litellm.types.utils.EmbeddingResponse() argument after ** must be a mapping, not list

---

File "litellm/llms/openai/openai.py", line 1269, in aembedding
    stringified_response = response.model_dump()
AttributeError: 'list' object has no attribute 'model_dump'

---

docker run ghcr.io/ggml-org/llama.cpp:server-vulkan \
  --embeddings --host 0.0.0.0 --port 8080 \
  --hf-repo nomic-ai/nomic-embed-text-v2-moe-GGUF \
  --hf-file nomic-embed-text-v2-moe.Q4_K_M.gguf

---

model_list:
  - model_name: my-embeddings
    litellm_params:
      model: lm_studio/nomic-embed-text-v2-moe
      api_base: http://llamacpp-server:8080
      api_key: dummy

---

curl -X POST http://localhost:4000/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <master_key>" \
  -d '{"model": "my-embeddings", "input": ["hello world"]}'

---

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": ["hello world"], "model": "nomic-embed-text-v2-moe"}'

---

{
  "model": "nomic-embed-text-v2-moe",
  "object": "list",
  "usage": { "prompt_tokens": 5, "total_tokens": 5 },
  "data": [
    {
      "object": "embedding",
      "embedding": [0.01502, -0.03024, ...],
      "index": 0
    }
  ]
}
RAW_BUFFERClick to expand / collapse

Bug Description

When using lm_studio/ (or hosted_vllm/, openai/) as the provider for an OpenAI-compatible embedding server (llama.cpp), LiteLLM raises a TypeError because response_json passed to EmbeddingResponse(**response_json) is a list instead of a dict.

The upstream server returns a valid, standard OpenAI-format response. The bug is in LiteLLM's internal handling of that response.


Error

lm_studio/ and hosted_vllm/:

File "litellm/llms/openai_like/embedding/handler.py", line 75, in aembedding
    return EmbeddingResponse(**response_json)
TypeError: litellm.types.utils.EmbeddingResponse() argument after ** must be a mapping, not list

openai/ — same root cause, different layer:

File "litellm/llms/openai/openai.py", line 1269, in aembedding
    stringified_response = response.model_dump()
AttributeError: 'list' object has no attribute 'model_dump'

Steps to Reproduce

  1. Run a llama.cpp embedding server with --embeddings flag:
docker run ghcr.io/ggml-org/llama.cpp:server-vulkan \
  --embeddings --host 0.0.0.0 --port 8080 \
  --hf-repo nomic-ai/nomic-embed-text-v2-moe-GGUF \
  --hf-file nomic-embed-text-v2-moe.Q4_K_M.gguf
  1. Configure LiteLLM config.yaml:
model_list:
  - model_name: my-embeddings
    litellm_params:
      model: lm_studio/nomic-embed-text-v2-moe
      api_base: http://llamacpp-server:8080
      api_key: dummy
  1. Call the embeddings endpoint through the LiteLLM proxy:
curl -X POST http://localhost:4000/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <master_key>" \
  -d '{"model": "my-embeddings", "input": ["hello world"]}'

Verified: Server Response Is Valid

The llama.cpp server returns correct OpenAI format for both string and list inputs — confirmed via direct curl, bypassing LiteLLM:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": ["hello world"], "model": "nomic-embed-text-v2-moe"}'

Response:

{
  "model": "nomic-embed-text-v2-moe",
  "object": "list",
  "usage": { "prompt_tokens": 5, "total_tokens": 5 },
  "data": [
    {
      "object": "embedding",
      "embedding": [0.01502, -0.03024, ...],
      "index": 0
    }
  ]
}

This is a fully valid EmbeddingResponse structure.


Root Cause (suspected)

In litellm/llms/openai_like/embedding/handler.py, response_json appears to be set to response["data"] (the list of embedding objects) rather than the full response dict before reaching line 75. EmbeddingResponse(**list) then fails because ** unpacking requires a mapping, not a list.

For the openai/ provider, response appears to become a list (likely response.data) somewhere before .model_dump() is called in openai.py.


Environment

LiteLLM versionlatest (ghcr.io/berriai/litellm, no pinned tag)
Python3.13
Embedding serverghcr.io/ggml-org/llama.cpp:server-vulkan
Modelnomic-ai/nomic-embed-text-v2-moe-GGUF Q4_K_M
Providers triedlm_studio/, hosted_vllm/, openai/ — all fail

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING