Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Bug]: Voyage multimodal embedding models (voyage-multimodal-3/3.5) route to wrong API endpoint [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#24957•Fetched 2026-04-08 02:23:32
View on GitHub
Comments
0
Participants
1
Timeline
2
Reactions
0
Author
silencedoctorsilencedoctor
Participants
silencedoctorsilencedoctor
Timeline (top)
cross-referenced ×1referenced ×1

Root Cause

This means any multimodal input (text + images) sent through LiteLLM will fail because the standard /v1/embeddings endpoint does not accept image content.

PR fix notes

PR #24972: feat(voyage): add multimodal embedding support for voyage-multimodal-3/3.5

Description (problem / solution / changelog)

Summary

Voyage multimodal embedding models (voyage-multimodal-3, voyage-multimodal-3.5) currently route to the standard /v1/embeddings endpoint, which does not support multimodal inputs (images, videos). This PR adds proper support by:

  • Creating VoyageMultimodalEmbeddingConfig that routes to the correct /v1/multimodalembeddings endpoint
  • Transforming inputs to the Voyage multimodal format (supporting text, image_url, image_base64, video_url, video_base64 content types)
  • Converting OpenAI-style image_url content blocks (with nested {"url": "..."}) to Voyage's flat format
  • Automatically detecting base64 data URIs in image_url blocks and converting them to image_base64 type
  • Adding voyage-multimodal-3.5 to the model pricing registry

Fixes #24957

Changes

FileDescription
litellm/llms/voyage/embedding/transformation_multimodal.pyNew VoyageMultimodalEmbeddingConfig class
litellm/utils.pyRoute multimodal models to new config
litellm/__init__.pyRegister new config export
litellm/_lazy_imports_registry.pyRegister lazy import
litellm/model_prices_and_context_window_backup.jsonAdd voyage-multimodal-3.5, add supports_vision flag
tests/test_litellm/llms/voyage/embedding/test_voyage_multimodal_embedding.py32 unit tests

Usage

import litellm

# Text-only input
response = litellm.embedding(
    model="voyage/voyage-multimodal-3",
    input=["Hello world"],
)

# Multimodal input (text + image)
response = litellm.embedding(
    model="voyage/voyage-multimodal-3",
    input=[{
        "content": [
            {"type": "text", "text": "A photo of a cat"},
            {"type": "image_url", "image_url": "https://example.com/cat.jpg"},
        ]
    }],
)

# OpenAI-style image_url format also supported
response = litellm.embedding(
    model="voyage/voyage-multimodal-3.5",
    input=[{
        "content": [
            {"type": "text", "text": "Describe this"},
            {"type": "image_url", "image_url": {"url": "https://example.com/img.png"}},
        ]
    }],
)

PR Checklist

  • Tests added (tests/test_litellm/llms/voyage/embedding/test_voyage_multimodal_embedding.py - 32 tests)
  • All unit tests pass
  • Linting passes
  • PR scope is isolated - only adds Voyage multimodal embedding support

What type of PR is this?

🆕 New Feature

Reference

Changed files

  • litellm/__init__.py (modified, +3/-0)
  • litellm/_lazy_imports_registry.py (modified, +5/-0)
  • litellm/llms/voyage/embedding/transformation_multimodal.py (added, +225/-0)
  • litellm/model_prices_and_context_window_backup.json (modified, +11/-1)
  • litellm/utils.py (modified, +16/-0)
  • tests/test_litellm/llms/voyage/embedding/test_voyage_multimodal_embedding.py (added, +418/-0)

Code Example

if LlmProviders.VOYAGE == provider and is_contextualized_embeddings(model):
    return VoyageContextualEmbeddingConfig()
elif LlmProviders.VOYAGE == provider:
    return VoyageEmbeddingConfig()  # <-- multimodal models fall through to here

---

import litellm

# This fails because it hits /v1/embeddings instead of /v1/multimodalembeddings
response = litellm.embedding(
    model="voyage/voyage-multimodal-3",
    input=[{"content": [{"type": "text", "text": "A photo of a cat"}, {"type": "image_url", "image_url": "https://example.com/cat.jpg"}]}],
)
RAW_BUFFERClick to expand / collapse

What happened?

When using Voyage multimodal embedding models (voyage-multimodal-3, voyage-multimodal-3.5), LiteLLM routes requests to the standard text embedding endpoint (/v1/embeddings) instead of the correct multimodal endpoint (/v1/multimodalembeddings).

This means any multimodal input (text + images) sent through LiteLLM will fail because the standard /v1/embeddings endpoint does not accept image content.

Relevant code

The routing logic in litellm/utils.py (ProviderConfigManager.get_provider_embedding_config) currently only distinguishes between standard and contextual Voyage embeddings:

if LlmProviders.VOYAGE == provider and is_contextualized_embeddings(model):
    return VoyageContextualEmbeddingConfig()
elif LlmProviders.VOYAGE == provider:
    return VoyageEmbeddingConfig()  # <-- multimodal models fall through to here

And VoyageEmbeddingConfig.get_complete_url() always returns https://api.voyageai.com/v1/embeddings.

Expected behavior

  • voyage-multimodal-3 and voyage-multimodal-3.5 should route to https://api.voyageai.com/v1/multimodalembeddings
  • Input should be transformed to the Voyage multimodal format (content blocks with type, text, image_url, image_base64, etc.)
  • Reference: https://docs.voyageai.com/reference/multimodal-embeddings-api

Reproduction

import litellm

# This fails because it hits /v1/embeddings instead of /v1/multimodalembeddings
response = litellm.embedding(
    model="voyage/voyage-multimodal-3",
    input=[{"content": [{"type": "text", "text": "A photo of a cat"}, {"type": "image_url", "image_url": "https://example.com/cat.jpg"}]}],
)

LiteLLM component

SDK

Are you a LiteLLM Enterprise customer?

No

Twitter / LinkedIn details

No response

extent analysis

TL;DR

Update the routing logic in litellm/utils.py to correctly distinguish between standard, contextual, and multimodal Voyage embeddings.

Guidance

  • Modify the ProviderConfigManager.get_provider_embedding_config method to include a condition for multimodal embeddings, checking for models like voyage-multimodal-3 and voyage-multimodal-3.5.
  • Update VoyageEmbeddingConfig.get_complete_url() to return https://api.voyageai.com/v1/multimodalembeddings for multimodal models.
  • Verify the fix by testing the litellm.embedding function with a multimodal input, ensuring it routes to the correct endpoint.
  • Consider adding a new config class, e.g., VoyageMultimodalEmbeddingConfig, to handle the specific requirements of multimodal embeddings.

Example

if LlmProviders.VOYAGE == provider and is_contextualized_embeddings(model):
    return VoyageContextualEmbeddingConfig()
elif LlmProviders.VOYAGE == provider and model in ["voyage-multimodal-3", "voyage-multimodal-3.5"]:
    return VoyageMultimodalEmbeddingConfig()
elif LlmProviders.VOYAGE == provider:
    return VoyageEmbeddingConfig()

Notes

The current implementation does not account for multimodal embeddings, leading to incorrect routing. The proposed fix aims to address this issue by introducing a new condition and config class for multimodal models.

Recommendation

Apply workaround by updating the routing logic and config classes to support multimodal embeddings, as this will allow for correct routing and processing of multimodal input.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

  • voyage-multimodal-3 and voyage-multimodal-3.5 should route to https://api.voyageai.com/v1/multimodalembeddings
  • Input should be transformed to the Voyage multimodal format (content blocks with type, text, image_url, image_base64, etc.)
  • Reference: https://docs.voyageai.com/reference/multimodal-embeddings-api
#api#retriever error#indexing error#inference speed#output truncation

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

litellm - ✅(Solved) Fix [Bug]: Voyage multimodal embedding models (voyage-multimodal-3/3.5) route to wrong API endpoint [1 pull requests, 1 participants]