langchain - ✅(Solved) Fix `convert_to_openai_image_block` drops the `detail` field for image content blocks [10 pull requests, 9 comments, 8 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
langchain-ai/langchain#36297Fetched 2026-04-08 01:41:14
View on GitHub
Comments
9
Participants
8
Timeline
31
Reactions
0
Timeline (top)
cross-referenced ×11commented ×9referenced ×6labeled ×4

Summary

convert_to_openai_image_block does not detect or forward the optional detail field from ImageContentBlock / Base64ContentBlock inputs. This means the detail parameter (which controls image analysis resolution — "low", "high", or "auto") is silently lost when formatting blocks for both the OpenAI Chat Completions and Responses APIs:

Root Cause

convert_to_openai_image_block constructs the image_url dict with only the url key, ignoring any detail field present in the input block: https://github.com/langchain-ai/langchain/blob/2aeeb58ef11e74ab98f869d6ae5c2ae04721be06/libs/core/langchain_core/messages/block_translators/openai.py#L22-L55

The downstream convert_to_openai_data_block calls convert_to_openai_image_block and then checks chat_completions_block["image_url"].get("detail") to forward detail to the Responses API format. Since convert_to_openai_image_block never includes detail, this check always evaluates to False. In the end both APIs affected: https://github.com/langchain-ai/langchain/blob/2aeeb58ef11e74ab98f869d6ae5c2ae04721be06/libs/core/langchain_core/messages/block_translators/openai.py#L58-L90

Expected Behavior

Models don't always analyze original images. Sometimes, images are resized based on their size or the optional detail field before analysis. If present, the optional detail field must be forwarded. When a detail field is supplied (at the top level, or nested in extras/metadata for backwards compatibility), the formatted output should include it:

Chat Completions API:

{"type": "image_url", "image_url": {"url": "https://...", "detail": "high"}}

Responses API:

{"type": "input_image", "image_url": "https://...", "detail": "high"}

Actual Behavior

The detail field is dropped in all cases:

Chat Completions API:

{"type": "image_url", "image_url": {"url": "https://..."}}

Responses API:

{"type": "input_image", "image_url": "https://..."}

Test Coverage Gap

Existing tests for both test_convert_to_openai_image_block and test_convert_to_openai_data_block do not include any inputs with a detail field, so this behavior was never caught.

Error Message

Error Message and Stack Trace (if applicable)

Root Cause

Root Cause

Fix Action

Fix / Workaround

  • This is a bug, not a usage question.
  • I added a clear and descriptive title that summarizes this issue.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
  • This is not related to the langchain-community package.
  • I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Other Dependencies

httpx: 0.28.1 jsonpatch: 1.33 langgraph: 1.1.3 openai: 2.30.0 opentelemetry-api: 1.39.1 opentelemetry-exporter-otlp-proto-http: 1.39.1 opentelemetry-sdk: 1.39.1 orjson: 3.11.7 packaging: 26.0 pydantic: 2.12.5 pytest: 9.0.2 pyyaml: 6.0.3 requests: 2.33.0 requests-toolbelt: 1.0.0 rich: 14.3.3 tenacity: 9.1.4 tiktoken: 0.12.0 typing-extensions: 4.15.0 uuid-utils: 0.14.1 websockets: 16.0 wrapt: 2.1.1 xxhash: 3.6.0 zstandard: 0.25.0

PR fix notes

PR #36298: fix(core): forward detail field in convert_to_openai_image_block

Description (problem / solution / changelog)

convert_to_openai_image_block constructs the image_url dict with only the url key, ignoring any detail field in the input block.

This also breaks convert_to_openai_data_block for the Responses API, which checks chat_completions_block["image_url"].get("detail") — a condition that can never be true since the upstream function never includes detail.

Extract detail from the block (top-level, extras, or metadata) and include it in the output when present. Fix the tests covering both functions and both APIs.

Fixes #36297

Social handles

Twitter: @insilications

Changed files

  • libs/core/langchain_core/messages/block_translators/openai.py (modified, +18/-18)
  • libs/core/tests/unit_tests/messages/block_translators/test_openai.py (modified, +323/-80)
  • libs/core/tests/unit_tests/test_messages.py (modified, +62/-1)

PR #36327: core: Fix convert_to_openai_image_block dropping detail field

Description (problem / solution / changelog)

Fixes #36297

convert_to_openai_image_block constructs the image_url dict with only the url key, silently dropping the detail field that controls image resolution. This affects both Chat Completions and Responses API paths.

The fix extracts detail from the input block (checking direct attributes, extras, and metadata for backwards compatibility) and includes it in the output dict when present. Added tests covering all lookup paths for both convert_to_openai_image_block and convert_to_openai_data_block, plus no-detail cases to prevent regression.

Changed files

  • libs/core/langchain_core/messages/block_translators/openai.py (modified, +25/-6)
  • libs/core/tests/unit_tests/messages/block_translators/test_openai.py (modified, +89/-0)

PR #36337: fix(core): preserve detail parameter in openai image blocks

Description (problem / solution / changelog)

Fixes #36297

This PR updates convert_to_openai_image_block to correctly preserve the optional detail parameter and forward it into the final image_url dictionary. I also added explicit test cases mimicking the reporter's examples to the OpenAI block translator test suite to prevent future regressions.

Added 6 new parameterized test cases to libs/core/tests/unit_tests/messages/block_translators/test_openai.py and successfully ran make test, make format, and make lint against the langchain_core package.

Changed files

  • libs/core/langchain_core/messages/block_translators/openai.py (modified, +24/-6)
  • libs/core/tests/unit_tests/messages/block_translators/test_openai.py (modified, +73/-1)

PR #36345: fix(core): forward detail field in convert_to_openai_image_block

Description (problem / solution / changelog)

The convert_to_openai_image_block function silently dropped the optional detail field from image content blocks, affecting both the OpenAI Chat Completions and Responses APIs.

This commit adds a private _extract_detail helper that checks for detail at the top level, in extras, and in metadata (for backwards compatibility). The extracted value is included in the image_url dict when present.

Fixes #36297

Fixes #

<!-- Replace everything above this line with a 1-2 sentence description of your change. Keep the "Fixes #xx" keyword and update the issue number. -->

Read the full contributing guidelines: https://docs.langchain.com/oss/python/contributing/overview

All contributions must be in English. See the language policy.

If you paste a large clearly AI generated description here your PR may be IGNORED or CLOSED!

Thank you for contributing to LangChain! Follow these steps to have your pull request considered as ready for review.

  1. PR title: Should follow the format: TYPE(SCOPE): DESCRIPTION
  1. PR description:
  • Write 1-2 sentences summarizing the change.
  • The Fixes #xx line at the top is required for external contributions — update the issue number and keep the keyword. This links your PR to the approved issue and auto-closes it on merge.
  • If there are any breaking changes, please clearly describe them.
  • If this PR depends on another PR being merged first, please include "Depends on #PR_NUMBER" in the description.
  1. Run make format, make lint and make test from the root of the package(s) you've modified.
  • We will not consider a PR unless these three are passing in CI.
  1. How did you verify your code works?

Additional guidelines:

  • All external PRs must link to an issue or discussion where a solution has been approved by a maintainer, and you must be assigned to that issue. PRs without prior approval will be closed.
  • PRs should not touch more than one package unless absolutely necessary.
  • Do not update the uv.lock files or add dependencies to pyproject.toml files (even optional ones) unless you have explicit permission to do so by a maintainer.

Social handles (optional)

<!-- If you'd like a shoutout on release, add your socials below -->

LinkedIn: https://www.linkedin.com/in/md-samiul-islam-17738a1b9

Changed files

  • libs/core/langchain_core/messages/block_translators/openai.py (modified, +36/-6)
  • libs/core/tests/unit_tests/messages/block_translators/test_openai.py (modified, +76/-0)
  • libs/core/tests/unit_tests/test_messages.py (modified, +78/-0)

PR #36381: fix(core): preserve detail field in convert_to_openai_image_block

Description (problem / solution / changelog)

Summary

Fixes #36297 — convert_to_openai_image_block silently drops the detail field ("low", "high", "auto") from image content blocks.

Root cause: Both the url and base64 branches built image_url with only {"url": ...}, never forwarding detail. The downstream convert_to_openai_data_block Responses API path then conditionally copies detail from the already-built Chat Completions block — so the field was lost for both API targets.

Fix: Extract detail from the input block (top-level key → extrasmetadata fallback chain) and conditionally include it in the image_url dict.

# Before
return {"type": "image_url", "image_url": {"url": block["url"]}}

# After
detail = block.get("detail") or block.get("extras", {}).get("detail") or block.get("metadata", {}).get("detail")
image_url: dict[str, Any] = {"url": block["url"]}
if detail:
    image_url["detail"] = detail
return {"type": "image_url", "image_url": image_url}

The same fix applies to the base64 branch.

Test plan

  • test_convert_to_openai_image_block_preserves_detail — url path, base64 path, extras-nested detail, absent detail
  • test_convert_to_openai_data_block_image_preserves_detail — Chat Completions API and Responses API
  • Run pytest libs/core/tests/unit_tests/messages/block_translators/test_openai.py

Changed files

  • libs/core/langchain_core/messages/block_translators/openai.py (modified, +9/-12)
  • libs/core/tests/unit_tests/messages/block_translators/test_openai.py (modified, +53/-0)

PR #36425: fix(core): forward detail field in convert_to_openai_image_block

Description (problem / solution / changelog)

Why

convert_to_openai_image_block constructs the image_url dict with only the url key, silently dropping any detail field present in the input block. Since convert_to_openai_data_block delegates to convert_to_openai_image_block and checks chat_completions_block['image_url'].get('detail') to forward detail to the Responses API format, the detail field was lost for both the Chat Completions API and the Responses API.

The detail parameter controls image analysis resolution ('low', 'high', 'auto') and is documented by OpenAI. Silently dropping it causes models to silently fall back to their default behavior, which may result in lower-quality or more expensive image analysis than the caller intended.

Fixes #36297

What changed

libs/core/langchain_core/messages/block_translators/openai.py

In convert_to_openai_image_block: resolve detail from the input block (top-level → extras → metadata for backwards compat, following the existing pattern used throughout the file for ilename) and include it in the constructed image_url dict when present. No change needed in convert_to_openai_data_block — it already forwards detail from the inner dict.

No public API changes. Fully backward compatible.

Tests

19 new parametrized unit tests in est_openai.py:

TestCoverage
est_convert_to_openai_image_block_url_preserves_detail6 cases: URL source × {top-level, extras, metadata} × {with/without source_type}
est_convert_to_openai_image_block_base64_preserves_detail6 cases: base64 source × {top-level, extras, metadata} × {legacy ase64 key, source_type=base64}
est_convert_to_openai_image_block_no_detail_omitteddetail key absent when not supplied
est_convert_to_openai_data_block_chat_completions_preserves_detail3 cases: Chat Completions API end-to-end
est_convert_to_openai_data_block_responses_preserves_detail3 cases: Responses API end-to-end

All 19 pass. All pre-existing tests in the file continue to pass.

Areas requiring review

  • The detail resolution chain uses or short-circuit: lock.get('detail') or (block.get('extras') or {}).get('detail') or (block.get('metadata') or {}).get('detail'). This matches the existing pattern for ilename lookup in convert_to_openai_data_block for file blocks. If detail could ever be a falsy value (it can't per OpenAI spec — valid values are 'low', 'high', 'auto'), an explicit is not None check would be safer, but this mirrors existing style.

AI disclosure

This PR was developed with AI assistance. I have reviewed every line of the fix and the tests, understand the root cause, and verified all tests pass locally.

Changed files

  • libs/core/langchain_core/messages/block_translators/openai.py (modified, +15/-6)
  • libs/core/tests/unit_tests/messages/block_translators/test_openai.py (modified, +179/-0)

PR #36468: fix(core): forward detail field in convert_to_openai_image_block

Description (problem / solution / changelog)

Summary

Fixes #36297

The detail field controls image analysis resolution (low, high, auto) in OpenAI API. Previously, this field was silently dropped when converting image blocks via convert_to_openai_image_block.

Changes

  • Extract detail from top-level, extras, or metadata fields
  • Forward the detail field to the output image_url dict

Testing

Tested manually with all combinations:

  • URL with detail at top level ✅
  • URL with detail in extras ✅
  • URL with detail in metadata ✅
  • Base64 with detail ✅
  • No detail (backward compat) ✅
  • convert_to_openai_data_block for chat/completions API ✅
  • convert_to_openai_data_block for responses API ✅

Example

Before:

block = {'type': 'image', 'url': 'https://...', 'detail': 'high'}
convert_to_openai_image_block(block)
# {'type': 'image_url', 'image_url': {'url': 'https://...'}}  # detail lost!

After:
block = {'type': 'image', 'url': 'https://...', 'detail': 'high'}
convert_to_openai_image_block(block)
# {'type': 'image_url', 'image_url': {'url': 'https://...', 'detail': 'high'}}  # detail preserved!

## Changed files

- `libs/core/langchain_core/messages/block_translators/openai.py` (modified, +15/-6)


---

# PR #36469: fix(core): forward detail field in convert_to_openai_image_block

- Repository: langchain-ai/langchain
- Author: hanxujoy88
- State: closed | merged: False
- Link: https://github.com/langchain-ai/langchain/pull/36469

## Description (problem / solution / changelog)

## Summary

Fixes #36297

The `detail` field controls image analysis resolution (`low`, `high`, `auto`) in OpenAI API. Previously, this field was silently dropped when converting image blocks via `convert_to_openai_image_block`.

## Changes

- Extract `detail` from top-level, `extras`, or `metadata` fields
- Forward the `detail` field to the output `image_url` dict

## Testing

Tested manually with all combinations:
- URL with detail at top level ✅
- URL with detail in extras ✅  
- URL with detail in metadata ✅
- Base64 with detail ✅
- No detail (backward compat)- `convert_to_openai_data_block` for chat/completions API ✅
- `convert_to_openai_data_block` for responses API ✅

## Example

Before:
```python
block = {'type': 'image', 'url': 'https://...', 'detail': 'high'}
convert_to_openai_image_block(block)
# {'type': 'image_url', 'image_url': {'url': 'https://...'}}  # detail lost!

After:

block = {'type': 'image', 'url': 'https://...', 'detail': 'high'}
convert_to_openai_image_block(block)
# {'type': 'image_url', 'image_url': {'url': 'https://...', 'detail': 'high'}}  # detail preserved!

Changed files

  • libs/core/langchain_core/messages/block_translators/openai.py (modified, +15/-6)

PR #36505: fix(core): forward detail field in convert_to_openai_image_block

Description (problem / solution / changelog)

Summary

  • convert_to_openai_image_block was silently dropping the detail field when constructing the image_url dict for OpenAI. The detail parameter controls image analysis resolution ("low", "high", "auto") and is important for controlling token usage and analysis quality.
  • Added a _get_detail() helper that extracts detail from the block's top-level, extras, or metadata dicts.
  • The fix applies to both the Chat Completions API path ({"type": "image_url", "image_url": {"url": "...", "detail": "high"}}) and the Responses API path ({"type": "input_image", "image_url": "...", "detail": "high"}). The Responses API path in convert_to_openai_data_block already had code to forward detail from the intermediate chat completions block, but it was always empty before this fix.
  • Added comprehensive tests covering detail from all three source locations, both API paths, and the absence case.

Closes #36297

Test plan

  • Added test_convert_to_openai_data_block_detail_field covering:
    • Chat Completions: detail at top level (url and base64), in extras, in metadata, and absent
    • Responses API: detail at top level (url and base64), and absent
  • AST validation passes on both modified files

🤖 Generated with Claude Code

Changed files

  • libs/core/langchain_core/messages/block_translators/openai.py (modified, +24/-6)
  • libs/core/tests/unit_tests/messages/block_translators/test_openai.py (modified, +94/-0)

PR #36549: Fix convert_to_openai_image_block dropping detail field

Description (problem / solution / changelog)

Summary

Fixes #36297

convert_to_openai_image_block was dropping the detail field from image content blocks when converting to OpenAI format, for both URL and base64 source types.

Fix

Forward the detail field to the output dict when present in the input block.

Before

return {
    "type": "image_url",
    "image_url": {"url": block["url"]},
}

After

result = {
    "type": "image_url",
    "image_url": {"url": block["url"]},
}
if "detail" in block:
    result["image_url"]["detail"] = block["detail"]
return result

Same pattern applied to the base64 path.

Changed files

  • libs/core/langchain_core/messages/block_translators/openai.py (modified, +8/-2)

Code Example

"""Regression tests for detail field forwarding in OpenAI image block converters.

Run: pytest -vvvl test_image_detail.py
"""

import pytest

from langchain_core.messages.block_translators.openai import (
    convert_to_openai_data_block,
    convert_to_openai_image_block,
)


def _test_id(val: dict) -> str:
    """Generate readable test ID from input block."""
    if "detail" in val:
        loc = "top_level"
    elif "extras" in val:
        loc = "extras"
    elif "metadata" in val:
        loc = "metadata"
    else:
        loc = "no_detail"

    if "source_type" in val:
        src = f"source_type_{val['source_type']}"
    elif "base64" in val:
        src = "base64"
    elif "url" in val:
        src = "url"
    else:
        src = "unknown"

    return f"{src}-detail_in_{loc}"


# ============================================================
# convert_to_openai_image_block
# ============================================================

IMAGE_URL_INPUTS = [
    {"type": "image", "url": "https://...", "detail": "high"},
    {"type": "image", "url": "https://...", "extras": {"detail": "high"}},
    {"type": "image", "url": "https://...", "metadata": {"detail": "high"}},
    {
        "type": "image",
        "source_type": "url",
        "url": "https://...",
        "detail": "high",
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://...",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://...",
        "metadata": {"detail": "high"},
    },
]

EXPECTED_IMAGE_URL = {
    "type": "image_url",
    "image_url": {"url": "https://...", "detail": "high"},
}


@pytest.mark.parametrize("input_block", IMAGE_URL_INPUTS, ids=_test_id)
def test_convert_to_openai_image_block_url(input_block: dict) -> None:
    assert convert_to_openai_image_block(input_block) == EXPECTED_IMAGE_URL


IMAGE_BASE64_INPUTS = [
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/jpeg",
        "detail": "high",
    },
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/jpeg",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/jpeg",
        "metadata": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/jpeg",
        "detail": "high",
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/jpeg",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/jpeg",
        "metadata": {"detail": "high"},
    },
]

EXPECTED_IMAGE_BASE64 = {
    "type": "image_url",
    "image_url": {"url": "data:image/jpeg;base64,<b64>", "detail": "high"},
}


@pytest.mark.parametrize("input_block", IMAGE_BASE64_INPUTS, ids=_test_id)
def test_convert_to_openai_image_block_base64(input_block: dict) -> None:
    assert convert_to_openai_image_block(input_block) == EXPECTED_IMAGE_BASE64


# ============================================================
# convert_to_openai_data_block — Chat Completions API
# ============================================================

DATA_IMAGE_URL_INPUTS = [
    {"type": "image", "url": "https://example.com/img.png", "detail": "high"},
    {
        "type": "image",
        "url": "https://example.com/img.png",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "url": "https://example.com/img.png",
        "metadata": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://example.com/img.png",
        "detail": "high",
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://example.com/img.png",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://example.com/img.png",
        "metadata": {"detail": "high"},
    },
]

EXPECTED_DATA_CHAT_URL = {
    "type": "image_url",
    "image_url": {"url": "https://example.com/img.png", "detail": "high"},
}


@pytest.mark.parametrize("input_block", DATA_IMAGE_URL_INPUTS, ids=_test_id)
def test_convert_to_openai_data_block_chat_url(input_block: dict) -> None:
    assert convert_to_openai_data_block(input_block) == EXPECTED_DATA_CHAT_URL


DATA_IMAGE_BASE64_INPUTS = [
    {"type": "image", "base64": "<b64>", "mime_type": "image/png", "detail": "high"},
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/png",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/png",
        "metadata": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/png",
        "detail": "high",
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/png",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/png",
        "metadata": {"detail": "high"},
    },
]

EXPECTED_DATA_CHAT_BASE64 = {
    "type": "image_url",
    "image_url": {"url": "data:image/png;base64,<b64>", "detail": "high"},
}


@pytest.mark.parametrize("input_block", DATA_IMAGE_BASE64_INPUTS, ids=_test_id)
def test_convert_to_openai_data_block_chat_base64(input_block: dict) -> None:
    assert convert_to_openai_data_block(input_block) == EXPECTED_DATA_CHAT_BASE64


# ============================================================
# convert_to_openai_data_block — Responses API
# ============================================================

EXPECTED_DATA_RESPONSES_URL = {
    "type": "input_image",
    "image_url": "https://example.com/img.png",
    "detail": "high",
}


@pytest.mark.parametrize("input_block", DATA_IMAGE_URL_INPUTS, ids=_test_id)
def test_convert_to_openai_data_block_responses_url(input_block: dict) -> None:
    assert convert_to_openai_data_block(input_block, api="responses") == EXPECTED_DATA_RESPONSES_URL


EXPECTED_DATA_RESPONSES_BASE64 = {
    "type": "input_image",
    "image_url": "data:image/png;base64,<b64>",
    "detail": "high",
}


@pytest.mark.parametrize("input_block", DATA_IMAGE_BASE64_INPUTS, ids=_test_id)
def test_convert_to_openai_data_block_responses_base64(input_block: dict) -> None:
    assert convert_to_openai_data_block(input_block, api="responses") == EXPECTED_DATA_RESPONSES_BASE64

---



---

{"type": "image_url", "image_url": {"url": "https://...", "detail": "high"}}

---

{"type": "input_image", "image_url": "https://...", "detail": "high"}

---

{"type": "image_url", "image_url": {"url": "https://..."}}

---

{"type": "input_image", "image_url": "https://..."}
RAW_BUFFERClick to expand / collapse

Checked other resources

  • This is a bug, not a usage question.
  • I added a clear and descriptive title that summarizes this issue.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
  • This is not related to the langchain-community package.
  • I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Package (Required)

  • langchain
  • langchain-openai
  • langchain-anthropic
  • langchain-classic
  • langchain-core
  • langchain-model-profiles
  • langchain-tests
  • langchain-text-splitters
  • langchain-chroma
  • langchain-deepseek
  • langchain-exa
  • langchain-fireworks
  • langchain-groq
  • langchain-huggingface
  • langchain-mistralai
  • langchain-nomic
  • langchain-ollama
  • langchain-openrouter
  • langchain-perplexity
  • langchain-qdrant
  • langchain-xai
  • Other / not sure / general

Related Issues / PRs

No response

Reproduction Steps / Example Code (Python)

"""Regression tests for detail field forwarding in OpenAI image block converters.

Run: pytest -vvvl test_image_detail.py
"""

import pytest

from langchain_core.messages.block_translators.openai import (
    convert_to_openai_data_block,
    convert_to_openai_image_block,
)


def _test_id(val: dict) -> str:
    """Generate readable test ID from input block."""
    if "detail" in val:
        loc = "top_level"
    elif "extras" in val:
        loc = "extras"
    elif "metadata" in val:
        loc = "metadata"
    else:
        loc = "no_detail"

    if "source_type" in val:
        src = f"source_type_{val['source_type']}"
    elif "base64" in val:
        src = "base64"
    elif "url" in val:
        src = "url"
    else:
        src = "unknown"

    return f"{src}-detail_in_{loc}"


# ============================================================
# convert_to_openai_image_block
# ============================================================

IMAGE_URL_INPUTS = [
    {"type": "image", "url": "https://...", "detail": "high"},
    {"type": "image", "url": "https://...", "extras": {"detail": "high"}},
    {"type": "image", "url": "https://...", "metadata": {"detail": "high"}},
    {
        "type": "image",
        "source_type": "url",
        "url": "https://...",
        "detail": "high",
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://...",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://...",
        "metadata": {"detail": "high"},
    },
]

EXPECTED_IMAGE_URL = {
    "type": "image_url",
    "image_url": {"url": "https://...", "detail": "high"},
}


@pytest.mark.parametrize("input_block", IMAGE_URL_INPUTS, ids=_test_id)
def test_convert_to_openai_image_block_url(input_block: dict) -> None:
    assert convert_to_openai_image_block(input_block) == EXPECTED_IMAGE_URL


IMAGE_BASE64_INPUTS = [
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/jpeg",
        "detail": "high",
    },
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/jpeg",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/jpeg",
        "metadata": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/jpeg",
        "detail": "high",
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/jpeg",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/jpeg",
        "metadata": {"detail": "high"},
    },
]

EXPECTED_IMAGE_BASE64 = {
    "type": "image_url",
    "image_url": {"url": "data:image/jpeg;base64,<b64>", "detail": "high"},
}


@pytest.mark.parametrize("input_block", IMAGE_BASE64_INPUTS, ids=_test_id)
def test_convert_to_openai_image_block_base64(input_block: dict) -> None:
    assert convert_to_openai_image_block(input_block) == EXPECTED_IMAGE_BASE64


# ============================================================
# convert_to_openai_data_block — Chat Completions API
# ============================================================

DATA_IMAGE_URL_INPUTS = [
    {"type": "image", "url": "https://example.com/img.png", "detail": "high"},
    {
        "type": "image",
        "url": "https://example.com/img.png",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "url": "https://example.com/img.png",
        "metadata": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://example.com/img.png",
        "detail": "high",
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://example.com/img.png",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "url",
        "url": "https://example.com/img.png",
        "metadata": {"detail": "high"},
    },
]

EXPECTED_DATA_CHAT_URL = {
    "type": "image_url",
    "image_url": {"url": "https://example.com/img.png", "detail": "high"},
}


@pytest.mark.parametrize("input_block", DATA_IMAGE_URL_INPUTS, ids=_test_id)
def test_convert_to_openai_data_block_chat_url(input_block: dict) -> None:
    assert convert_to_openai_data_block(input_block) == EXPECTED_DATA_CHAT_URL


DATA_IMAGE_BASE64_INPUTS = [
    {"type": "image", "base64": "<b64>", "mime_type": "image/png", "detail": "high"},
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/png",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "base64": "<b64>",
        "mime_type": "image/png",
        "metadata": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/png",
        "detail": "high",
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/png",
        "extras": {"detail": "high"},
    },
    {
        "type": "image",
        "source_type": "base64",
        "data": "<b64>",
        "mime_type": "image/png",
        "metadata": {"detail": "high"},
    },
]

EXPECTED_DATA_CHAT_BASE64 = {
    "type": "image_url",
    "image_url": {"url": "data:image/png;base64,<b64>", "detail": "high"},
}


@pytest.mark.parametrize("input_block", DATA_IMAGE_BASE64_INPUTS, ids=_test_id)
def test_convert_to_openai_data_block_chat_base64(input_block: dict) -> None:
    assert convert_to_openai_data_block(input_block) == EXPECTED_DATA_CHAT_BASE64


# ============================================================
# convert_to_openai_data_block — Responses API
# ============================================================

EXPECTED_DATA_RESPONSES_URL = {
    "type": "input_image",
    "image_url": "https://example.com/img.png",
    "detail": "high",
}


@pytest.mark.parametrize("input_block", DATA_IMAGE_URL_INPUTS, ids=_test_id)
def test_convert_to_openai_data_block_responses_url(input_block: dict) -> None:
    assert convert_to_openai_data_block(input_block, api="responses") == EXPECTED_DATA_RESPONSES_URL


EXPECTED_DATA_RESPONSES_BASE64 = {
    "type": "input_image",
    "image_url": "data:image/png;base64,<b64>",
    "detail": "high",
}


@pytest.mark.parametrize("input_block", DATA_IMAGE_BASE64_INPUTS, ids=_test_id)
def test_convert_to_openai_data_block_responses_base64(input_block: dict) -> None:
    assert convert_to_openai_data_block(input_block, api="responses") == EXPECTED_DATA_RESPONSES_BASE64

Error Message and Stack Trace (if applicable)

Description

Summary

convert_to_openai_image_block does not detect or forward the optional detail field from ImageContentBlock / Base64ContentBlock inputs. This means the detail parameter (which controls image analysis resolution — "low", "high", or "auto") is silently lost when formatting blocks for both the OpenAI Chat Completions and Responses APIs:

Root Cause

convert_to_openai_image_block constructs the image_url dict with only the url key, ignoring any detail field present in the input block: https://github.com/langchain-ai/langchain/blob/2aeeb58ef11e74ab98f869d6ae5c2ae04721be06/libs/core/langchain_core/messages/block_translators/openai.py#L22-L55

The downstream convert_to_openai_data_block calls convert_to_openai_image_block and then checks chat_completions_block["image_url"].get("detail") to forward detail to the Responses API format. Since convert_to_openai_image_block never includes detail, this check always evaluates to False. In the end both APIs affected: https://github.com/langchain-ai/langchain/blob/2aeeb58ef11e74ab98f869d6ae5c2ae04721be06/libs/core/langchain_core/messages/block_translators/openai.py#L58-L90

Expected Behavior

Models don't always analyze original images. Sometimes, images are resized based on their size or the optional detail field before analysis. If present, the optional detail field must be forwarded. When a detail field is supplied (at the top level, or nested in extras/metadata for backwards compatibility), the formatted output should include it:

Chat Completions API:

{"type": "image_url", "image_url": {"url": "https://...", "detail": "high"}}

Responses API:

{"type": "input_image", "image_url": "https://...", "detail": "high"}

Actual Behavior

The detail field is dropped in all cases:

Chat Completions API:

{"type": "image_url", "image_url": {"url": "https://..."}}

Responses API:

{"type": "input_image", "image_url": "https://..."}

Test Coverage Gap

Existing tests for both test_convert_to_openai_image_block and test_convert_to_openai_data_block do not include any inputs with a detail field, so this behavior was never caught.

System Info

System Information

OS: Linux OS Version: #1 SMP Wed Sep 13 01:36:35 PDT 2023 Python Version: 3.12.3 (main, May 18 2024, 09:22:28) [GCC 14.1.1 20240516 releases/gcc-14.1.0-39-g82e4bdc5c3]

Package Information

langchain_core: 1.2.22 langchain: 1.2.13 langsmith: 0.7.22 langchain_openai: 1.1.12 langgraph_sdk: 0.3.12

Optional packages not installed

deepagents deepagents-cli

Other Dependencies

httpx: 0.28.1 jsonpatch: 1.33 langgraph: 1.1.3 openai: 2.30.0 opentelemetry-api: 1.39.1 opentelemetry-exporter-otlp-proto-http: 1.39.1 opentelemetry-sdk: 1.39.1 orjson: 3.11.7 packaging: 26.0 pydantic: 2.12.5 pytest: 9.0.2 pyyaml: 6.0.3 requests: 2.33.0 requests-toolbelt: 1.0.0 rich: 14.3.3 tenacity: 9.1.4 tiktoken: 0.12.0 typing-extensions: 4.15.0 uuid-utils: 0.14.1 websockets: 16.0 wrapt: 2.1.1 xxhash: 3.6.0 zstandard: 0.25.0

extent analysis

Fix Plan

To fix the issue, we need to modify the convert_to_openai_image_block function to include the detail field in the image_url dictionary. We can do this by checking if the detail field is present in the input block and adding it to the image_url dictionary if it is.

Here are the steps to fix the issue:

  • Modify the convert_to_openai_image_block function to include the detail field in the image_url dictionary.
  • Update the convert_to_openai_data_block function to handle the new image_url dictionary format.

Here is an example of the modified code:

def convert_to_openai_image_block(input_block: dict) -> dict:
    # ... (rest of the function remains the same)

    image_url = {"url": url}
    if "detail" in input_block:
        image_url["detail"] = input_block["detail"]
    elif "extras" in input_block and "detail" in input_block["extras"]:
        image_url["detail"] = input_block["extras"]["detail"]
    elif "metadata" in input_block and "detail" in input_block["metadata"]:
        image_url["detail"] = input_block["metadata"]["detail"]

    return {"type": "image_url", "image_url": image_url}

Verification

To verify that the fix worked, we can run the existing tests for convert_to_openai_image_block and convert_to_openai_data_block with the modified code. We should also add new tests to cover the cases where the detail field is present in the input block.

Here is an example of a new test:

def test_convert_to_openai_image_block_with_detail():
    input_block = {"type": "image", "url": "https://...", "detail": "high"}
    expected_output = {"type": "image_url", "image_url": {"url": "https://...", "detail": "high"}}
    assert convert_to_openai_image_block(input_block) == expected_output

Extra Tips

  • Make sure to update the documentation for the convert_to_openai_image_block and convert_to_openai_data_block functions to reflect the new behavior.
  • Consider adding a deprecation warning for the old behavior to ensure that users are aware of the change.
  • Make sure to test the modified code thoroughly to ensure that it works as expected in all scenarios.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING