Issue Station
Sign In
Tools hereToolbox

langchain - ✅(Solved) Fix langchain + xinference llm.with_structured_output Probabilistic error reporting occurs under high concurrency [2 pull requests, 4 comments, 3 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
langchain-ai/langchain#35226•Fetched 2026-04-08 00:27:06
View on GitHub
Comments
4
Participants
3
Timeline
15
Reactions
0
Author
berserker3912berserker3912
Participants
berserker3912berserker3912Godson-OziomaGodson-Oziomasamuraikillerssamuraikillers
Timeline (top)
commented ×3labeled ×3mentioned ×3subscribed ×3

langchain + xinference llm.with_structured_output Probabilistic error reporting occurs under high concurrency

Error Message

Traceback (most recent call last): File "/xxxx/xxxx.py", line 130, in classification_execution final_answer = structured_llm.invoke(messages) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/xxxx/site-packages/langchain_core/runnables/base.py", line 3149, in invoke input_ = context.run(step.invoke, input_, config, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/xxxx/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 402, in invoke self.generate_prompt( File "/xxxx/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 1121, in generate_prompt return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/xxxx/site-packages/langchain_core/language_models/chat_models.py", line 931, in generate self._generate_with_cache( File "/xxxx/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 1225, in _generate_with_cache result = self._generate( ^^^^^^^^^^^^^^^ File "/xxxx/lib/python3.12/site-packages/langchain_xinference/chat_models.py", line 226, in _generate final_chunk = self._chat_with_aggregation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/xxxx/lib/python3.12/site-packages/langchain_xinference/chat_models.py", line 310, in _chat_with_aggregation for stream_resp in response: ^^^^^^^^ File "/xxxx/lib/python3.12/site-packages/xinference/client/common.py", line 62, in streaming_response_iterator raise Exception(str(error)) Exception: [address=127.0.0.1:39175, pid=167075] Expecting ',' delimiter: line 3 column 1 (char 120)

Root Cause

langchain + xinference llm.with_structured_output Probabilistic error reporting occurs under high concurrency

Fix Action

Fix / Workaround

  • This is a bug, not a usage question.
  • I added a clear and descriptive title that summarizes this issue.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
  • This is not related to the langchain-community package.
  • I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

PR fix notes

PR #35241: Fixes/35226

Description (problem / solution / changelog)

Description

This PR fixes intermittent JSON parsing failures in with_structured_output() when used under high concurrency with streaming backends like Xinference and vLLM. The issue manifested as JSONDecodeError exceptions during streaming aggregation despite valid model output.

Fixes #35226

AI Contribution Disclaimer: This contribution was developed with assistance from an AI coding agent to analyze the codebase, identify root causes, and implement the fix.

Changes Root Cause The streaming output parsers lacked clear documentation about their concurrency safety guarantees, and error messages did not provide sufficient context to diagnose concurrency-related failures versus other issues.

Solution Enhanced documentation in BaseCumulativeTransformOutputParser to explicitly document per-invocation state isolation Improved error messages in parse_tool_call() and ProviderStrategyBinding.parse() to include detailed context and guidance for debugging concurrency issues Added input validation to detect empty or incomplete content before JSON parsing Files Modified transform.py : Added concurrency safety documentation openai_tools.py : Enhanced error messages and input validation structured_output.py : Improved error handling with detailed context Tests Added test_structured_output_concurrency.py : 9 tests validating concurrent parsing isolation test_streaming_concurrency.py : 9 tests for streaming parser concurrency safety All tests verify that concurrent invocations (10-20 parallel operations) maintain proper isolation without data corruption.

Breaking Changes None. All existing APIs are preserved.

Verification Ran make format, make lint, and make test in both libs/core and libs/langchain_v1 All 18 new concurrency tests pass All existing tests continue to pass Manually tested concurrent structured output parsing with simulated high-load scenarios The fix ensures deterministic behavior under concurrency by documenting existing state isolation guarantees and providing actionable error messages when failures occur.

Changed files

  • .vscode/settings.json (modified, +2/-1)
  • libs/core/langchain_core/output_parsers/openai_tools.py (modified, +18/-4)
  • libs/core/langchain_core/output_parsers/transform.py (modified, +36/-1)
  • libs/core/tests/unit_tests/output_parsers/test_streaming_concurrency.py (added, +374/-0)
  • libs/core/uv.lock (modified, +1/-1)
  • libs/langchain_v1/langchain/agents/structured_output.py (modified, +28/-2)
  • libs/langchain_v1/tests/unit_tests/agents/test_responses.py (modified, +1/-1)
  • libs/langchain_v1/tests/unit_tests/agents/test_structured_output_concurrency.py (added, +251/-0)
  • libs/langchain_v1/uv.lock (modified, +2/-2)

PR #35243: fix(anthropic): pass metadata.user_id from config to API requests

Description (problem / solution / changelog)

Description Fixes metadata.user_id being dropped when passed via config to ChatAnthropic and AnthropicLLM, preventing prompt caching and user tracking with third-party Claude providers.

Fixes #35226

Changes Added extract_metadata_from_run_manager() helper in _client_utils.py to extract user_id from run manager metadata Updated ChatAnthropic._generate() and _agenerate() to pass metadata to API requests Updated AnthropicLLM._call() and _acall() with same metadata extraction logic Added 10 unit tests covering both sync and async operations Breaking Changes None. This is a bug fix that enables existing functionality.

Verification All 10 new unit tests pass All 15 existing Anthropic tests pass Ran make format, make lint, and make test successfully Manually verified metadata is correctly passed to Anthropic API Technical Details The issue occurred because _generate(), _agenerate(), _call(), and _acall() methods received run_manager with metadata but never extracted it. The Anthropic API expects a metadata field with user_id for features like prompt caching. This fix extracts user_id from run_manager.metadata and includes it in the API request payload.

AI Contribution Disclaimer: This PR was developed with assistance from an AI coding assistant. All code has been reviewed, tested, and verified to meet LangChain standards.

Changed files

  • libs/partners/anthropic/langchain_anthropic/_client_utils.py (modified, +29/-0)
  • libs/partners/anthropic/langchain_anthropic/chat_models.py (modified, +9/-0)
  • libs/partners/anthropic/langchain_anthropic/llms.py (modified, +60/-4)
  • libs/partners/anthropic/tests/unit_tests/test_llms_metadata.py (added, +42/-0)
  • libs/partners/anthropic/tests/unit_tests/test_metadata_user_id.py (added, +174/-0)
  • libs/partners/anthropic/uv.lock (modified, +1/-1)

Code Example

vllm 0.11.0 xinference 1.17.0 langchain 1.2.6 qwen3-install-30b

class DicInfo(BaseModel):
"""xxxx"""
category: StateType = Field(description="xxxx")
confidence: Optional[float] = Field(
default=0.0,
ge=0.0,
le=1.0,
description="xxxx"
)
detail_categories: Optional[List[str]] = Field(
default_factory=list,
description="xxxx"
)

structured_llm = llm.with_structured_output(DicInfo)
messages = [SystemMessage(content=system_prompt), HumanMessage(content=f"{user_input}")]
final_answer = structured_llm.invoke(messages)

---

Traceback (most recent call last):
File "/xxxx/xxxx.py", line 130, in classification_execution
final_answer = structured_llm.invoke(messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxxx/site-packages/langchain_core/runnables/base.py", line 3149, in invoke
input_ = context.run(step.invoke, input_, config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxxx/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 402, in invoke
self.generate_prompt(
File "/xxxx/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 1121, in generate_prompt
return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxxx/site-packages/langchain_core/language_models/chat_models.py", line 931, in generate
self._generate_with_cache(
File "/xxxx/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 1225, in _generate_with_cache
result = self._generate(
^^^^^^^^^^^^^^^
File "/xxxx/lib/python3.12/site-packages/langchain_xinference/chat_models.py", line 226, in _generate
final_chunk = self._chat_with_aggregation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxxx/lib/python3.12/site-packages/langchain_xinference/chat_models.py", line 310, in _chat_with_aggregation
for stream_resp in response:
^^^^^^^^
File "/xxxx/lib/python3.12/site-packages/xinference/client/common.py", line 62, in streaming_response_iterator
raise Exception(str(error))
Exception: [address=127.0.0.1:39175, pid=167075] Expecting ',' delimiter: line 3 column 1 (char 120)
RAW_BUFFERClick to expand / collapse

Checked other resources

  • This is a bug, not a usage question.
  • I added a clear and descriptive title that summarizes this issue.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
  • This is not related to the langchain-community package.
  • I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Package (Required)

  • langchain
  • langchain-openai
  • langchain-anthropic
  • langchain-classic
  • langchain-core
  • langchain-model-profiles
  • langchain-tests
  • langchain-text-splitters
  • langchain-chroma
  • langchain-deepseek
  • langchain-exa
  • langchain-fireworks
  • langchain-groq
  • langchain-huggingface
  • langchain-mistralai
  • langchain-nomic
  • langchain-ollama
  • langchain-perplexity
  • langchain-qdrant
  • langchain-xai
  • Other / not sure / general

Related Issues / PRs

langchain + xinference llm.with_structured_output Probabilistic error reporting occurs under high concurrency

Reproduction Steps / Example Code (Python)

vllm 0.11.0 xinference 1.17.0 langchain 1.2.6 qwen3-install-30b

class DicInfo(BaseModel):
"""xxxx"""
category: StateType = Field(description="xxxx")
confidence: Optional[float] = Field(
default=0.0,
ge=0.0,
le=1.0,
description="xxxx"
)
detail_categories: Optional[List[str]] = Field(
default_factory=list,
description="xxxx"
)

structured_llm = llm.with_structured_output(DicInfo)
messages = [SystemMessage(content=system_prompt), HumanMessage(content=f"{user_input}")]
final_answer = structured_llm.invoke(messages)

Error Message and Stack Trace (if applicable)

Traceback (most recent call last):
File "/xxxx/xxxx.py", line 130, in classification_execution
final_answer = structured_llm.invoke(messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxxx/site-packages/langchain_core/runnables/base.py", line 3149, in invoke
input_ = context.run(step.invoke, input_, config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxxx/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 402, in invoke
self.generate_prompt(
File "/xxxx/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 1121, in generate_prompt
return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxxx/site-packages/langchain_core/language_models/chat_models.py", line 931, in generate
self._generate_with_cache(
File "/xxxx/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 1225, in _generate_with_cache
result = self._generate(
^^^^^^^^^^^^^^^
File "/xxxx/lib/python3.12/site-packages/langchain_xinference/chat_models.py", line 226, in _generate
final_chunk = self._chat_with_aggregation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxxx/lib/python3.12/site-packages/langchain_xinference/chat_models.py", line 310, in _chat_with_aggregation
for stream_resp in response:
^^^^^^^^
File "/xxxx/lib/python3.12/site-packages/xinference/client/common.py", line 62, in streaming_response_iterator
raise Exception(str(error))
Exception: [address=127.0.0.1:39175, pid=167075] Expecting ',' delimiter: line 3 column 1 (char 120)

Description

langchain + xinference llm.with_structured_output Probabilistic error reporting occurs under high concurrency

System Info

vllm 0.11.0 xinference 1.17.0 langchain 1.2.6 qwen3-install-30b

extent analysis

Fix Plan

1. Update xinference to the latest version

The error message suggests that there is a parsing issue with the xinference response. This might be due to a bug in the xinference library. Try updating xinference to the latest version.

pip install --upgrade xinference

2. Use a try-except block to catch and handle the parsing error

Even after updating xinference, the parsing error might still occur. To handle this, wrap the code that invokes the xinference model in a try-except block.

try:
    final_answer = structured_llm.invoke(messages)
except Exception as e:
    if "Expecting ',' delimiter" in str(e):
        # Handle the parsing error
        print("Error parsing xinference response:", e)
        # You can also retry the invocation or return a default value
    else:
        raise

3. Check the xinference response for any issues

The error message suggests that there is an issue with the xinference response. Check the response for any errors or issues that might be causing the parsing error.

response = structured_llm.invoke(messages)
if response.status_code != 200:
    print("Error:", response.text)

Verification

To verify that the fix worked, you can try running the code again with high concurrency and check if the error occurs. If the error does not occur, the fix is successful.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#agent setup#task chaining#parallel task

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

langchain - ✅(Solved) Fix langchain + xinference llm.with_structured_output Probabilistic error reporting occurs under high concurrency [2 pull requests, 4 comments, 3 participants]