langchain - ✅(Solved) Fix [langchain] trace=False on agent model node in factory.py prevents callback setup [4 pull requests, 4 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
langchain-ai/langchain#36395Fetched 2026-04-08 01:57:55
View on GitHub
Comments
4
Participants
2
Timeline
22
Reactions
0
Author
Timeline (top)
commented ×4cross-referenced ×4mentioned ×4subscribed ×4

Error Message

Error Message and Stack Trace (if applicable)

No error is thrown - this is a silent failure where callbacks are never set up.

Root Cause

print(f"\nWith create_react_agent, events: {handler.events}") if "on_llm_start" not in handler.events: print("❌ CONFIRMED: Callbacks don't reach LLM calls in agent model node") print(" This is because trace=False bypasses callback setup in RunnableCallable")

Fix Action

Fix / Workaround

  • This is a bug, not a usage question.
  • I added a clear and descriptive title that summarizes this issue.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
  • This is not related to the langchain-community package.
  • I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

PR fix notes

PR #36402: fix(langchain): trace create_agent model node callbacks

Description (problem / solution / changelog)

Fixes #36395

Enable tracing on the create_agent model node by removing trace=False, so callback hierarchy includes model_node and downstream model runs inherit callback context correctly. Add sync/async regression tests that assert chat model callbacks are parented under model_node.

No breaking changes.

How did you verify your code works?

  • cd libs/langchain_v1 && uv run --group test pytest tests/unit_tests/agents/middleware/core/test_wrap_model_call.py -q
  • cd libs/langchain_v1 && make test_fast
  • cd libs/langchain_v1 && make lint

This PR was prepared with assistance from an AI coding agent; the final diff and validations were reviewed before submission.

Changed files

  • libs/langchain_v1/langchain/agents/factory.py (modified, +1/-1)
  • libs/langchain_v1/tests/unit_tests/agents/middleware/core/test_wrap_model_call.py (modified, +87/-0)

PR #36403: fix(langchain): trace create_agent model node callbacks

Description (problem / solution / changelog)

Fixes #36395

Enable tracing on the create_agent model node by removing trace=False so callback hierarchy includes model_node. Add sync and async regression tests that verify chat model callbacks are parented under model_node.

No breaking changes.

How did you verify your code works?

  • cd libs/langchain_v1 && uv run --group test pytest tests/unit_tests/agents/middleware/core/test_wrap_model_call.py -q
  • cd libs/langchain_v1 && make test_fast
  • cd libs/langchain_v1 && make lint

This PR was prepared with assistance from an AI coding agent; the final diff and validations were reviewed before submission.

Changed files

  • libs/langchain_v1/langchain/agents/factory.py (modified, +1/-1)
  • libs/langchain_v1/tests/unit_tests/agents/middleware/core/test_wrap_model_call.py (modified, +87/-0)

PR #1: test(langchain): add callback propagation test for agent model nodes

Description (problem / solution / changelog)

Fixes #36395

Summary

Adds a unit test to verify that callbacks propagate correctly through agent model nodes.

Changes

  • Added async unit test for callback propagation in create_react_agent
  • Ensures on_llm_start is triggered during agent execution

Why this matters

Prevents regressions where callback chains silently break, affecting observability tools like LangSmith, Langfuse, and OpenTelemetry.

Verification

  • Test confirms on_llm_start is triggered during agent execution

Changed files

  • libs/langchain/tests/unit_tests/test_callback_propagation.py (added, +43/-0)

PR #36446: fix(agents): enable tracing for model node to support callbacks

Description (problem / solution / changelog)

Fixes #36395

The agent model node was using trace=False, which prevented callback handlers from being properly set up in RunnableCallable.ainvoke().

This caused on_llm_start and other callbacks to be skipped entirely, breaking observability tools like:

  • Langfuse
  • LangSmith
  • OpenTelemetry
  • Custom callback handlers

Changed trace=False to trace=True to allow proper callback propagation to LLM calls within agent workflows.

Verification

  • Syntax valid (Python compiles)
  • 1-line change only
  • No breaking changes (trace=True is the default)
  • Fixes real production issue (callbacks broken in agents)

Changed files

  • libs/langchain_v1/langchain/agents/factory.py (modified, +1/-1)

Code Example

"""
Minimal reproducible example showing trace=False prevents callback setup.

When trace=False is set on a RunnableCallable, the on_llm_start callback
is never triggered, breaking Langfuse, OpenTelemetry, and other observability.
"""
import asyncio
from typing import Any, Optional
from langchain.agents import create_react_agent
from langchain_core.callbacks import BaseCallbackHandler
from langchain_core.callbacks.manager import AsyncCallbackManager
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.messages import AIMessage
from langchain_core.runnables import RunnableCallable


class TestCallbackHandler(BaseCallbackHandler):
    """Test handler that logs all LLM starts to verify callback propagation."""
    
    def __init__(self):
        self.events: list[str] = []
    
    def on_llm_start(self, serialized: dict[str, Any], prompts: list[str], **kwargs) -> None:
        self.events.append("on_llm_start")
        print(f"[CALLBACK] on_llm_start called")
    
    def on_chain_start(self, serialized: dict[str, Any], inputs: dict, **kwargs) -> None:
        self.events.append("on_chain_start")
        print(f"[CALLBACK] on_chain_start called: {serialized.get('name', 'unknown')}")


async def demonstrate_trace_false_bug():
    """
    Demonstrates that trace=False prevents callback setup in RunnableCallable.
    
    This is exactly what happens in langchain/agents/factory.py line 1327:
    
    node = RunnableCallable(
        func=amodel_node,
        name="model_node", 
        trace=False,  # <-- THIS IS THE BUG
    )
    """
    
    async def dummy_model_func(input_data: dict, config: Optional[Any] = None) -> AIMessage:
        """Simulates what model_.ainvoke does internally."""
        print(f"[MODEL] Executing with config={config}")
        return AIMessage(content="test")
    
    handler = TestCallbackHandler()
    
    # Test 1: trace=False (current buggy behavior in factory.py)
    print("=" * 60)
    print("Test 1: trace=False (CURRENT BUG in factory.py line 1327)")
    print("=" * 60)
    
    buggy_node = RunnableCallable(
        func=dummy_model_func,
        name="model_node_trace_false",
        trace=False,  # This is what's in factory.py currently
    )
    
    result1 = await buggy_node.ainvoke(
        {"messages": [("human", "Hello")]},
        config={"callbacks": [handler]}
    )
    
    print(f"Events captured with trace=False: {handler.events}")
    if "on_llm_start" not in handler.events:
        print("❌ BUG: on_llm_start was NOT called - callbacks not set up!")
    
    handler.events.clear()
    
    # Test 2: trace=True (or omitted, defaults to True) - fixed behavior
    print("\n" + "=" * 60)
    print("Test 2: trace=True (or omitted - defaults to True, THE FIX)")
    print("=" * 60)
    
    fixed_node = RunnableCallable(
        func=dummy_model_func,
        name="model_node_trace_true",
        trace=True,  # This is the fix
    )
    
    result2 = await fixed_node.ainvoke(
        {"messages": [("human", "Hello")]},
        config={"callbacks": [handler]}
    )
    
    print(f"Events captured with trace=True: {handler.events}")
    if "on_llm_start" in handler.events:
        print("✅ FIXED: on_llm_start was called - callbacks work!")
    
    print("\n" + "=" * 60)
    print("CONCLUSION: trace=False prevents callback setup in RunnableCallable")
    print("=" * 60)


async def main():
    await demonstrate_trace_false_bug()
    
    # Show the actual impact with create_react_agent
    print("\n" + "=" * 60)
    print("Actual impact with create_react_agent:")
    print("=" * 60)
    
    from langchain.agents import create_react_agent
    
    class MockModel:
        async def ainvoke(self, messages, config=None, **kwargs):
            return AIMessage(content="test")
        def bind_tools(self, tools, **kwargs):
            return self
    
    model = MockModel()
    tools = []
    prompt = ChatPromptTemplate.from_messages([
        ("system", "You are a helpful assistant."),
        ("placeholder", "{messages}"),
    ])
    
    agent = create_react_agent(model, tools, prompt=prompt)
    
    handler = TestCallbackHandler()
    result = await agent.ainvoke(
        {"messages": [("human", "Hello")]},
        config={"callbacks": [handler]}
    )
    
    print(f"\nWith create_react_agent, events: {handler.events}")
    if "on_llm_start" not in handler.events:
        print("❌ CONFIRMED: Callbacks don't reach LLM calls in agent model node")
        print("   This is because trace=False bypasses callback setup in RunnableCallable")


if __name__ == "__main__":
    asyncio.run(main())

---

No error is thrown - this is a silent failure where callbacks are never set up.

The code path that's skipped (from `langchain_core/runnables/_runnable.py:410-429`):

# When trace=True (or omitted), this code runs:
if self.trace:
    await run_manager.on_llm_start(...)  # Callbacks set up here
    
# When trace=False, this is SKIPPED entirely:
else:
    # No callback setup happens!
    response = await self.afunc(*args, **kwargs)

---

node = RunnableCallable(
    func=amodel_node,
    name="model_node",
    trace=False,  # <-- This is the bug
)

---

async def ainvoke(self, input: Input, config: Optional[RunnableConfig] = None, **kwargs: Any) -> Output:
    if self.trace:
        # Callback setup happens here (this is skipped when trace=False)
        callback_manager = AsyncCallbackManager.configure(...)
        run_manager = await callback_manager.on_llm_start(...)
    else:
        # trace=False path - NO callback setup!
        response = await self.afunc(input, config=config, **kwargs)

---

System Information
------------------
> OS:  Linux/Mac/Windows
> Python:  3.12+

Package Information
-------------------
> langchain: 0.3.x
> langchain-core: 0.3.x
> langgraph: 0.2.x

Other Dependencies:
> langfuse: 2.x/3.x (for tracing reproduction)
RAW_BUFFERClick to expand / collapse

Checked other resources

  • This is a bug, not a usage question.
  • I added a clear and descriptive title that summarizes this issue.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
  • This is not related to the langchain-community package.
  • I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Package (Required)

  • langchain
  • langchain-openai
  • langchain-anthropic
  • langchain-classic
  • langchain-core
  • langchain-model-profiles
  • langchain-tests
  • langchain-text-splitters
  • langchain-chroma
  • langchain-deepseek
  • langchain-exa
  • langchain-fireworks
  • langchain-groq
  • langchain-huggingface
  • langchain-mistralai
  • langchain-nomic
  • langchain-ollama
  • langchain-openrouter
  • langchain-perplexity
  • langchain-qdrant
  • langchain-xai
  • Other / not sure / general

Related Issues / PRs

DeepAgents #2315 - Related callback propagation issue Langfuse #10721 - Nested traces broken (this issue contributes to that problem) https://github.com/langchain-ai/langchain/issues/30901 (config propagation in agent nodes)

Reproduction Steps / Example Code (Python)

"""
Minimal reproducible example showing trace=False prevents callback setup.

When trace=False is set on a RunnableCallable, the on_llm_start callback
is never triggered, breaking Langfuse, OpenTelemetry, and other observability.
"""
import asyncio
from typing import Any, Optional
from langchain.agents import create_react_agent
from langchain_core.callbacks import BaseCallbackHandler
from langchain_core.callbacks.manager import AsyncCallbackManager
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.messages import AIMessage
from langchain_core.runnables import RunnableCallable


class TestCallbackHandler(BaseCallbackHandler):
    """Test handler that logs all LLM starts to verify callback propagation."""
    
    def __init__(self):
        self.events: list[str] = []
    
    def on_llm_start(self, serialized: dict[str, Any], prompts: list[str], **kwargs) -> None:
        self.events.append("on_llm_start")
        print(f"[CALLBACK] on_llm_start called")
    
    def on_chain_start(self, serialized: dict[str, Any], inputs: dict, **kwargs) -> None:
        self.events.append("on_chain_start")
        print(f"[CALLBACK] on_chain_start called: {serialized.get('name', 'unknown')}")


async def demonstrate_trace_false_bug():
    """
    Demonstrates that trace=False prevents callback setup in RunnableCallable.
    
    This is exactly what happens in langchain/agents/factory.py line 1327:
    
    node = RunnableCallable(
        func=amodel_node,
        name="model_node", 
        trace=False,  # <-- THIS IS THE BUG
    )
    """
    
    async def dummy_model_func(input_data: dict, config: Optional[Any] = None) -> AIMessage:
        """Simulates what model_.ainvoke does internally."""
        print(f"[MODEL] Executing with config={config}")
        return AIMessage(content="test")
    
    handler = TestCallbackHandler()
    
    # Test 1: trace=False (current buggy behavior in factory.py)
    print("=" * 60)
    print("Test 1: trace=False (CURRENT BUG in factory.py line 1327)")
    print("=" * 60)
    
    buggy_node = RunnableCallable(
        func=dummy_model_func,
        name="model_node_trace_false",
        trace=False,  # This is what's in factory.py currently
    )
    
    result1 = await buggy_node.ainvoke(
        {"messages": [("human", "Hello")]},
        config={"callbacks": [handler]}
    )
    
    print(f"Events captured with trace=False: {handler.events}")
    if "on_llm_start" not in handler.events:
        print("❌ BUG: on_llm_start was NOT called - callbacks not set up!")
    
    handler.events.clear()
    
    # Test 2: trace=True (or omitted, defaults to True) - fixed behavior
    print("\n" + "=" * 60)
    print("Test 2: trace=True (or omitted - defaults to True, THE FIX)")
    print("=" * 60)
    
    fixed_node = RunnableCallable(
        func=dummy_model_func,
        name="model_node_trace_true",
        trace=True,  # This is the fix
    )
    
    result2 = await fixed_node.ainvoke(
        {"messages": [("human", "Hello")]},
        config={"callbacks": [handler]}
    )
    
    print(f"Events captured with trace=True: {handler.events}")
    if "on_llm_start" in handler.events:
        print("✅ FIXED: on_llm_start was called - callbacks work!")
    
    print("\n" + "=" * 60)
    print("CONCLUSION: trace=False prevents callback setup in RunnableCallable")
    print("=" * 60)


async def main():
    await demonstrate_trace_false_bug()
    
    # Show the actual impact with create_react_agent
    print("\n" + "=" * 60)
    print("Actual impact with create_react_agent:")
    print("=" * 60)
    
    from langchain.agents import create_react_agent
    
    class MockModel:
        async def ainvoke(self, messages, config=None, **kwargs):
            return AIMessage(content="test")
        def bind_tools(self, tools, **kwargs):
            return self
    
    model = MockModel()
    tools = []
    prompt = ChatPromptTemplate.from_messages([
        ("system", "You are a helpful assistant."),
        ("placeholder", "{messages}"),
    ])
    
    agent = create_react_agent(model, tools, prompt=prompt)
    
    handler = TestCallbackHandler()
    result = await agent.ainvoke(
        {"messages": [("human", "Hello")]},
        config={"callbacks": [handler]}
    )
    
    print(f"\nWith create_react_agent, events: {handler.events}")
    if "on_llm_start" not in handler.events:
        print("❌ CONFIRMED: Callbacks don't reach LLM calls in agent model node")
        print("   This is because trace=False bypasses callback setup in RunnableCallable")


if __name__ == "__main__":
    asyncio.run(main())

Error Message and Stack Trace (if applicable)

No error is thrown - this is a silent failure where callbacks are never set up.

The code path that's skipped (from `langchain_core/runnables/_runnable.py:410-429`):

# When trace=True (or omitted), this code runs:
if self.trace:
    await run_manager.on_llm_start(...)  # Callbacks set up here
    
# When trace=False, this is SKIPPED entirely:
else:
    # No callback setup happens!
    response = await self.afunc(*args, **kwargs)

Description

What is the problem?

The agent model node in langchain/agents/factory.py uses trace=False, which bypasses callback setup in RunnableCallable.ainvoke() and prevents callbacks from propagating to LLM calls.

Current behavior:

In langchain/agents/factory.py line ~1327:

node = RunnableCallable(
    func=amodel_node,
    name="model_node",
    trace=False,  # <-- This is the bug
)

When trace=False:

  1. RunnableCallable.ainvoke() skips the run_manager.on_llm_start() call
  2. Callback handlers never get registered for the model invocation
  3. Langfuse, OpenTelemetry, and other observability systems don't see LLM calls

Expected behavior:

trace should be True (or the parameter removed entirely, since it defaults to True), allowing callbacks to be properly set up for the model node.

Root cause in LangChain Core:

From langchain_core/runnables/_runnable.py lines ~410-429:

async def ainvoke(self, input: Input, config: Optional[RunnableConfig] = None, **kwargs: Any) -> Output:
    if self.trace:
        # Callback setup happens here (this is skipped when trace=False)
        callback_manager = AsyncCallbackManager.configure(...)
        run_manager = await callback_manager.on_llm_start(...)
    else:
        # trace=False path - NO callback setup!
        response = await self.afunc(input, config=config, **kwargs)

Impact:

This affects ALL agent-based workflows in LangChain:

  • create_react_agent
  • create_tool_calling_agent
  • create_xml_agent
  • create_structured_chat_agent
  • Any custom agents using the factory

Users cannot trace LLM calls within agents using:

  • Langfuse
  • LangSmith
  • OpenTelemetry
  • Custom callback handlers
  • Any observability relying on on_llm_start callbacks

Why trace=False was probably added:

Likely to avoid "double tracing" in some scenarios, but it breaks legitimate callback use cases. The proper solution is to fix config propagation (see related Issue #1) rather than disabling tracing entirely.

System Info

System Information
------------------
> OS:  Linux/Mac/Windows
> Python:  3.12+

Package Information
-------------------
> langchain: 0.3.x
> langchain-core: 0.3.x
> langgraph: 0.2.x

Other Dependencies:
> langfuse: 2.x/3.x (for tracing reproduction)

extent analysis

TL;DR

The issue can be resolved by setting trace=True or removing the trace parameter from the RunnableCallable constructor in langchain/agents/factory.py to allow callbacks to be properly set up for the model node.

Guidance

  • Identify the RunnableCallable constructor in langchain/agents/factory.py and update the trace parameter to True or remove it to default to True.
  • Verify that the on_llm_start callback is being triggered by checking the events list in the TestCallbackHandler instance.
  • Test the updated code with the provided minimal reproducible example to ensure that callbacks are being properly set up and triggered.
  • Review related issues, such as #30901, to ensure that config propagation is properly fixed to avoid "double tracing" scenarios.

Example

node = RunnableCallable(
    func=amodel_node,
    name="model_node",
    trace=True,  # Update or remove this line
)

Notes

The provided minimal reproducible example demonstrates the issue and can be used to test the fix. The trace=False parameter was likely added to avoid "double tracing" scenarios, but it breaks legitimate callback use cases. The proper solution is to fix config propagation rather than disabling tracing entirely.

Recommendation

Apply the workaround by setting trace=True or removing the trace parameter from the RunnableCallable constructor in langchain/agents/factory.py. This will allow callbacks to be properly set up and triggered for the model node, resolving the issue.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

langchain - ✅(Solved) Fix [langchain] trace=False on agent model node in factory.py prevents callback setup [4 pull requests, 4 comments, 2 participants]