langchain - 💡(How to fix) Fix Post-RAG Deterministic Verification Runnable & Header-Aware Splitter Checked other resources [2 comments, 2 participants]

Checked other resources

• This is a feature request, not a bug report or usage question.
• I added a clear and descriptive title that summarizes the feature request.
• I used the GitHub search to find a similar feature request and didn't find it.
• I checked the LangChain documentation and API reference to see if this feature already exists.
• This is not related to the langchain-community package.

Package (Required)

• langchain
• langchain-openai
• langchain-anthropic
• langchain-classic
• langchain-core
• langchain-model-profiles
• langchain-tests
• langchain-text-splitters
• langchain-chroma
• langchain-deepseek
• langchain-exa
• langchain-fireworks
• langchain-groq
• langchain-huggingface
• langchain-mistralai
• langchain-nomic
• langchain-ollama
• langchain-openrouter
• langchain-perplexity
• langchain-qdrant
• langchain-xai
• Other / not sure / general

Feature Description

I would like LangChain to support a deterministic "Verification Loop" as a native Runnable component for LCEL chains, as well as a more robust HeaderAwareMarkdownSplitter.

Currently, RAG pipelines built with LangChain generally end at the generation step. The LLM produces an AIMessage based on retrieved context, but there is no built-in, native component to force a deterministic audit of that message.

This feature would allow users to append a RunnableVerifier to the end of an LCEL chain. This verifier would explicitly take the generated output and the original context documents, prompt a secondary (often larger) LLM to Boolean-score every sentence against the source material, and append a "Confidence/Grounding Score" to the final output or strip out unsupported hallucinated claims entirely. Additionally, a HeaderAwareMarkdownSplitter would ensure documents are split by structure rather than strict chunk-size, preventing context separation before retrieval.

Use Case

I'm trying to build an enterprise-level, compliance-heavy application (Legal/Medical RAG) where hallucination is absolutely unacceptable.

Currently, I have to work around this by building custom, complex orchestration outside of standard LCEL. I have to parse the generated output, feed it back into an entirely new custom chain along with the original retrieved Documents, and manage the scoring logic manually.

This feature would help users to easily enforce strict hallucination checks seamlessly within LCEL (e.g., retriever | prompt | llm | verifier_runnable). It brings production-ready safety directly into the default LangChain workflow, transforming RAG from probabilistic generation into deterministic extraction.

Proposed Solution

I think this could be implemented by creating a RunnableVerification component in langchain-core.

The API could look like navigating standard LCEL:

from langchain_core.runnables import RunnablePassthrough
from langchain.chat_models import ChatGroq
from langchain_core.verification import RunnableFactChecker

# Standard RAG components
retriever = vectorstore.as_retriever()
draft_llm = ChatGroq(model="llama-3.1-8b-instant")
verifier_llm = ChatGroq(model="llama-3.3-70b-versatile") 

# The Proposed Component
fact_checker = RunnableFactChecker(
    llm=verifier_llm, 
    strict_mode=True # Strips unsupported claims from the final output
)

# Implementation
chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | draft_llm
    | fact_checker # The new step
)

result = chain.invoke("What are the terms of the contract?")
print(result.content) 
print(result.response_metadata['confidence_score']) # Returns 0.0 to 1.0 based on evidence


### Alternatives Considered


### Alternatives Considered
```markdown
I've tried using standard post-processing techniques and custom evaluation libraries (like Ragas or LangSmith evaluators). 

Alternative approaches I considered:
1. Using LangSmith evaluators (e.g., `labeled_criteria`).
2. Writing custom parallel chains to evaluate the output.

But these don't work natively for *runtime blocking* because evaluators like Ragas are typically used for offline metrics/testing, not as active, blocking filters in a live production pipeline protecting end-users.


### Additional Context

I recently built a custom extraction pipeline in raw Python solving this exact problem, combining small routing models with large verification models. I am highly motivated to contribute to this! I am willing to write the PR, tests, and documentation for either the `RunnableVerifier` or the `HeaderAwareMarkdownSplitter` if the core maintainers believe this aligns with LangChain's roadmap for safer RAG generation.