Checked other resources

• This is a feature request, not a bug report or usage question.
• I added a clear and descriptive title that summarizes the feature request.
• I used the GitHub search to find a similar feature request and didn't find it.
• I checked the LangChain documentation and API reference to see if this feature already exists.
• This is not related to the langchain-community package.

Package (Required)

• langchain
• langchain-openai
• langchain-anthropic
• langchain-classic
• langchain-core
• langchain-model-profiles
• langchain-tests
• langchain-text-splitters
• langchain-chroma
• langchain-deepseek
• langchain-exa
• langchain-fireworks
• langchain-groq
• langchain-huggingface
• langchain-mistralai
• langchain-nomic
• langchain-ollama
• langchain-openrouter
• langchain-perplexity
• langchain-qdrant
• langchain-xai
• Other / not sure / general

Feature Description

I would like LangChain to support a robust, core implementation of semantic text splitting within libs/text-splitters.

This feature would allow users to split documents based on the cosine similarity of sentence embeddings, ensuring that chunks are semantically coherent and focused on a single topic. Crucially, it includes Hierarchical Refinement: if a semantically identified chunk exceeds the chunk_size limit, the splitter recursively finds the next most significant semantic shifts (the "deepest valleys") within that chunk to satisfy the size constraint without breaking mid-sentence.

Use Case

I'm trying to build high-precision RAG (Retrieval-Augmented Generation) applications where context quality is paramount.

Currently, I have to work around this by using character-based splitters which often break mid-concept, or experimental semantic chunkers that do not strictly enforce chunk_size or handle redundant similarity distributions well.

This feature would help users generate "Topic-Pure" chunks that significantly improve retrieval precision and prevent LLMs from being confused by fragmented context.

Proposed Solution

I have implemented SemanticSimilarityTextSplitter which inherits from TextSplitter.

The API looks like:

python from langchain_text_splitters.semantic import SemanticSimilarityTextSplitter from langchain_openai import OpenAIEmbeddings embeddings = OpenAIEmbeddings() splitter = SemanticSimilarityTextSplitter( embeddings=embeddings, breakpoint_threshold_type="percentile", # percentile, standard_deviation, interquartile breakpoint_threshold_amount=95.0, chunk_size=4000 ) chunks = splitter.split_text("Your document here...")

• Flexible Thresholding: Supports percentile, standard_deviation, and interquartile strategies.
• Hierarchical Fallback: Uses a "Deepest Valley" recursive search to keep oversized chunks as semantically intact as possible.
• Robustness: Handles single-sentence documents and documents with zero-variance similarity distributions gracefully.

Alternatives Considered

I've tried using:

1. RecursiveCharacterTextSplitter: It is fast but structurally blind to meaning, often splitting at non-semantic character counts.
2. SemanticChunker (Experimental): It lacks strict chunk_size enforcement and sometimes fails on edge cases like redundant sentence similarity.

Alternative approaches I considered: Purely semantic splitting without size limits: This fails when chunks become too large for modern LLM context windows or efficient embedding storage.

Additional Context

I have fully implemented this feature in my fork (krish1440/langchain).

The code strictly follows Google-style docstrings and monorepo guidelines. I have added a comprehensive test suite (test_semantic_splitter.py) with 28 passing unit tests verified on Python 3.8 through 3.12.