dify - 💡(How to fix) Fix Knowledge Base: Duplicate content chunks from different documents are collapsed during retrieval (content-only deduplication ignores metadata) [1 comments, 2 participants]

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Dify version

1.13.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

1. Create a Knowledge Base with Hybrid Search enabled.
2. Upload two different documents (e.g. file_a.txt and file_b.txt) that share at least one section of identical text content but have different filenames/metadata.
3. Index both documents successfully.
4. Query the Knowledge Base with a phrase that matches the shared content.

✔️ Expected Behavior

Both documents are returned as separate retrieval results, each with their own source metadata (e.g. filename, document ID). The knowledge base correctly distinguishes two chunks that originate from different source documents, even if their text content is identical.

❌ Actual Behavior

Only one of the two documents is returned. The retrieval result for the second document is silently dropped. Any downstream citation or attribution that depends on the second document's metadata is lost.

Root Cause (for maintainers)

In api/core/rag/datasource/retrieval_service.py, the _deduplicate_documents method builds a deduplication key using page_content alone when the primary UUID path isn't taken:

# ~line 249
content_key = (doc.provider or "dify", doc.page_content)  # ← content only
if content_key not in chosen:
    chosen[content_key] = doc
    order.append(content_key)

This means two chunks with identical text but different doc_id UUIDs (originating from different source documents) are treated as duplicates, and only the first encountered is kept.

Additionally, api/libs/helper.py's generate_text_hash() hashes only page_content without including any metadata, so the same assumption propagates to embedding cache keying and thread group assignment in indexing_runner.py.

Suggested fix — prefer doc_id in the fallback branch when available:

else:
    available_id = (doc.metadata or {}).get("doc_id")
    content_key = (doc.provider or "dify", available_id if available_id else doc.page_content)
    if content_key not in chosen:
        chosen[content_key] = doc
        order.append(content_key)

Environment

• Dify version: (fill in)
• Vector store backend: (e.g. Weaviate, Qdrant, pgvector)
• Search mode: Hybrid Search