PR #35498: fix(rag): decouple hybrid recall from top_k for stable merged ranking…

• Repository: langgenius/dify
• Author: thomascolden585-svg
• State: open | merged: False
• Link: https://github.com/langgenius/dify/pull/35498

Description (problem / solution / changelog)

Summary

Fixes mixed (hybrid) RAG retrieval so changing only Top K no longer changes which segments participate in score fusion, which could reorder the head and make the first N results differ between a small Top K and a large Top K for the same query.

Problem

Hybrid search runs vector and full-text in parallel, deduplicates, then merges scores (weighted score or reranker). The final top_k was also used as the per-channel limit for each sub-retriever. A segment can sit below k in both channels but still get a high combined score. With a small k it never entered the candidate pool; with a larger k it did—so the merged top results were inconsistent with user expectations (reported in #35482).

Solution

• Introduce a dedicated per-channel recall for hybrid: min(200, max(50, final_top_k)).
• Use it for embedding_search and full_text_index_search in hybrid mode.
• Keep post-merge cut at the user’s top_k via DataPostProcessor.invoke(..., top_n=top_k).

Trade-offs

Slightly more work per hybrid query (larger per-channel fetch) in exchange for stable, correct fusion behavior.

Test plan

• uv run pytest tests/unit_tests/core/rag/datasource/test_datasource_retrieval.py::test_hybrid_recall_top_k_for_merge_contract -q
• uv run pytest tests/unit_tests/core/rag/retrieval/test_dataset_retrieval.py -k hybrid -q
• Manual: same query, hybrid + weighted / rerank, Top K 3 vs 8 — first 3 should match (same ordering and segments).

Related

• Closes / fixes #35482

Changed files

• api/core/rag/datasource/retrieval_service.py (modified, +33/-3)
• api/tests/unit_tests/core/rag/datasource/test_datasource_retrieval.py (modified, +9/-0)