Feature Description

Semantic Dedupulication

Current State: Absence of any Deduplication step

In llama_index/finetuning/cross_encoders/dataset_gen.py, generate_synthetic_queries_over_documents(...) generates synthetic questions per document chunk using an LLM and aggregates them into a flat list. However, there is currently no deduplication step before returning the final list of questions. Currently, we have (See full code):

questions.extend(response_questions)
...
return questions

All generated questions are appended and returned as-is, without normalization or deduplication.

This can result in identical questions across multiple chunks, and near-duplicate questions with minor formatting differences. Further down the training pipeline, this can entail redundant training examples in downstream cross-encoder datasets, and inflated dataset size without additional semantic coverage.

This issue becomes more noticeable when num_questions_per_chunk is large, and documents contain repetitive structure.

Proposed Improvement 1: Lightweight deduplication

Before returning questions, do a check for exact matches, preserving order, stripping whitespace, and removing empty strings:

deduped_questions = list(dict.fromkeys(q.strip() for q in questions if q.strip()))
return deduped_questions

Proposed Improvement 2: Semantic Deduplication

Main goal. We want near-duplicate detection (paraphrases, trivial rewrites, casing/punctuation changes, “What is X?” vs “Explain X.”)

We may use embedding-based near-duplicate removal -- encode each generated question into a vector embedding and compare it against embeddings of previously accepted questions. If the cosine similarity between a new question and any existing one exceeds a predefined threshold, the new question is treated as a semantic duplicate and discarded.

To implement, we should first normalize the text (e.g., stripping whitespace, removing numbering or bullet markers, optionally lowercasing), then generate embeddings using a sentence-level model such as OpenAI embeddings or a local sentence-transformers model.

To ensure scalability, an Approximate Nearest Neighbor (ANN) index (e.g., FAISS or hnswlib) can be used to efficiently retrieve the most similar existing vectors rather than comparing against all prior questions.

Key tradeoffs to consider. This semantic deduplication approach requires computing and storing embeddings, as well as maintaining a vector index for similarity search.

Reason

No response

Value of Feature

Deduplication helps further down the training pipeline as follows:

• Cross-encoder training quality improves with diverse queries
• Prevents redundant positives in ranking datasets
• Reduces training time and memory usage
• Encourages better semantic coverage across chunks