Hi LlamaIndex team — proposing a community evaluator backed by styxx, a hallucination detector cross-validated across 8 public benchmarks.

Directly relevant to RAG: our strongest new number is AUC 0.807 on HaluBench-RAGTruth (3-seed averaged, n=150/dataset). That's the benchmark measuring RAG faithfulness — the core LlamaIndex use case.

What this would be

A BaseEvaluator subclass that slots alongside your existing FaithfulnessEvaluator / RelevancyEvaluator:

from styxx.adapters.llamaindex import StyxxHallucinationEvaluator

evaluator = StyxxHallucinationEvaluator(threshold=0.7)

# Works with the standard BaseEvaluator signature
result = await evaluator.aevaluate(
    query="who directed inception?",
    response="Christopher Nolan directed Inception in 2010.",
    contexts=["Inception is a 2010 film directed by Christopher Nolan."],
)

result.passing    # True/False against threshold
result.score      # 1.0 - risk (high = faithful)
result.feedback   # human-readable with action + signals
result.metadata   # {"risk": ..., "action": ..., "signals": {...}}

The adapter is live today at https://github.com/fathom-lab/styxx/blob/main/styxx/adapters/llamaindex.py (commit a817732). pip install styxx[nli,llamaindex] gets you everything.

Numbers on RAG-relevant benchmarks (3-seed averaged, n=150/dataset)

Two published failure modes in the paper (HaluBench-DROP 0.424 extractive-span reading comp; FinanceBench 0.492 financial arithmetic) — declared openly in the weights module. LlamaIndex users should avoid this evaluator for those specific domains.

Paper + provenance

• Zenodo DOI: https://doi.org/10.5281/zenodo.19703527 (CC-BY-4.0)
• OSF project: https://osf.io/g2epj/
• Merged into EdinburghNLP/awesome-hallucination-detection#55 by Pasquale Minervini (HaluEval author)
• Working Guardrails AI validator already in the same repo (issue)

What I'd do

Happy to submit as a PR to llama-index-integrations under llama-index-evaluation-styxx (or whichever namespace you prefer for community evaluators), or keep it in the styxx repo where it lives now and just document it as a supported integration.

Also happy to run the 8-benchmark suite against any existing LlamaIndex evaluators (FaithfulnessEvaluator, GuidelineEvaluator) for a direct comparison if that would be useful.

Thanks, Flobi Fathom Lab