llamaIndex - 💡(How to fix) Fix [Feature Request]: Trust scoring and interaction history for tool and agent reliability [1 participants]

Feature Description

LlamaIndex agents can call tools, query external data sources, and delegate to sub-agents. But there's no way to track whether a particular tool or agent has been reliable across sessions. If your RAG pipeline queries an external API that returned bad data last time, or delegates to a sub-agent that failed 3 out of 5 recent tasks, nothing in LlamaIndex remembers that.

I built a sidecar that records interactions between agents and tools bilaterally. Both the caller and the callee cosign a record of what happened. From the graph of those records you get trust scores, skill-specific reliability (this tool is good at code search but bad at summarization), Sybil detection, and full audit trails. New tools/agents start UNPROVEN and graduate through 5 trust tiers as their track record grows.

Works as a transparent HTTP proxy (zero code changes) or through a Python SDK:

from trustchain import TrustClient

tc = TrustClient()

# before calling a tool: check its track record
trust = tc.check_trust(tool_pubkey)
if trust.tier < "ESTABLISHED" or trust.skill_reliability("retrieval") < 0.7:
    # use a different retriever or add verification
    ...

# after the call: record what happened
tc.record_interaction(counterparty=tool_pubkey, action="retrieval", quality=0.9)

Also has 7 MCP tools so MCP-capable agents can query trust natively. Verification works fully offline with just Ed25519 public keys.

The interaction graph is a substrate you can build on. Trust scoring is just the first use. The same graph powers agent discovery (rank by reputation within a capability), behavioral anomaly detection, dispute resolution, and compliance audit trails.

Reason

LlamaIndex currently has no mechanism for cross-session tool/agent reliability tracking. If a tool returns bad results, the agent has no memory of that in the next session. Guardrails validate output format but not source reliability over time.

Value of Feature

• Agents make better delegation decisions based on actual track record, not just tool descriptions
• Bad tools/agents get deprioritized automatically without manual curation
• RAG pipelines become more reliable by routing to sources with proven retrieval quality
• Full audit trail for compliance (EU AI Act Article 12 ready)

Working implementation with a LlamaIndex adapter: https://github.com/viftode4/trustchain Live demo with 21 LLM agents in a trust-scored economy: http://5.161.255.238:8888