dify - 💡(How to fix) Fix [URGENT] Langfuse trace isolation issue affecting users who installed 1.14.1 [1 participants]

Summary

We received reports from two users that Dify spans appeared in Langfuse projects that did not belong to them. One report indicated that sensitive Dify spans from other users may have been visible.

Users who already installed Dify 1.14.1 and use workflow-level Langfuse tracing should treat this as urgent.

Event timeline

All times below are UTC.

• 2026-03-27: #34186 reported that Dify pinned langfuse to 2.51.3, preventing Langfuse LLM-as-a-Judge on Observations.
• 2026-03-30: #34265 merged the Langfuse SDK upgrade to v3+ by changing the dependency from ~=2.51.3 to >=3.0.0,<5.0.0 and migrating Dify's Langfuse trace code to the v3 ingestion API.
• 2026-05-12 08:19:23: Dify 1.14.1 was published.
• 2026-05-13: We received reports from two users that Dify spans appeared in Langfuse projects that did not belong to them.
• 2026-05-13 07:05:15: #36107 was merged into hotfix/1.14.1-fix.1 to isolate the Langfuse v3 SDK TracerProvider.

What happened

Based on #36107, the Langfuse Python SDK v3 can attach its SpanProcessor to the global OpenTelemetry TracerProvider when no explicit tracer_provider is passed.

Dify already installs a global OpenTelemetry TracerProvider through ext_otel.py, with Flask / Celery / SQLAlchemy instrumentation active. Under that setup, spans emitted by the process across tenants could be exported to the first tenant that configured a workflow-level Langfuse integration.

Impact

Potentially affected users:

• Self-hosted Dify deployments running 1.14.1
• Deployments using workflow-level Langfuse tracing
• Deployments with multiple tenants or multiple Langfuse projects configured in the same Dify process

Possible impact:

• Dify spans may be exported to the wrong Langfuse project.
• Sensitive trace inputs, outputs, metadata, or operational spans may appear in a Langfuse project owned by another tenant/user.

Immediate action for users on 1.14.1

If you installed 1.14.1 and use Langfuse tracing:

1. Temporarily disable workflow-level Langfuse tracing until you are running a build that includes #36107.
2. Upgrade to the fixed hotfix/release as soon as it is published.
3. Review Langfuse projects for unexpected spans.
4. Treat any unexpected trace data as potentially sensitive.
5. Rotate Langfuse credentials after upgrading if you suspect exposure.
6. Avoid posting sensitive trace contents publicly; share details through the appropriate private security channel.

Notes:

• Deleting traces also deletes related observations and scores in Langfuse.
• Langfuse trace deletion may not be immediate. Re-query after the deletion job has had time to complete.
• If the UI cannot filter nested resourceAttributes directly in a specific Langfuse version, export/query traces for the affected time window and client-side filter by metadata.resourceAttributes.*, then delete matching trace IDs in small batches through the Langfuse trace deletion API.
• Avoid posting sensitive trace contents publicly. Share evidence only through a private security channel.

Fix

#36107 fixes this by:

• constructing an isolated OpenTelemetry TracerProvider per LangFuseDataTrace instance;
• passing it to Langfuse(..., tracer_provider=...);
• avoiding use of the global OpenTelemetry provider for Langfuse exports;
• shutting down the isolated provider when cached trace instances are evicted.