PR #35515: refactor(redis): unify main + pub/sub client construction via RedisConnectionSpec

• Repository: langgenius/dify
• Author: lin-snow
• State: open | merged: False
• Link: https://github.com/langgenius/dify/pull/35515

Description (problem / solution / changelog)

Fixes #35480 Closes #35516

Summary

Replaces the URL-string Redis config contract with a structured RedisConnectionSpec dataclass so the main client and the Event Bus (pub/sub) client share one construction path across all three topologies. Addresses #35480 at the contract level; see #35483 for the minimal-invasion alternative for the same root cause.

Why the URL contract was the wrong abstraction

A URL cannot encode Sentinel topology (list of sentinel nodes + service name + sentinel credentials). Under Sentinel, pub/sub silently mis-routed to REDIS_HOST; under Cluster, it hit a hard ValueError at startup. Pub/sub could not participate in Sentinel HA under any configuration.

What this PR does

• New RedisConnectionSpec (api/configs/middleware/cache/redis_connection_spec.py) — frozen dataclass covering all three topologies. Masks passwords in __repr__; hashable so pubsub_spec == main_spec comparison drives the "reuse main client" decision.
• build_main_redis_spec / build_pubsub_spec factories read env into specs; cross-validate invalid combinations (e.g. REDIS_USE_SENTINEL + REDIS_USE_CLUSTERS) at the config layer.
• ext_redis._create_*_client now take (spec, transport) — topology from the spec, tuning knobs from a consolidated RedisTransportParamsDict (replaces three parallel dicts).
• ext_redis.init_app reuses the main client object when pubsub_spec == main_spec (the default "inherit" path), otherwise builds a dedicated client via the same factory.
• ext_celery._reject_cluster_broker_url — startup guard that fails fast when CELERY_BROKER_URL / CELERY_RESULT_BACKEND points at redis+cluster:// (Kombu limitation).
• Hardening: malformed entries in REDIS_SENTINELS / REDIS_CLUSTERS / PUBSUB_REDIS_SENTINELS / PUBSUB_REDIS_CLUSTERS report the 1-based position (never the raw content), so a password accidentally pasted into these env vars does not leak into startup logs.

Unlocked capabilities

• Pub/sub under Sentinel HA — works by default via main-client reuse, or via PUBSUB_REDIS_MODE=sentinel for an independent Sentinel.
• Pub/sub under Cluster — works by default via main-client reuse.
• Startup rejection for invalid combinations, surfaced with clear, env-var-named error messages.

⚠️ Breaking change

PUBSUB_REDIS_URL / EVENT_BUS_REDIS_URL and PUBSUB_REDIS_USE_CLUSTERS / EVENT_BUS_REDIS_USE_CLUSTERS are removed. All three deployment templates shipped these with empty defaults, so most deployments will silently fall back to the inherit path and start working correctly under Sentinel for the first time. Deployments that genuinely want an independent pub/sub Redis should migrate to PUBSUB_REDIS_MODE plus the structured PUBSUB_REDIS_* fields (which, unlike the URL, can express independent Sentinel topologies).

Tests

51 new unit tests across four files covering spec construction, builder cross-validation, password-redaction in __repr__, inheritance semantics, independent Sentinel pub/sub, and the Celery Cluster guard. Full local run: 11723 passed, 4 skipped, 0 failed.

Screenshots

N/A (backend / configuration layer change).

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues.
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods.

Changed files

• api/.env.example (modified, +31/-8)
• api/configs/middleware/cache/redis_connection_spec.py (added, +386/-0)
• api/configs/middleware/cache/redis_pubsub_config.py (modified, +123/-60)
• api/extensions/ext_celery.py (modified, +36/-1)
• api/extensions/ext_redis.py (modified, +148/-157)
• api/tests/unit_tests/configs/middleware/cache/test_build_main_redis_spec.py (added, +263/-0)
• api/tests/unit_tests/configs/middleware/cache/test_build_pubsub_spec.py (added, +210/-0)
• api/tests/unit_tests/configs/middleware/cache/test_redis_connection_spec.py (added, +236/-0)
• api/tests/unit_tests/configs/test_dify_config.py (modified, +113/-10)
• api/tests/unit_tests/extensions/test_celery_broker_validation.py (added, +89/-0)
• api/tests/unit_tests/extensions/test_redis.py (modified, +19/-43)
• docker/.env.example (modified, +31/-8)
• docker/docker-compose.yaml (modified, +13/-2)