dify - ✅(Solved) Fix [Worker] trigger_refresh_publisher queue missing from DEFAULT_QUEUES, causing unbounded Redis memory growth [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
langgenius/dify#35280Fetched 2026-04-16 07:06:41
View on GitHub
Comments
0
Participants
1
Timeline
2
Reactions
1
Author
Participants
Timeline (top)
closed ×1cross-referenced ×1

Root Cause

PR #27644 (feat: introduce trigger functionality, merged 2025-11-12) introduced two queues:

FileQueueAdded to DEFAULT_QUEUES?
api/tasks/trigger_subscription_refresh_tasks.py:84trigger_refresh_executor✅ Yes
api/schedule/trigger_provider_refresh_task.py:52trigger_refresh_publisherNo

The follow-up PR #28179 (merged 2025-11-13) updated .devcontainer/post_create_command.sh but also only added trigger_refresh_executor, not trigger_refresh_publisher.

Verified on upstream source: both v1.10.0 and v1.13.3 api/docker/entrypoint.sh contain trigger_refresh_executor but not trigger_refresh_publisher.

Fix Action

Fixed

PR fix notes

PR #35282: fix: add miss celery queue

Description (problem / solution / changelog)

[!IMPORTANT]

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

fix #35280, add miss celery queue

<!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> <!-- If this PR was created by an automated agent, add `From <Tool Name>` as the final line of the description. Example: `From Codex`. -->

Screenshots

BeforeAfter
......

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods

Changed files

  • .devcontainer/post_create_command.sh (modified, +1/-1)
  • api/docker/entrypoint.sh (modified, +2/-2)

Code Example

redis-cli -n 1 LLEN trigger_refresh_publisher

---

# Line 38 (Cloud)
- DEFAULT_QUEUES="...,trigger_refresh_executor,retention,..."
+ DEFAULT_QUEUES="...,trigger_refresh_publisher,trigger_refresh_executor,retention,..."

# Line 41 (Community / Self-hosted)
- DEFAULT_QUEUES="...,trigger_refresh_executor,retention,..."
+ DEFAULT_QUEUES="...,trigger_refresh_publisher,trigger_refresh_executor,retention,..."
RAW_BUFFERClick to expand / collapse

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.13.3 (affects all versions since 1.10.0)

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Deploy Dify (any version ≥ 1.10.0) with default configuration — ENABLE_TRIGGER_PROVIDER_REFRESH_TASK defaults to true
  2. Start the Celery worker using the standard Docker entrypoint (api/docker/entrypoint.sh)
  3. Wait several days or weeks
  4. Check Redis DB 1 (Celery broker database):
    redis-cli -n 1 LLEN trigger_refresh_publisher
  5. Observe a continuously growing count

✔️ Expected Behavior

The trigger_provider_refresh Celery Beat task (defined in api/schedule/trigger_provider_refresh_task.py:52 with queue="trigger_refresh_publisher") should be consumed by workers. The trigger_refresh_publisher queue should be listed in DEFAULT_QUEUES in api/docker/entrypoint.sh, alongside trigger_refresh_executor which is already correctly registered.

❌ Actual Behavior

trigger_refresh_publisher is not listed in DEFAULT_QUEUES in api/docker/entrypoint.sh (lines 38 and 41 as of v1.13.3), so no worker consumes this queue. Celery Beat enqueues a task every TRIGGER_PROVIDER_REFRESH_INTERVAL minutes (default: 1), and all messages accumulate indefinitely in Redis.

Observed impact on a production instance running for ~3 months:

  • 285,000+ Celery messages accumulated in the Redis list key trigger_refresh_publisher (DB 1)
  • ~410 MB of Redis memory consumed (94% of total used_memory)
  • Memory grows at ~5 MB/day with no upper bound
  • The key has no TTL, and under volatile-lru policy it is not eligible for eviction

Additionally, because trigger_provider_refresh never executes, TriggerSubscription credential and subscription pre-renewal is silently non-functional.

Root Cause

PR #27644 (feat: introduce trigger functionality, merged 2025-11-12) introduced two queues:

FileQueueAdded to DEFAULT_QUEUES?
api/tasks/trigger_subscription_refresh_tasks.py:84trigger_refresh_executor✅ Yes
api/schedule/trigger_provider_refresh_task.py:52trigger_refresh_publisherNo

The follow-up PR #28179 (merged 2025-11-13) updated .devcontainer/post_create_command.sh but also only added trigger_refresh_executor, not trigger_refresh_publisher.

Verified on upstream source: both v1.10.0 and v1.13.3 api/docker/entrypoint.sh contain trigger_refresh_executor but not trigger_refresh_publisher.

Suggested Fix

Add trigger_refresh_publisher to DEFAULT_QUEUES in api/docker/entrypoint.sh (both Cloud and Community editions):

# Line 38 (Cloud)
- DEFAULT_QUEUES="...,trigger_refresh_executor,retention,..."
+ DEFAULT_QUEUES="...,trigger_refresh_publisher,trigger_refresh_executor,retention,..."

# Line 41 (Community / Self-hosted)
- DEFAULT_QUEUES="...,trigger_refresh_executor,retention,..."
+ DEFAULT_QUEUES="...,trigger_refresh_publisher,trigger_refresh_executor,retention,..."

Also consider adding trigger_refresh_publisher to .devcontainer/post_create_command.sh (same omission in #28179).

Additional Notes

  • There may be other queues with the same pattern of "declared in task decorator but missing from DEFAULT_QUEUES". For example, monitor (defined in schedule/queue_monitor_task.py) appears to have a Beat schedule but is not in DEFAULT_QUEUES.
  • A CI check that validates all @app.celery.task(queue=...) / @shared_task(queue=...) queues are present in DEFAULT_QUEUES would prevent similar regressions.
  • Users who have already accumulated messages can safely purge them: redis-cli -n 1 UNLINK trigger_refresh_publisher (after stopping Beat to prevent re-accumulation).

extent analysis

TL;DR

Add trigger_refresh_publisher to DEFAULT_QUEUES in api/docker/entrypoint.sh to fix the issue with the Celery Beat task not being consumed by workers.

Guidance

  • Verify that the trigger_refresh_publisher queue is not listed in DEFAULT_QUEUES in api/docker/entrypoint.sh by checking lines 38 and 41.
  • Add trigger_refresh_publisher to DEFAULT_QUEUES in api/docker/entrypoint.sh as suggested in the diff provided.
  • Consider adding trigger_refresh_publisher to .devcontainer/post_create_command.sh to ensure consistency.
  • After applying the fix, stop the Celery Beat task and purge accumulated messages using redis-cli -n 1 UNLINK trigger_refresh_publisher to prevent re-accumulation.

Example

The suggested fix can be applied by modifying the DEFAULT_QUEUES variable in api/docker/entrypoint.sh as follows:

- DEFAULT_QUEUES="...,trigger_refresh_executor,retention,..."
+ DEFAULT_QUEUES="...,trigger_refresh_publisher,trigger_refresh_executor,retention,..."

Notes

The issue may not be limited to the trigger_refresh_publisher queue, and other queues may have the same pattern of being declared in task decorators but missing from DEFAULT_QUEUES. A CI check can be implemented to validate all queues and prevent similar regressions.

Recommendation

Apply the workaround by adding trigger_refresh_publisher to DEFAULT_QUEUES in api/docker/entrypoint.sh to fix the issue with the Celery Beat task not being consumed by workers. This will prevent further accumulation of messages in the Redis database and ensure that the TriggerSubscription credential and subscription pre-renewal functionality works as expected.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING