dify - 💡(How to fix) Fix Bug: Async workflow time-slice scheduler for sandbox tier is incomplete (risk of stuck RUNNING state) [1 participants]

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Summary

On Dify Cloud, asynchronous workflows for the sandbox (free) queue use a CFS (completely fair scheduling) time-slice plan. The current AsyncWorkflowCFSPlanScheduler implementation does not implement a real “can this run continue” decision for sandbox: it always returns RESOURCE_LIMIT_REACHED unless the run is on the professional or team queue. There is an explicit FIXME in code stating the need to prevent a sandbox user’s workflow from remaining in a running state indefinitely.

This is a reliability / state-machine issue: incorrect or over-aggressive pause signaling can leave workflow runs in an inconsistent or stuck state relative to user expectations, and the scheduler logic is acknowledged as unfinished.

Severity: High (core workflow execution, long-running / async paths, Dify Cloud)

Area: api/tasks/workflow_cfs_scheduler/, api/core/app/layers/timeslice_layer.py, api/tasks/async_workflow_tasks.py

Evidence in codebase

1. FIXME on sandbox scheduling — api/tasks/workflow_cfs_scheduler/cfs_scheduler.py:

   • can_schedule() returns SchedulerCommand.NONE only for PROFESSIONAL_QUEUE and TEAM_QUEUE.
   • For all other cases (including SANDBOX_QUEUE), it returns RESOURCE_LIMIT_REACHED.
   • Comment: “FIXME: avoid the sandbox user's workflow at a running state for ever”.

2. Time-slice layer behavior — api/core/app/layers/timeslice_layer.py:

   • When can_schedule() == RESOURCE_LIMIT_REACHED, the background job sends GraphEngineCommand with CommandType.PAUSE and payload reason: resource_limit_reached.
   • This ties the scheduler’s boolean outcome directly to pausing the graph.

3. When this path is active (Cloud only) — api/tasks/workflow_cfs_scheduler/entities.py:

   • If EDITION == "CLOUD", AsyncWorkflowSystemStrategy is TimeSlice and separate Celery queues exist for professional, team, and sandbox.
   • If not Cloud, strategy is Nop, so the time-slice checker is not used the same way — this bug primarily concerns Dify Cloud and the sandbox queue.

4. Related uncertainty on workflow run status — api/repositories/sqlalchemy_api_workflow_run_repository.py (pause path):

   • TODO notes that WorkflowRun.status persistence may occur in an order that interacts badly with graph execution (“before the execution of GraphLayer”), which can exacerbate user-visible “wrong status” symptoms when combined with aggressive pause/limit logic.

✔️ Expected Behavior

• Sandbox tier workflows should either:
  • be throttled in a defined way (clear quotas, backoff, or bounded pause) without leaving runs stuck in RUNNING, or
  • if paused for quota, transition to a consistent status (PAUSED / failed / resumable) and recover predictably.
• can_schedule() for sandbox should encode real resource / quota state, not a permanent RESOURCE_LIMIT_REACHED if that does not match product intent.

Exact product rules (limits, when to pause vs. queue) should be confirmed with the Dify Cloud team.

❌ Actual Behavior

How to reproduce (indicative)

1. Run Dify Cloud with async workflow execution routed to the sandbox Celery queue (execute_workflow_sandbox).
2. Use a workflow whose execution uses the TimeSlice CFS plan (same as production Cloud config in entities.py).
3. Observe graph scheduling: TimeSliceLayer will periodically call can_schedule(); for sandbox, it always gets RESOURCE_LIMIT_REACHED (non–professional/team), so pause commands are emitted on every check interval.
4. Inspect workflow run and node run records for stuck or inconsistent RUNNING / pause states under load or over long runs.

Concrete UI/API steps depend on your Cloud environment and app configuration; backend logs for the sandbox queue and workflow run status fields are the primary signal.

Suggested fix direction (non-prescriptive)

1. Implement sandbox can_schedule() using real quota / usage (or a deliberate cooldown), and align pause/resume with WorkflowRun / pause entities.
2. Add integration coverage for: sandbox + TimeSlice + long run + pause/resume, asserting terminal/suspended states are correct.
3. Revisit the WorkflowRun status ordering TODO if pause/continue races appear in support tickets.

Related upstream issue (feature regression)

• GitHub: #20725 — Knowledge retrieval node: structured_output_enabled is forced to false in api/core/workflow/nodes/knowledge_retrieval/entities.py as a temporary workaround; proper fix is architectural (node inheritance). Track separately if the “bug” you care about is RAG/structured output rather than async scheduling.

Metadata for filing on GitHub

• Title: [Cloud] Sandbox async workflow CFS scheduler: FIXME re. stuck RUNNING; can_schedule always resource-limited for non–paid queues
• Labels (suggested): bug, backend, workflow, cloud (if applicable)
• Environment: Dify Cloud, sandbox queue, async workflow tasks

This file was generated from a static analysis of the repository at the time of writing, not from runtime reproduction on a live Dify Cloud tenant.