Memory leak: API gunicorn (gevent) workers show linear RSS growth, while Celery workers on same code are stable

Describe the bug

API gunicorn workers with gevent worker class exhibit linear memory growth under production traffic. Worker RSS increases from ~850MB at startup to >3GB after ~1.5 days. Celery workers running the same codebase (same workflow execution paths) do not show this pattern — they remain stable at ~1GB after 4 days.

To Reproduce

1. Run Dify API with gunicorn + gevent: --workers 3 --worker-class gevent --worker-connections 200
2. Do not set --max-requests (no worker auto-restart)
3. Submit normal production workload (workflow-based app executions + streaming responses)
4. Monitor worker RSS over time

Observed Behavior

Production API workers:

PID    RSS      Uptime
53045  868 MB   ~0.5 h    (fresh restart)
47107  2.0 GB   ~8 h      (+1.1 GB)
32892  3.0 GB   ~36 h     (+2.1 GB)

Roughly +130-140 MB/hour per worker.

Production Celery worker (same codebase, same deployment):

PID    RSS      Uptime
1      1.0 GB   4 days    (flat)

UAT API workers (same config, low traffic):

PID    RSS      Uptime
5167   728 MB   4 days    (only ~100MB growth over 4 days)

The rate of growth correlates with request volume, suggesting request-scoped objects are not being fully released between requests.

Key Observation: API vs Celery divergence

The same workflow execution code runs in both processes. The difference:

This suggests the leak is in the API request layer (Flask context, SSE generator lifecycle, queue manager, or request-scoped middleware) rather than in shared workflow execution code.

What we've ruled out

• _model_asset_cache (module-level dict): stable across all processes, few dozen entries, not growing
• ProviderConfigurationsCache: TTL-based, limited size

Temporary Mitigation

Adding --max-requests 1000 --max-requests-jitter 100 to gunicorn would recycle workers before they grow large enough to OOM, but this masks rather than fixes the root cause.

Environment

• Dify version: 1.13.3 (with custom modifications)
• Python: 3.11
• Gunicorn: --workers 3 --worker-class gevent --worker-connections 200 --timeout 200
• Celery: -P gevent -c 50 --max-tasks-per-child 50
• Deployment: Kubernetes, single tenant
• Monitoring: ps aux RSS

Questions for maintainers

1. Are there known memory retention issues with gevent workers in Flask/gunicorn that would explain API-only growth?
2. Are SSE streaming generators properly cleaned up when the client disconnects or the request completes? Could dangling generator references prevent GC?
3. Has anyone else reported similar API-vs-Celery memory divergence in recent versions?
4. Would upgrading gunicorn or switching to a different worker class (e.g., uvicorn.workers.UvicornWorker) help?

Related Issues

• #30183 — Memory leak under high load
• #30777 — Memory of dify-api was not released
• #34409 — Out of memory constantly