Bug Description

In N8N_RUNNERS_MODE=external, the broker↔runner channel can enter a stuck state where every task request times out at 60 seconds while the runner process is still alive and its HTTP /healthz keeps returning 200. The launcher's HTTP-based health monitor cannot detect this state, and there is no self-recovery path: the system stays stuck until something external (a deploy, a process OOM, or the launcher panicking) forces a container restart.

This is the same observable symptom as #23381 (Roc-Jia7's comment), #26629, #27406, #28881 and #28625 — but unlike those reports, this one has a single controlled instance, full timeline data across multiple recurrences, and a clear differentiation between "runner alive but unreachable via task channel" and "runner dead". I am also the author of #25958 / #25959, which fixed a different mechanism that produced the same surface symptom on the same Cloud Run + external runner setup, so this is not a duplicate of that fix.

Reproduced across versions and across multiple recurrences

Examination of 7 days of Cloud Logging (2026-04-21 to 2026-04-28) reveals three distinct stuck windows with the same signature (Task request timed out sustained at ~24/hour for the duration):

Total stuck time over 7 days: ~59 hours, recovered three different ways but never via any self-healing mechanism in n8n or the launcher.

This rules out the most common dismissals upfront:

• "Fixed in a newer version" — bug occurs on both 2.15.0 and 2.17.7.
• "One-off / cold-start" — three distinct windows on the same instance, repeating at ~1.3 day cadence on average.
• "Resource exhaustion is the cause" — the three recoveries had nothing in common with each other (panic, manual deploy, separate main OOM); during the 37-hour window 2 the runner did not exit or OOM at all, ruling out runner-side resource pressure as the trigger.
• "Single environment / Cloud Run-specific" — happens repeatedly in our environment, and the same surface symptom has been reported across Cloud Run, self-hosted Kubernetes (#26629), and n8n Cloud (#28881).

Detailed observed timeline of stuck window 1 (best-instrumented)

So for ~11 hours: runner process up + /healthz 200 + every task times out at exactly 60 s.

Stuck windows 2 and 3 produced the same hourly cadence of Task request timed out errors at 21–28 / hour, sustained until the next external recovery event. Full Cloud Logging exports of all three windows are available on request.

Stack trace (from n8n-main, Cloud Logging)

Error: Task request timed out
    at LocalTaskRequester.requestExpired (/usr/.../task-requester.ts:309:22)
    at LocalTaskRequester.onMessage    (/usr/.../task-requester.ts:272:10)
    at TaskBroker.handleRequestTimeout (/usr/.../task-broker.service.ts:120:50)
    at Timeout.<anonymous>             (/usr/.../task-broker.service.ts:107:9)

The 60 s figure matches the default N8N_RUNNERS_TASK_TIMEOUT. The broker genuinely never receives a runner response for any task request during the stuck window, while the runner process and /healthz listener stay up.

Why this report is distinct from prior closed reports

• Same instance, same revision, no scaling/cold-start involvement — rules out the "runner not started yet" / "wrong sidecar" / "queue-mode worker config" causes that prior reports landed on.
• /healthz is unambiguously up the entire time — observable by the absence of "Found runner unresponsive" warnings from the launcher during all three stuck windows.
• The stuck state ends only via external intervention (panic, manual deploy, or unrelated main-side OOM). It never recovers on its own — strongly indicating a stale connection / state on the broker↔runner channel that the current health model cannot see.

Why this is not a resource-sizing or environment issue

To preempt the natural "is this just OOM / sizing?" reading:

1. The stuck state observed before any process self-died is the bug. Stuck windows 1 and 3 ended via a process OOM on the runner side and main side respectively, but stuck window 2 ran for 37 hours with no OOM, no exit(134), no exit(2), and no resource-exhaustion signal anywhere — and was only broken by a manual deploy. Resource pressure is not the trigger.
2. Window 1 ended when the runner self-died after 12 hours; window 2 ended without any runner death after 37 hours. If runner memory exhaustion were the trigger, the durations would scale with runner memory — they do not, and the runner did not OOM at all in window 2.
3. Throughout all three stuck windows, /healthz returned 200 and the launcher emitted no "Found runner unresponsive" warnings — i.e. the system reported healthy while every task failed. This is a logical contradiction in the observability surface, regardless of what resources are allocated.
4. The same surface symptom has been reported on Cloud Run, self-hosted Kubernetes (#26629), and n8n Cloud (#28881), arguing against a single-environment cause.
5. The fix class previously accepted by the team in #25959 — broker /healthz returning a value that does not reflect the actual readiness/health of the task channel — is exactly this kind of bug, just at runtime instead of at startup.

Possible root causes (for triage)

1. WebSocket / long-poll connection between broker and runner appearing alive at TCP level but no longer carrying messages (idle Cloud Run CPU throttling, missing keepalive, half-closed socket).
2. Runner-side event loop blocked while /healthz answers from a separate handler thread.
3. Internal lock / ack-tracking deadlock in task-broker.service.ts.

What I think the right fix shape is

The launcher and the orchestrator (Cloud Run / Kubernetes) already have everything needed to restart a stuck runner — the launcher's monitorRunnerHealth polls /healthz and kills the runner on 6 consecutive failures, and Cloud Run / K8s liveness probes will do the same. The reason recovery does not happen is that none of these health signals reflect the actual liveness of the task channel during this stuck state.

So I do not think the right fix is a new self-recovery subsystem. The right fix is to close the gap so that the existing health signals tell the truth. Concretely, the broker (which is the only component that can observe "task channel is dead") should treat N consecutive task timeouts to a given runner as a connection failure and disconnect that runner. Once the connection drops, the existing launcher and orchestrator paths take over and recovery happens within seconds rather than hours.

This matches the shape of #25959 — surfacing real readiness/health on the existing endpoints, rather than building parallel mechanisms.

Happy to send a PR if this framing is acceptable.

To Reproduce

We have not been able to reduce this to a deterministic short test case. The conditions under which it occurs in our environment are:

• External runner mode on Cloud Run (sidecar container layout: nginx, n8n-main, n8n-runners on a single instance, maxScale=1, containerConcurrency=80).
• n8n-main env: N8N_RUNNERS_ENABLED=true, N8N_RUNNERS_MODE=external, N8N_RUNNERS_BROKER_LISTEN_ADDRESS=127.0.0.1, N8N_NATIVE_PYTHON_RUNNER=true, N8N_ENDPOINT_HEALTH=health (Cloud Run reserves /healthz, fixed in #25959).
• n8n-runners sidecar with N8N_RUNNERS_TASK_BROKER_URI=http://127.0.0.1:5679.
• Continuous Code-node traffic at any frequency, including very low (e.g. one Code node call every 2 minutes from a watchdog workflow).
• Sustained uptime of multiple hours.

Under these conditions in our environment, the stuck state has recurred on average once every ~1.3 days and persists 10–37 hours each time, until something external (deploy, OOM, panic) forces a container restart. Full Cloud Logging exports for all three windows are available on request.

Expected behavior

When the broker↔runner task channel becomes unable to deliver requests, n8n (or the launcher) should detect this within a small multiple of N8N_RUNNERS_TASK_TIMEOUT and recover automatically — either by resetting the broker connection for that runner or by terminating and restarting the runner — rather than allowing every Code node execution to time out indefinitely while the system reports healthy.

Debug Info

Debug info

core

• n8nVersion: 2.17.7
• platform: npm
• nodeJsVersion: 24.14.1
• nodeEnv: production
• database: postgres
• executionMode: regular
• concurrency: -1
• license: enterprise (production)
• consumerId: def9e2f5-a092-4761-8f72-48e3a8a75628

storage

• success: all
• error: all
• progress: false
• manual: true
• binaryMode: filesystem

pruning

• enabled: true
• maxAge: 336 hours
• maxCount: 10000 executions

client

• userAgent: mozilla/5.0 (macintosh; intel mac os x 10_15_7) applewebkit/537.36 (khtml, like gecko) chrome/147.0.0.0 safari/537.36
• isTouchDevice: false

Generated at: 2026-04-28T08:16:59.017Z

Operating System

unknown

n8n Version

2.15.0 and 2.17.7 (bug observed on both)

Node.js Version

24.14.1

Database

PostgreSQL

Execution mode

main (default)

Hosting

self hosted