openclaw - 💡(How to fix) Fix Zombie embedded run blocks heartbeats permanently after provider timeout [1 comments, 2 participants]

Summary

When an embedded run times out but the underlying provider promise never settles (e.g., dead connection, hung HTTP stream), the run handle stays in ACTIVE_EMBEDDED_RUNS permanently. Since resolveActiveRunQueueAction returns "drop" for heartbeats when isActive === true, all subsequent heartbeat ticks are silently discarded until the gateway is restarted.

Reproduction

1. Configure a heartbeat that shares a session with the main agent (default behavior, no isolatedSession)
2. Trigger a condition where the provider connection hangs (e.g., a 404 from a misconfigured Azure endpoint, or a network interruption)
3. Wait for the embedded run timeout (default 600s) to fire
4. Observe that ACTIVE_EMBEDDED_RUNS still contains the zombie handle after timeout
5. All subsequent heartbeat ticks are silently dropped

Root Cause

In src/agents/pi-embedded-runner/run/attempt.ts, clearActiveEmbeddedRun is correctly placed in a finally block:

} finally {
    clearTimeout(abortTimer);
    // ...
    clearActiveEmbeddedRun(params.sessionId, queueHandle, params.sessionKey);
}

However, this finally only executes when the await abortable(activeSession.prompt(...)) promise resolves or rejects. If the abort signal fires but the provider stream/promise never settles (hung connection, broken pipe without TCP reset), the finally block never runs.

The abort timer at the outer scope fires and calls abortRun(true), which signals the AbortController. But if the underlying HTTP client doesn't honor the abort (or the connection is in a state where it can't be interrupted), the promise hangs indefinitely.

Meanwhile, in src/auto-reply/reply/queue-policy.ts:

export function resolveActiveRunQueueAction(params) {
  if (!params.isActive) return "run-now";
  if (params.isHeartbeat) return "drop";  // <-- heartbeat silently killed
  // ...
}

Since the zombie handle keeps isEmbeddedPiRunActive(sessionId) returning true, every heartbeat tick hits "drop" and exits without any log message.

Observed Behavior

From gateway logs (v2026.3.13-1):

2026-03-21T13:08:05 [agent/embedded] embedded run timeout: runId=620fd6bf sessionId=50d88d7d timeoutMs=600000
2026-03-21T13:08:13 [diagnostic] wait for active embedded runs timed out: activeRuns=1 timeoutMs=90000
2026-03-21T17:34:03 [diagnostic] wait for active embedded runs timed out: activeRuns=1 timeoutMs=90000
2026-03-21T19:35:41 [diagnostic] wait for active embedded runs timed out: activeRuns=1 timeoutMs=90000
2026-03-21T23:08:57 [diagnostic] wait for active embedded runs timed out: activeRuns=1 timeoutMs=90000

The zombie run persisted for 10+ hours until the machine was shut down. During that entire period, no heartbeat was delivered.

Suggested Fix

After the abort timer fires and a grace period elapses (e.g., 30-60s), forcibly remove the handle from ACTIVE_EMBEDDED_RUNS regardless of whether the promise settled:

// In the abort timer callback, after abortRun(true):
setTimeout(() => {
  if (ACTIVE_EMBEDDED_RUNS.get(params.sessionId) === queueHandle) {
    ACTIVE_EMBEDDED_RUNS.delete(params.sessionId);
    notifyEmbeddedRunEnded(params.sessionId);
    log.warn(`force-cleared zombie run: sessionId=${params.sessionId} runId=${params.runId}`);
  }
}, ZOMBIE_CLEANUP_GRACE_MS); // e.g., 30_000

Alternatively, waitForActiveEmbeddedRuns (used during restarts) could forcibly clear runs that have exceeded their timeout, rather than just logging and returning { drained: false }.

Environment

• OpenClaw v2026.3.13-1
• Provider: azure-openai-responses (Azure OpenAI Responses API)
• macOS, Node v22.22.0

Related

• #51735 (azure-openai-responses missing from MODEL_APIS enum — the misconfiguration that triggered the initial 404s)
• The "drop" path in resolveActiveRunQueueAction produces no log output, making this failure mode completely silent and very hard to diagnose