openclaw - 💡(How to fix) Fix Stuck-session detector only logs, no abort path; wedges cascade into event-loop blocking [1 comments, 2 participants]

Summary

The gateway's stuck-session detector at dist/diagnostic-DitKp9ni.js only logs a warning when a session has been in processing state past diagnostics.stuckSessionWarnMs. There is no abort path. A wedged session stays wedged indefinitely until the operator hard-restarts the gateway, and during that window the rest of the gateway can pile up behind it (event-loop pressure, websocket pong timeouts, session-write-lock contention, restart-queue coalescing).

Reproduction

dist/diagnostic-DitKp9ni.js:340-368:

heartbeatInterval = setInterval(() => {
  const stuckSessionWarnMs = resolveStuckSessionWarnMs(heartbeatConfig);
  const now = Date.now();
  // ... pruning, memory sample, etc. ...
  for (const [, state] of diagnosticSessionStates) {
    const ageMs = now - state.lastActivity;
    if (state.state === "processing" && ageMs > stuckSessionWarnMs) logSessionStuck({
      sessionId: state.sessionId,
      sessionKey: state.sessionKey,
      state: state.state,
      ageMs
    });
  }
}, 3e4);

logSessionStuck (line ~277) only writes a warn log + emits a diagnostic event:

function logSessionStuck(params) {
  if (!areDiagnosticsEnabledForProcess()) return;
  const state = getDiagnosticSessionState(params);
  diagnosticLogger.warn(`stuck session: sessionId=... ageMs=${params.ageMs}`);
  emitDiagnosticEvent({ type: "session.stuck", ... });
  markDiagnosticActivity();
}

There is no call into the run controller, no abort, no signal — only telemetry.

Observed impact

On 2026-05-04 ~21:11–21:46 EDT this gateway wedged for ~2 hours with:

• 134% CPU sustained for 2+ hrs
• 5 Discord/Slack sessions stuck in processing state
• Oldest stuck session age: 883s
• [session-write-lock] releasing lock held for 18082ms (max=15000ms)
• Slack websockets disconnecting every ~30–45s (event-loop blocked, pong timeouts)
• A /restart issued at 21:45 was coalesced and never executed (the restart queue silently dropped it because a prior restart was still draining the same stuck sessions)

The user had no recovery path other than a hard process kill.

Proposed fix

Two-stage timeout: warn, then abort.

1. Add diagnostics.stuckSessionAbortMs config (default e.g. 600s; should be ≥ 2× stuckSessionWarnMs).

2. After a session crosses the abort threshold, call into the run controller to forcibly resolve the in-flight tool/LLM call with a stuck-session abort error:

   for (const [key, state] of diagnosticSessionStates) {
     const ageMs = now - state.lastActivity;
     if (state.state !== "processing") continue;
     if (ageMs > stuckSessionAbortMs) {
       abortStuckSession(state, { reason: "stuck-session-watchdog", ageMs });
     } else if (ageMs > stuckSessionWarnMs) {
       logSessionStuck({ ...state, ageMs });
     }
   }

3. abortStuckSession needs to:

   • Resolve any pending LLM call promise with an abort error (today there's a per-LLM-call 300s timeout; this should be a per-session ceiling that fires across tool transitions).
   • Mark the session idle so the queue can advance.
   • Emit a session.aborted diagnostic event with reason and age.
   • Be safe to call concurrently with normal completion (idempotent).

4. Restart queue: when a queued restart's first attempt fails to drain stuck sessions within N seconds, escalate to hard restart instead of silently coalescing subsequent /restart requests. Today, the second /restart during a wedge is dropped on the floor; this is the worst possible UX (operator thinks they triggered recovery and walks away).

Why this matters

The current state of the gateway is: stuck-session detection exists, has a configurable threshold, emits OTel metrics, and writes warning logs — but does not actually do anything to recover. From an operator's perspective, the warn log is misleading: it implies the system is monitoring and handling the condition, when it's only monitoring.

A stuckSessionAbortMs feature with a sensible default would make multi-session pile-up self-healing within ~10 minutes of onset, instead of indefinite.

Environment

• OpenClaw installed from npm at /opt/homebrew/lib/node_modules/openclaw/ (macOS, M-series, node v22)
• Diagnostic timestamp: 2026-05-05