openclaw - 💡(How to fix) Fix Heartbeat scheduler chain permanently broken when requests-in-flight returned [1 comments, 1 participants]

Summary

When runOnce() returns { status: "skipped", reason: "requests-in-flight" }, the heartbeat scheduler's scheduleNext() is never called again, permanently breaking the heartbeat chain for all agents until the gateway is restarted.

Root Cause

In the heartbeat runner's run() function (line ~989-1060):

let requestsInFlight = false;
try {
    for (const agent of state.agents.values()) {
        // ...
        if (res.status === "skipped" && res.reason === "requests-in-flight") {
            requestsInFlight = true;
            return res;  // ← exits the for-loop AND the try block
        }
        // ...
    }
} finally {
    if (!requestsInFlight) scheduleNext();  // ← skipped when requestsInFlight = true
}

The logic intends to avoid scheduling the next heartbeat while requests are still in flight. However, nothing ever re-invokes scheduleNext() after the in-flight requests complete. The chain breaks permanently.

Expected behavior

When requests are in-flight, the scheduler should defer scheduleNext() but ensure it fires once the in-flight requests complete (or after a safety timeout).

Actual behavior

scheduleNext() is never called again. The heartbeat scheduler goes permanently silent. No error is logged. The only recovery is a full gateway restart.

Reproduction

1. Configure multiple agents with heartbeats (at least 3-4)
2. Have an agent's heartbeat trigger a long-running session (e.g., enriched heartbeat with file reads and graph operations)
3. While that session is running, trigger another heartbeat interval tick
4. The second tick sees requests-in-flight, sets requestsInFlight = true, and returns early
5. scheduleNext() is skipped in the finally block
6. No future heartbeats fire for any agent

Environment: OpenClaw 2026.4.5 (3e72c03), 14-agent mesh with mixed heartbeat intervals (1h, 2h, 4h). Observed 4 times in 72 hours. More likely when multiple enriched heartbeats have overlapping windows.

Suggested Fix

Option A: Safety-net timeout (minimal change)

} finally {
    if (!requestsInFlight) {
        scheduleNext();
    } else {
        setTimeout(() => {
            scheduleNext();
        }, 30_000); // 30s — enough for most in-flight requests to complete
    }
}

Option B: Always schedule (if scheduleNext is idempotent)

} finally {
    scheduleNext(); // Always schedule. If requests are in-flight, next tick will check again.
}

Impact

• Silent heartbeat failure across entire mesh
• No error logged — extremely difficult to diagnose without reading source
• Only recovery is gateway restart
• Affects any deployment with multiple agents and overlapping heartbeat windows
• More likely to trigger under load (exactly when heartbeats are most needed)

Workaround

openclaw gateway restart

Filed from a 14-agent OpenClaw deployment. Reproduction confirmed across 4 incidents in 72 hours.