PR #63675: fix(cron): replay interrupted one-shot jobs on startup recovery

• Repository: openclaw/openclaw
• Author: hclsys
• State: closed | merged: False
• Link: https://github.com/openclaw/openclaw/pull/63675

Description (problem / solution / changelog)

Summary

One-shot cron jobs (schedule.kind === "at") that were mid-execution when the gateway restarted were silently lost: runningAtMs got cleared but the job was then skipped instead of retried, with 0 runs recorded and no notification delivered. Reminders that happened to fire during a model switch, container restart, or update disappeared without any user-visible signal.

Fixes #63657

Root cause

In src/cron/service/ops.ts start(), the startup recovery path used to collect any job with a stale runningAtMs marker into a local interruptedOneShotIds set and pass that set as skipJobIds to runMissedJobs, but the collection filter was gated on schedule.kind === "at":

```ts // Before (ops.ts:106-134) const interruptedOneShotIds = new Set<string>(); // ... if (typeof job.state.runningAtMs === "number") { // ... job.state.runningAtMs = undefined; // One-shot jobs are not retried after interruption; recurring jobs // (cron/every) are eligible for startup catch-up so they don't // require a second restart to recover (#60495). if (job.schedule.kind === "at") { interruptedOneShotIds.add(job.id); } } // ... await runMissedJobs(state, { skipJobIds: interruptedOneShotIds.size > 0 ? interruptedOneShotIds : undefined, }); ```

The existing comment acknowledged recurring jobs are eligible via #60583 and one-shots were deliberately excluded — but the excluded path leaves reminders irrecoverable. A reminder that fires exactly at docker restart, openclaw update, or a model-switch reload is lost forever with no log line beyond the generic "clearing stale running marker".

Fix

Drop the one-shot skip entirely and let runMissedJobs decide. The existing skipAtIfAlreadyRan guard in runMissedJobs -> isRunnableJob (src/cron/service/timer.ts:850-866) already distinguishes "interrupted before completion" from "already completed" by checking job.state.lastStatus:

• runningAtMs and lastStatus are written in the same atomic block at timer.ts:408-411. Either both are cleared (successful completion) or neither is touched (crash mid-execution).
• Interrupted one-shot: lastStatus === undefined → skipAtIfAlreadyRan check at timer.ts:850 falls through to the normal nowMs >= nextRunAtMs overdue check → retries ✅
• Completed one-shot (defensive — e.g. crash between settle and flush leaves stale runningAtMs): lastStatus === \"ok\" → timer.ts:850-866 returns false → skipped ✅
• Failed one-shot with retry eligibility: lastStatus === \"error\" + nextRun > lastRun → retries per timer.ts:856-864 ✅
• Failed one-shot without retry: lastStatus === \"error\" + no retry window → skipped ✅

This preserves the invariant protected by PR #56509 (no double-delivery of already-completed one-shots) while fixing the symmetric data-loss bug.

Tests

Two new cases in src/cron/service/ops.test.ts, mirroring the existing #60495 recurring-interrupted startup test:

1. replays interrupted one-shot jobs on startup recovery (#63657)

   • Setup: one-shot at: job with runningAtMs set, nextRunAtMs in the past, lastStatus unset
   • Action: start(state)
   • Asserts: stale marker cleared, enqueueSystemEvent called, lastStatus === \"ok\", lastRunAtMs === now

2. does not re-run a completed one-shot with stale runningAtMs on startup (#63657) (the safety guard)

   • Setup: one-shot at: job with both runningAtMs AND lastStatus: \"ok\" (simulates a crash between settle and flush)
   • Action: start(state)
   • Asserts: stale marker cleared, enqueueSystemEvent NOT called, lastStatus and lastRunAtMs preserved

The second test is the critical regression guard: it proves the fix doesn't reintroduce the #56509-class of one-shot double-delivery bugs.

Precedent

• #60583 (merged, joelnishanth): fix(cron): resume interrupted recurring jobs on first restart — the identical fix pattern for recurring jobs. Same catch-up path, same runMissedJobs entry point. Referenced in the existing ops.ts comment.
• #56509 (open, claygeo): fix(cron): prevent one-shot at jobs from re-triggering after completion — orthogonal to this PR. #56509 fixes completed one-shots re-triggering from onTimer/runDueJobs paths that don't pass skipAtIfAlreadyRan. This PR works through runMissedJobs which DOES pass skipAtIfAlreadyRan: true (timer.ts:970), so both fixes coexist.

Scope

• Files: src/cron/service/ops.ts (-10, +10), src/cron/service/ops.test.ts (+143)
• LOC: <20 production changes, no new functions, no new abstractions
• No changes to timer.ts, to isRunnableJob, or to the settle path — the safety comes from existing guards
• oxlint clean

cc @steipete — cron service startup recovery, same area as #60583.

Credit to @myradon for the precise RCA in #63657 — the issue body named the exact file, the exact skip logic, and the exact symptom, which made this a ~20 LOC fix with a 143-line test suite.

Changed files

• src/cron/service/ops.test.ts (modified, +143/-0)
• src/cron/service/ops.ts (modified, +10/-10)