openclaw - ✅(Solved) Fix tasks audit: 700+ inconsistent_timestamps warnings on cron tasks (startedAt < createdAt by 1ms) [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
openclaw/openclaw#70887Fetched 2026-04-24 10:38:16
View on GitHub
Comments
0
Participants
1
Timeline
1
Reactions
0
Author
Participants
Timeline (top)
cross-referenced ×1

After upgrading to 2026.4.22 (00bd2cf) and running openclaw tasks audit, 710 of 711 findings are inconsistent_timestamps warnings on cron-runtime tasks where startedAt is recorded as 1–4ms earlier than createdAt.

Error Message

"severity": "warn", openclaw tasks maintenance --apply does not clear the warnings (they are recomputed each audit). Filtering at the consumer with --severity error works for now.

Root Cause

After upgrading to 2026.4.22 (00bd2cf) and running openclaw tasks audit, 710 of 711 findings are inconsistent_timestamps warnings on cron-runtime tasks where startedAt is recorded as 1–4ms earlier than createdAt.

Fix Action

Workaround

openclaw tasks maintenance --apply does not clear the warnings (they are recomputed each audit). Filtering at the consumer with --severity error works for now.

Happy to attach a sanitized diagnostics bundle (openclaw gateway diagnostics export) if useful.

PR fix notes

PR #70891: fix(tasks): stop cron audit noise from 1ms startedAt/createdAt skew (#70887)

Description (problem / solution / changelog)

Summary

openclaw tasks audit was producing 700+ inconsistent_timestamps warnings per run, 99% of them startedAt exactly 1ms before createdAt on cron-runtime tasks. Root cause is two separate Date.now() reads: the cron runner captures startedAt = Date.now() just before invoking createRunningTaskRun(...), and createTaskRecord reads Date.now() again for createdAt. When the wall clock ticks between the two reads (very common under load), the audit's startedAt < createdAt check fires for every successful cron run, drowning out real warnings.

Fix is a one-branch monotonicity clamp inside createTaskRecord: when a caller supplies a startedAt that is strictly earlier than the registry's own Date.now(), anchor createdAt to that startedAt. This also keeps the natural semantics for retroactive records (task was started before the ledger row was persisted). Callers that do not supply startedAt (the common subagent/ACP/queued paths) are unchanged — createdAt still comes from Date.now().

Fixes #70887.

Root cause

// src/cron/service/timer.ts
const startedAt = state.deps.nowMs();          // first Date.now()
...
createRunningTaskRun({ ..., startedAt, lastEventAt: startedAt });
  -> createTaskRecord(...)
       const now = Date.now();                  // second Date.now(), 1-4ms later
       ...
       createdAt: now,
       startedAt: params.startedAt,             // < now on any ms tick

src/cron/service/ops.ts (manual cron runs) has the same shape, and is covered by the same single fix since both paths funnel through createTaskRecord.

Why this is safe

  • Only affects the branch where params.startedAt is explicitly supplied and strictly earlier than the in-function Date.now(); every other call site keeps the existing createdAt = now behavior.
  • cleanupAfter computation uses endedAt ?? lastEventAt ?? createdAt, and lastEventAt already defaulted to params.startedAt ?? now, so retention math stays consistent with the pre-fix behavior on the caller-supplied startedAt path.
  • No change to the audit check, to mergeExistingTaskForCreate, to persistence schema, or to the TaskRecord type shape.
  • Existing src/tasks/task-registry.test.ts (43 tests including the terminal-task prune test that passes startedAt: Date.now() - 9 * 24 * 60 * 60_000) continues to pass.

Security / runtime controls unchanged

  • No changes to owner/scope enforcement, delivery status, notify policy, or any permission/approval surface.
  • inconsistent_timestamps is an observability/debug aid in openclaw tasks audit, not a runtime security gate.
  • No prompt/text-level behavior was relied on; the fix is a pure write-time data-consistency clamp.
  • No CODEOWNERS-protected paths touched.

Testing

  • Added focused regression: src/tasks/task-registry.test.ts - "anchors createdAt to caller-supplied startedAt when the wall clock ticks between reads". Uses vi.spyOn(Date, 'now') to deterministically simulate the 1ms race, verifies (a) task.createdAt === startedAt, and (b) listTaskAuditFindings returns no inconsistent_timestamps findings for the record.
  • pnpm test src/tasks/task-registry.test.ts - 43/43 pass.
  • pnpm test src/tasks/task-registry.audit.test.ts src/tasks/task-registry.maintenance.issue-60299.test.ts src/tasks/detached-task-runtime.test.ts - 13/13 pass.
  • pnpm test:changed (origin/main base) - 116/116 pass across the tasks lane.
  • npx oxlint src/tasks/task-registry.ts src/tasks/task-registry.test.ts - 0 warnings, 0 errors.
  • pnpm format src/tasks/task-registry.ts src/tasks/task-registry.test.ts applied.
  • pnpm tsgo shows no new errors in the touched files; the remaining tsgo/lint:core failures (typebox module, plugin-sdk/*, gateway/server-methods/*, ui/*) reproduce on an unmodified origin/main checkout and are unrelated to this change.

Notes for reviewers

  • AI-assisted PR.
  • Lightly tested (unit-level regression + adjacent lanes green). Not exercised against a live cron runtime.
  • One changelog entry added under ## Unreleased > ### Fixes.

Made with Cursor

Changed files

  • CHANGELOG.md (modified, +1/-0)
  • src/tasks/task-registry.test.ts (modified, +36/-0)
  • src/tasks/task-registry.ts (modified, +8/-1)

Code Example

openclaw tasks audit --json | jq '[.findings[] | select(.code=="inconsistent_timestamps") | (.task.startedAt - .task.createdAt)] | group_by(.) | map({delta:.[0], count:length}) | sort_by(.delta)'

---

[
  { "delta": -4, "count": 2 },
  { "delta": -2, "count": 1 },
  { "delta": -1, "count": 706 }
]

---

{
  "kind": "task",
  "severity": "warn",
  "code": "inconsistent_timestamps",
  "detail": "startedAt is earlier than createdAt",
  "status": "succeeded",
  "task": {
    "runtime": "cron",
    "label": "Trading cycle (PROD)",
    "createdAt": 1776393146131,
    "startedAt": 1776393146130,
    "endedAt": 1776393146131
  }
}
RAW_BUFFERClick to expand / collapse

Summary

After upgrading to 2026.4.22 (00bd2cf) and running openclaw tasks audit, 710 of 711 findings are inconsistent_timestamps warnings on cron-runtime tasks where startedAt is recorded as 1–4ms earlier than createdAt.

Environment

  • openclaw 2026.4.22 (00bd2cf)
  • Linux x86_64, Node 22.22.2
  • Gateway running under supervisord (LXC, no time skew on host; chrony synced)

Reproduction

openclaw tasks audit --json | jq '[.findings[] | select(.code=="inconsistent_timestamps") | (.task.startedAt - .task.createdAt)] | group_by(.) | map({delta:.[0], count:length}) | sort_by(.delta)'

Output from a healthy production gateway:

[
  { "delta": -4, "count": 2 },
  { "delta": -2, "count": 1 },
  { "delta": -1, "count": 706 }
]

99.4% of warnings are exactly -1ms.

Sample finding

{
  "kind": "task",
  "severity": "warn",
  "code": "inconsistent_timestamps",
  "detail": "startedAt is earlier than createdAt",
  "status": "succeeded",
  "task": {
    "runtime": "cron",
    "label": "Trading cycle (PROD)",
    "createdAt": 1776393146131,
    "startedAt": 1776393146130,
    "endedAt": 1776393146131
  }
}

Hypothesis

For cron-launched tasks the bookkeeping records createdAt and startedAt from two separate Date.now() calls. When the wall clock advances between the two reads, the order can invert by 1ms. Likely the cron dispatcher writes startedAt first (when the task object is constructed in the runner) and the queue layer then stamps createdAt slightly later (when persisting to the audit log).

This generates a permanent stream of warnings for every successful cron run, drowning out real warnings.

Suggested fix

Either:

  1. Treat a negative delta of ≤ a small threshold (e.g., 5ms) as not a warning, OR
  2. Force startedAt = max(startedAt, createdAt) at write time, OR
  3. Capture both timestamps in a single const now = Date.now() and reuse.

Workaround

openclaw tasks maintenance --apply does not clear the warnings (they are recomputed each audit). Filtering at the consumer with --severity error works for now.

Happy to attach a sanitized diagnostics bundle (openclaw gateway diagnostics export) if useful.

extent analysis

TL;DR

Implement one of the suggested fixes to ignore or correct the minor timestamp discrepancies causing the inconsistent_timestamps warnings.

Guidance

  • Investigate implementing a threshold for ignoring small negative deltas (e.g., ≤ 5ms) to reduce noise from these warnings.
  • Consider modifying the code to ensure startedAt is not earlier than createdAt by setting startedAt = max(startedAt, createdAt) at write time.
  • Evaluate capturing both timestamps with a single Date.now() call to prevent future discrepancies.
  • As a temporary workaround, filter warnings by severity using --severity error when consuming the audit output.

Example

No code example is provided due to the lack of specific implementation details in the issue.

Notes

The suggested fixes aim to address the root cause of the warnings, which appears to be related to the timing of Date.now() calls. However, the best approach may depend on the specific requirements and constraints of the system.

Recommendation

Apply workaround: Filtering at the consumer with --severity error is a viable temporary solution until one of the suggested fixes can be implemented, as it allows for the suppression of the noise from these specific warnings without affecting the overall functionality of the system.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

openclaw - ✅(Solved) Fix tasks audit: 700+ inconsistent_timestamps warnings on cron tasks (startedAt < createdAt by 1ms) [1 pull requests, 1 participants]