hermes - ✅(Solved) Fix [Bug]: Cron jobs double-fire after timezone offset migration [3 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
NousResearch/hermes-agent#28934Fetched 2026-05-20 04:01:03
View on GitHub
Comments
0
Participants
1
Timeline
8
Reactions
0
Participants
Timeline (top)
labeled ×5cross-referenced ×3

Error Message

Additional Logs / Traceback (optional)

Root Cause

Root Cause Analysis (optional)

Fix Action

Fix / Workaround

The scheduler compares the persisted old-offset instant after converting it to the current local timezone, treats it as already due, dispatches it early, and then later schedules/fires the same cron occurrence again at the correct wall-clock time.

Not attached: this is a source-level regression with a deterministic unit-test reproducer. The new regression test constructs the persisted jobs.json state directly and monkeypatches _hermes_now() to the migrated timezone.

A PR is ready. The fix detects aware cron next_run_at values whose stored UTC offset differs from current Hermes time and whose converted instant appears due while the stored wall-clock time is still in the future. In that migration case it recomputes next_run_at from the current Hermes timezone and skips dispatch for that tick. It also preserves the existing stale-run fast-forward behavior when the stored wall-clock time has already passed.

PR fix notes

PR #28941: fix(cron): repair migrated timezone offsets

Description (problem / solution / changelog)

Summary

  • prevent cron jobs from firing early after persisted next_run_at timezone offsets differ from the current Hermes timezone
  • repair affected cron jobs by recomputing their next wall-clock occurrence before dispatch
  • add regression coverage for migrated-offset double-fire behavior and stale already-passed wall-clock behavior

Related issue

Fixes #28934

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Refactor / cleanup
  • Test-only change

Changed files

  • cron/jobs.py
  • tests/cron/test_jobs.py

Test plan

  • Proved regression test fails without the production fix by temporarily restoring cron/jobs.py from upstream/main
  • scripts/run_tests.sh tests/cron375 passed
  • python -m py_compile cron/jobs.py tests/cron/test_jobs.py
  • git diff --check

Full suite note: I attempted the canonical full scripts/run_tests.sh after installing .[all,dev], but this sandbox was killed with exit 137 around ~30% of the ~24k-test run. The focused cron suite and syntax/diff checks passed, and the failure mode appears to be the sandbox resource ceiling rather than this patch.

Details

When a Hermes deployment changes timezone offset, persisted cron next_run_at values can retain the old offset. Example: a job scheduled for 21:00+10 becomes 13:00+02 as an absolute instant. At 13:02+02, the scheduler sees the job as due and dispatches it early, then the next cron calculation schedules the intended 21:00+02 wall-clock run as well.

This patch detects that migrated-offset case narrowly:

  • only for cron-triggered jobs
  • only when the stored aware offset differs from the current Hermes offset
  • only when the converted instant appears due but the stored local wall-clock time is still in the future

In that case, Hermes repairs next_run_at to the current timezone's next wall-clock occurrence and skips dispatch for the current tick. Already-passed stale wall-clock jobs keep the existing fast-forward behavior.

Checklist

  • I have read the contributing guidelines
  • I have added tests that prove the fix
  • I have run relevant tests locally
  • I have kept the change minimal and scoped

Changed files

  • cron/jobs.py (modified, +48/-3)
  • tests/cron/test_jobs.py (modified, +70/-0)

PR #28951: fix(cron): Normalize timestamps to UTC to prevent double-firing after timezone migration

Description (problem / solution / changelog)

PR: Fix Cron Double-Firing After Timezone Migration (#28934)

Summary

Fixes a sticky bug where cron jobs could double-fire after system timezone changes, DST transitions, or config/profile migrations.

Root Cause

_ensure_aware() interpreted naive timestamps using the current system local timezone. After any timezone shift, previously stored last_run_at / next_run_at values were re-interpreted, causing the scheduler to incorrectly consider jobs due again.

Changes

  1. Core Fix (cron/jobs.py): Normalize all cron timestamps to UTC (industry standard).
  2. Migration Helper: Added _migrate_timestamps_to_utc() — safe, idempotent, non-destructive migration for legacy naive timestamps.
  3. Tests: Added TestCronTimezoneMigration covering naive and aware timestamp handling.

Testing

  • All new tests pass.
  • Existing cron test suite remains green.
  • Manual verification: jobs no longer double-fire under simulated timezone shifts.

Security & Standards

  • No new attack surface (pure datetime normalization).
  • Follows existing error-handling and logging patterns.
  • Migration is backward-compatible and non-destructive.
  • Changes are minimal and highly auditable.

Linked Issue

Closes #28934

Commits on branch

  • fix(cron): Normalize timestamps to UTC to prevent double-firing after timezone migration
  • test(cron): Add regression tests for UTC timestamp normalization (#28934)
  • refactor(cron): Add safe UTC migration helper for legacy timestamps (#28934)

Changed files

  • cron/jobs.py (modified, +32/-13)
  • tests/cron/test_compute_next_run_last_run_at.py (modified, +33/-0)

PR #28985: fix(cron): repair migrated timezone offsets

Description (problem / solution / changelog)

Summary

  • Detect persisted cron next_run_at values that still use an old UTC offset after Hermes timezone migration
  • Rebase those timestamps to the current Hermes timezone while preserving the intended cron wall-clock time
  • Keep stale-run fast-forward behavior intact so missed schedules do not backfill unexpectedly

Fixes #28934

Test Plan

  • python -m pytest tests/cron/test_jobs.py::TestGetDueJobs -q -o addopts= — 10 passed
  • python -m pytest tests/cron/test_jobs.py -q -o addopts= — 81 passed
  • python -m pytest tests/cron -q -o addopts= — 375 passed

Changed files

  • cron/jobs.py (modified, +63/-1)
  • tests/cron/test_jobs.py (modified, +70/-0)
RAW_BUFFERClick to expand / collapse

Bug Description

Recurring cron jobs can fire early and then fire again at the intended wall-clock time after the Hermes/system timezone changes while next_run_at still contains the old UTC offset.

This is a recurrence of the symptom reported in #24289, but with a confirmed persisted-offset migration reproducer: for example, a weekly 0 21 * * 2 job persisted as 2026-05-19T21:00:00+10:00 is evaluated after the runtime moves to +02:00. The scheduler converts that instant to 13:00+02, sees it as due at 13:02+02, runs it early, then advances it to the same day 21:00+02, producing a second run.

Steps to Reproduce

  1. Persist a recurring cron job with an old local offset, e.g. schedule 0 21 * * 2 and next_run_at: 2026-05-19T21:00:00+10:00.
  2. Run Hermes after the local/configured timezone has moved to +02:00.
  3. Evaluate due jobs at 2026-05-19T13:02:00+02:00.
  4. Observe the stored next_run_at converts to 13:00+02:00, so get_due_jobs() returns the job even though the local wall-clock cron time is still 21:00.

Expected Behavior

The scheduler should preserve local wall-clock cron intent across timezone-offset migration. A future local 21:00 run should be repaired/rescheduled to 21:00 in the current Hermes timezone and should not be returned as due at 13:02.

Actual Behavior

The scheduler compares the persisted old-offset instant after converting it to the current local timezone, treats it as already due, dispatches it early, and then later schedules/fires the same cron occurrence again at the correct wall-clock time.

Affected Component

  • Gateway (Telegram/Discord/Slack/WhatsApp)
  • Configuration (config.yaml, .env, hermes setup)
  • Other: cron scheduler

Messaging Platform (if gateway-related)

Discord observed, but the bug is in cron scheduling and is platform-independent.

Debug Report

Not attached: this is a source-level regression with a deterministic unit-test reproducer. The new regression test constructs the persisted jobs.json state directly and monkeypatches _hermes_now() to the migrated timezone.

Operating System

Observed on macOS host runtime; reproduced in Linux test environment.

Python Version

Python 3.11.15 in the test environment.

Hermes Version

Current upstream/main at the time of filing.

Additional Logs / Traceback (optional)

Related historical symptom: #24289. Related but distinct timezone feature work: #26549 and PR #27393 (CRON_TZ per-schedule support). This bug does not require CRON_TZ; it happens with existing persisted next_run_at offsets after the global/runtime timezone changes.

Root Cause Analysis (optional)

cron/jobs.py::_get_due_jobs_locked() parses persisted next_run_at, normalizes it through _ensure_aware(), and compares the converted absolute instant to _hermes_now(). If next_run_at was persisted with an old offset, e.g. 21:00+10, and current Hermes time is +02, that instant becomes 13:00+02. At 13:02+02 the scheduler thinks the job is due, even though the stored local cron wall time (21:00) has not arrived in the current timezone.

Proposed Fix (optional)

A PR is ready. The fix detects aware cron next_run_at values whose stored UTC offset differs from current Hermes time and whose converted instant appears due while the stored wall-clock time is still in the future. In that migration case it recomputes next_run_at from the current Hermes timezone and skips dispatch for that tick. It also preserves the existing stale-run fast-forward behavior when the stored wall-clock time has already passed.

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING