Bug Description

The cron ticker thread inside the gateway died silently with zero log output and remained dead for 15+ hours. The gateway process itself stayed fully functional (Telegram, Discord, memory monitor all running), but all scheduled cron jobs stopped firing. hermes cron status misleadingly reported "jobs will fire automatically" because it only checks whether the gateway process exists — not whether the ticker thread is alive.

Timeline (from gateway.log)

May 25 13:51:38  Cron ticker started (interval=60s)   ← last ticker log
May 25 13:56:26  Memory monitor: rss=143MB             ← gateway alive
May 25 18:11:08  Telegram "Bad Gateway" (recovered)    ← gateway alive
May 26 00:01+    Memory monitor logging continuously   ← gateway alive all night
May 26 05:24:31  Gateway restart → Cron ticker started ← ticker back after manual restart

15.5 hours of complete ticker silence — no ticks, no errors logged, no "stopped" message. The gateway process never crashed. Memory monitor kept logging every 5 minutes the entire time.

Root Cause

Three things combine to make this fatal:

1. Errors are logged at DEBUG level (invisible by default)

# gateway/run.py line 17678-17682
while not stop_event.is_set():
    try:
        cron_tick(verbose=False, adapters=adapters, loop=loop)
    except Exception as e:
        logger.debug("Cron tick error: %s", e)   # ← DEBUG level only!

When something goes wrong inside cron_tick(), the error message is swallowed because the default Gateway log level is INFO. The operator sees nothing.

2. No BaseException catch

If cron_tick() (or any of the periodic tasks that follow) raises a BaseException subclass (SystemExit, KeyboardInterrupt, or something from a C extension), the except Exception block won't catch it. The thread dies immediately with no log.

3. No watchdog or health check

Nothing monitors whether the ticker thread is still alive. hermes cron status calls _get_gateway_pid() and reports "jobs will fire automatically" if ANY gateway process is found — it never checks whether the ticker thread is running.

# hermes cron status output while ticker was dead:
✓ Gateway is running — cron jobs will fire automatically
  PID: 567
  1 active job(s)
  Next run: 2026-05-26T07:00:00-07:00

This status was shown at 08:17 when the ticker had been dead for 15+ hours.

Steps to Reproduce

Hard to reproduce deterministically since the ticker died without an error trace. Setup that triggered it:

1. Gateway running with Telegram + Discord adapters
2. Cron job configured: no_agent=true, script-based, 0 7-19 * * *
3. Gateway survived multiple DNS-triggered restarts earlier in the day (logs show SIGTERM → restart → ticker restarted each time)
4. After the last restart at 13:51, ticker started, then silently died within 60 seconds with no error logged
5. macOS 26.5, Python 3.11, running via systemd user service

Impact

• All scheduled cron jobs silently stop firing
• No alert, no notification, no error log
• hermes cron status reports healthy
• Operator only discovers the failure when they notice a scheduled job didn't deliver (hours later, in this case)

Proposed Fixes

Immediate (low risk)

1. Log tick errors at WARNING level, not DEBUG:

   except Exception as e:
       logger.warning("Cron tick error: %s", e)

2. Catch BaseException to ensure the ticker gets a chance to log before dying:

   except BaseException as e:
       logger.error("Cron ticker fatal error: %s", e, exc_info=True)
       raise

Medium term

1. Add a ticker liveness watchdog: track last_tick_at timestamp. If no tick for >2× interval, log an ERROR, attempt restart of the ticker thread, and notify the operator via home channel.

2. Fix hermes cron status: instead of just checking _get_gateway_pid(), also verify the ticker thread is alive (check last_tick_at from a shared timestamp, or inspect the thread via threading.enumerate()).

Related

Possibly related to #23491 (Feishu WebSocket disconnect kills cron ticker), but this case is distinct: no platform disconnect occurred, the gateway was fully functional, and the ticker died silently with no error log.

Environment

• OS: macOS 26.5
• Python: 3.11
• Gateways: Telegram + Discord
• Cron job type: no_agent=true, script-based
• Schedule: 0 7-19 * * *