hermes - 💡(How to fix) Fix cron: no health monitoring for cron ticker — can silently stop for hours [1 participants]

There is no mechanism to detect when the cron ticker stops processing jobs. If scheduler.py has a syntax error, an import failure, or the ticker encounters an unhandled exception, all scheduled jobs silently stop running — but the gateway continues to operate normally with no alert.

1. Introduce a syntax error in cron/scheduler.py (e.g. a bad f-string)
2. Check gateway.log — no critical error about cron ticker failure High severity for production deployments. All automated tasks (backups, health checks, scheduled reports, data cleanup) silently stop. In one observed case, a syntax error caused the cron ticker to stop for 7.5 hours with zero alerts — all cron jobs did not run during that window.

TL;DR

Implement preflight checks at startup and runtime heartbeat monitoring to detect and alert when the cron ticker stops processing jobs.

Guidance

• Introduce a preflight check in the gateway startup sequence to verify that cron/scheduler.py can be compiled and imported without errors before starting the cron ticker.
• Implement a runtime heartbeat monitoring mechanism to periodically check the cron ticker's status and emit a critical alert if it appears dead for an extended period (e.g., 30 minutes).
• Modify the gateway.log to include critical error messages when the cron ticker fails to start or stops running unexpectedly.
• Consider adding a fallback mechanism to restart the cron ticker or trigger an alert when it fails to run for an extended period.

Example

import importlib.util
import os

def check_cron_scheduler():
    file_path = 'cron/scheduler.py'
    if not os.path.exists(file_path):
        print("Critical: Cron scheduler file not found.")
        return False
    
    spec = importlib.util.spec_from_file_location("scheduler", file_path)
    if spec is None:
        print("Critical: Failed to import cron scheduler.")
        return False
    
    try:
        module = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(module)
    except Exception as e:
        print(f"Critical: Error importing cron scheduler: {e}")
        return False
    
    return True

Notes

The provided example is a basic illustration of how to implement a preflight check for the cron scheduler. The actual implementation may vary depending on the specific requirements and the existing codebase.

Recommendation

Apply workaround: Implement the suggested preflight checks and runtime monitoring to detect and alert when the cron ticker stops processing jobs, as this will provide an immediate solution to the problem.