hermes - 💡(How to fix) Fix Feature: Automatic retry for failed cron jobs

Problem

When a cron job fails (provider hiccup, rate limit, transient error), it records the error and waits for the next scheduled run. For daily jobs, that means waiting 24 hours because of a one-time blip.

Real-world example: A daily ParentVUE grade monitor failed because ZAI rate-limited simultaneously with another cron job, fell back to the DeepSeek fallback model, and hit a thinking-mode context compaction bug. The job won't retry until tomorrow at the same time — a full day of missed grade monitoring because of a transient concurrency spike.

Current Behavior

• Failed jobs record last_status = "error" and advance next_run_at normally
• No retry_on_failure, max_retries, retry_delay, or backoff config exists
• The job schema has no retry-related fields
• Only manual retry via cronjob action=run or gateway restart (one-shot only)

Proposed Feature

Add configurable retry-on-failure for cron jobs:

Schema additions:

retry:
  enabled: true
  max_retries: 3          # Max retry attempts before giving up
  delay: 5m               # Initial delay before first retry
  backoff: exponential     # "exponential" or "linear" or "fixed"
  max_delay: 30m           # Cap on retry delay

Behavior:

• On failure, schedule a retry after delay (with optional backoff)
• If retry also fails, try again up to max_retries times
• Each retry attempt is logged in the job record
• If all retries exhausted, record final error and advance next_run_at as normal
• Successful retry clears the error state

Default: retry: enabled: false — backward compatible, no behavior change for existing jobs

Why This Can't Be a Plugin

The cron scheduler's tick cycle (tick() → _process_job() → run_job() → mark_job_run()) is core engine code. Retry scheduling requires modifying mark_job_run() and advance_next_run() to support re-queuing failed jobs with computed delay — this is fundamental scheduler behavior, not something a plugin can intercept.

Alternative Approaches Considered

1. Wrapper script with retry logic — works for no_agent=True script jobs but not for LLM-driven agent cron jobs
2. External watchdog — defeats the purpose of built-in scheduling
3. More granular scheduling — running a daily job every hour "in case it fails" wastes resources and produces duplicate output

Environment

• Hermes Agent v0.14.x
• Multiple daily cron jobs (homework, grade monitor, weather, journal, email)
• Flat-rate provider (ZAI) with occasional concurrency rate limiting that triggers fallback to thinking-mode models