PR #24998: fix(agent): add circuit-breaker to _try_activate_fallback to prevent tight retry loops

• Repository: NousResearch/hermes-agent
• Author: drabekj
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/24998

Description (problem / solution / changelog)

Summary

Adds a circuit-breaker to AIAgent._try_activate_fallback so that when every provider in the fallback chain returns non-retryable errors back-to-back, the function self-throttles and ultimately trips, preventing host memory exhaustion from a runaway retry loop.

Two protections, both at the top of the function:

1. Throttle: enforce ≥2s between consecutive activations
2. Breaker: ≥5 activations in a 60s rolling window → return False, which existing call sites already handle as "chain exhausted"

Thresholds exposed as class constants (_FALLBACK_THROTTLE_INTERVAL_S, _FALLBACK_BREAKER_WINDOW_S, _FALLBACK_BREAKER_LIMIT) for easy tuning.

Motivation

Refs #24996.

When the primary provider returns a non-retryable error (HTTP 402 — depleted credits, HTTP 400 — request rejected, etc.) and the fallback chain has providers that themselves fail non-retryably (rate-limited, model-incompatible request shape), the existing code resets retry_count = 0 and continues the outer loop. There is no minimum interval between activations and no cap. On a large accumulated session context (~80k tokens), each iteration re-marshals the context, and hundreds of activations per second can pile up in under a second of wall-clock time, exhausting RAM + swap on constrained hosts (single-board / self-hosted setups).

The existing #1630 guard ("Skipping session persistence for large failed session") prevents context growth across persisted-restart cycles. This PR addresses a complementary failure mode: rapid in-process activation churn.

How it works

_FALLBACK_THROTTLE_INTERVAL_S = 2.0
_FALLBACK_BREAKER_WINDOW_S = 60.0
_FALLBACK_BREAKER_LIMIT = 5

def _try_activate_fallback(self, reason=None) -> bool:
    # ... docstring ...
    if not hasattr(self, "_fallback_activations"):
        self._fallback_activations = []
    now = time.monotonic()
    # Drop activations older than the breaker window
    self._fallback_activations = [t for t in self._fallback_activations
                                  if now - t < self._FALLBACK_BREAKER_WINDOW_S]
    if len(self._fallback_activations) >= self._FALLBACK_BREAKER_LIMIT:
        self._emit_status("🛑 Circuit-breaker tripped: ...")
        return False
    if self._fallback_activations:
        elapsed = now - self._fallback_activations[-1]
        if elapsed < self._FALLBACK_THROTTLE_INTERVAL_S:
            time.sleep(self._FALLBACK_THROTTLE_INTERVAL_S - elapsed)
    self._fallback_activations.append(time.monotonic())
    # ... existing logic (rate-limit cooldown etc.) continues ...

Return value False reaches existing handlers cleanly — every if self._try_activate_fallback(): continue call site already has the fall-through "fallback exhausted" path.

Notes / open questions

• Recursive intra-chain skips (e.g., the existing return self._try_activate_fallback() for invalid/duplicate entries) will increment the activation counter under this implementation. In practice this is harmless: those recursive calls are cheap (no API call) and rare (mis-configured fallback entries). If maintainers prefer, the counter could be scoped to only external entries via a recursion-depth flag.
• No tests added yet — happy to add one modeled on tests/run_agent/test_1630_context_overflow_loop.py if the approach is accepted.
• Thresholds are conservative defaults (2s interval, 5/60s limit). Tuning may be warranted based on the expected legitimate fallback velocity.

Closes / Refs

Refs #24996.

Changed files

• run_agent.py (modified, +35/-0)

PR #25059: fix(run_agent): circuit-breaker on fallback activations to prevent memory storm (fixes #24996)

• Repository: NousResearch/hermes-agent
• Author: shanewas
• State: closed | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/25059

Description (problem / solution / changelog)

Fixes #24996

What broke

When all fallback providers fail back-to-back (402, 429, 400), _try_activate_fallback() is called with no interval and no cumulative cap. With large sessions (~80k tokens, 100+ messages), this produces a tight loop that re-marshals full context per attempt and exhausts host memory/swap. On constrained hosts this can drive load into the dozens and require manual power-cycle recovery.

Root cause

_try_activate_fallback() at ~line 8620 had no throttle between activations and no cap on how many times it could fire in a time window.

Fix

Added a per-instance circuit-breaker at the top of _try_activate_fallback():

• Throttle: ≥2s between consecutive activations (sleeps if called sooner)
• Breaker: ≥5 activations in 60s → returns False (chain-exhausted signal existing code already handles cleanly)
• Status messages emitted for both throttle sleeps and breaker trips

Tests

• 8 new unit tests in tests/run_agent/test_fallback_circuit_breaker.py
• All 30 fallback + circuit-breaker tests pass
• Pre-existing failure in test_async_httpx_del_neuter.py confirmed as pre-existing (exists on main, unrelated to this change)

Files changed

• run_agent.py — circuit-breaker added to _try_activate_fallback()
• tests/run_agent/test_fallback_circuit_breaker.py — 8 new tests

Changed files

• run_agent.py (modified, +29/-0)
• tests/run_agent/test_fallback_circuit_breaker.py (added, +247/-0)