Hermes Agent version: v0.14.0 (release tag 2026.5.16, installed via npm install -g openclaw-equivalent path on Linux x86_64) Python: 3.11.15 (uv-managed venv) Provider: gemini (Google AI Studio direct, free tier)

Symptom

When the primary Gemini provider returns HTTP 429 RESOURCE_EXHAUSTED (free-tier quota exhausted, ~250 requests/day on gemini-3-flash), the fallback provider chain is never activated. Every user message subsequently fails with Agent error in session … despite having fallback_providers: configured. Other failure modes (e.g. HTTP 400) DO activate the fallback correctly — only the rate-limit branch is broken.

Root cause

The function _pool_may_recover_from_rate_limit(credential_pool, provider, base_url) is defined in run_agent.py:239 but never imported in agent/conversation_loop.py, where it is called at line 2335:

# agent/conversation_loop.py
if is_rate_limited and agent._fallback_index < len(agent._fallback_chain):
    # Don't eagerly fallback if credential pool rotation may
    # still recover.  See _pool_may_recover_from_rate_limit
    # for the single-credential-pool and CloudCode-quota
    # exceptions.  Fixes #11314 and #13636.
    pool_may_recover = _pool_may_recover_from_rate_limit(   # ← NameError here
        agent._credential_pool,
        provider=agent.provider,
        base_url=getattr(agent, "base_url", None),
    )
    if not pool_may_recover:
        agent._emit_status("⚠️ Rate limited — switching to fallback provider...")
        if agent._try_activate_fallback(reason=classified.reason):
            ...

Inspecting the module-level imports of agent/conversation_loop.py (lines 17–65), there is no from run_agent import _pool_may_recover_from_rate_limit anywhere. The function reference resolves to the global namespace and raises NameError.

Stack trace (from a real production session)

File "/.../hermes-betty/agent/conversation_loop.py", line 2335, in run_conversation
    pool_may_recover = _pool_may_recover_from_rate_limit(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name '_pool_may_recover_from_rate_limit' is not defined

The error is caught by the outer _handle_message_with_agent wrapper in gateway/run.py:7930, which logs it as Agent error in session … — masking the actual NameError unless one inspects the full traceback in journalctl.

Why a module-level import is risky (circular dep)

A naive from run_agent import _pool_may_recover_from_rate_limit at the top of conversation_loop.py will deadlock module loading because run_agent.py imports agent.conversation_loop (line 47 in v0.14.0). Both modules would block waiting for the other to finish initializing.

Suggested fix (lazy import, minimum surface area)

Move the import inside the conditional, right before the call:

if is_rate_limited and agent._fallback_index < len(agent._fallback_chain):
    # Lazy import — defined in run_agent.py:239 but never imported at module
    # load in conversation_loop.py. Lazy avoids the circular dep that a
    # top-of-file import would trigger (run_agent.py:47 imports this module).
    from run_agent import _pool_may_recover_from_rate_limit
    pool_may_recover = _pool_may_recover_from_rate_limit(
        agent._credential_pool,
        provider=agent.provider,
        base_url=getattr(agent, "base_url", None),
    )
    if not pool_may_recover:
        ...

I've applied this patch locally and verified that import agent.conversation_loop no longer raises at startup, AST parses cleanly, and HTTP 429 paths now reach agent._try_activate_fallback() instead of crashing.

Alternative fix (move helper into a shared module)

If you prefer a longer-term cleanup: extract _pool_may_recover_from_rate_limit (and the related credential-pool helpers in run_agent.py:200-260) into a new module like agent/credential_pool_recovery.py. Both run_agent and conversation_loop could then import it without any circular risk.

Affected referenced bugs

The comment block in conversation_loop.py references #11314 and #13636 as the original motivation for this rate-limit guard. The guard logic itself is correct; the import is the only thing missing. Both of those original fixes are effectively regressed in v0.14.0 until this import lands.

Repro

1. Set model.provider: gemini with a Google AI Studio free-tier key in config.yaml.
2. Add at least one fallback_providers: entry (e.g. nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free via OpenRouter).
3. Drive enough turns to exhaust the daily 250-request quota (Hermes makes 3–10 calls/turn, so ~30 turns suffices).
4. Send one more message. Expected: fallback activates. Actual: NameError in the gateway logs, agent_error in the session, no message delivered.