hermes - 💡(How to fix) Fix Feature Request: Fallback chain should continue after first fallback fails with non-retryable error

Problem Statement

Currently, when the primary model fails and Hermes switches to the first fallback provider, if that fallback returns a non-retryable error (e.g., HTTP 403), the entire request is aborted instead of trying the next fallback in the chain.

Current Behavior

From the logs:

⚠️ Non-retryable error (HTTP 403) — trying fallback...
   🔌 Provider: custom  Model: gpt-5.5
   💡 Your API key was rejected by the provider.
❌ Non-retryable client error (HTTP 403). Aborting.

After this, the agent stops and no further fallback entries are attempted, even though valid providers (like deepseek-official and ds2api) remain unused.

Expected Behavior

When a fallback provider returns a non-retryable error, Hermes should log the failure and proceed to the next fallback provider in the list, only aborting if all configured fallbacks have been exhausted or if the primary model itself fails with a non-retryable error and there is no fallback.

The documentation at https://hermes-agent.nousresearch.com/docs/user-guide/features/fallback-providers states that fallback activates on auth failures (401/403) "immediately (no point retrying)". This makes sense for the primary model, but when applied to a fallback provider, it effectively prevents the chain from continuing, defeating the purpose of having multiple fallbacks.

Reproduction Steps

1. Configure multiple fallback providers in config.yaml:

   fallback_providers:
     - provider: huaibao
       model: gpt-5.5
     - provider: deepseek-official
       model: deepseek-v4-pro
     - provider: ds2api
       model: deepseek-v4-pro

2. Cause the primary model to fail (e.g., exceed quota).
3. Ensure the first fallback (huabiao/gpt-5.5) returns an HTTP 403 auth error.
4. Observe that Hermes aborts immediately, without trying the second or third fallback.

Proposed Solution

Modify the fallback logic to treat errors from fallback providers as retryable when there are remaining fallback entries. Specifically:

• When a fallback returns a non-retryable error AND there are more fallback providers in the list, log a warning and proceed to the next fallback.
• When the last fallback fails with a non-retryable error, or when there are no more fallbacks, abort as currently.

This would align the behavior with user expectations: "I have multiple backup models, keep trying until one works."

Environment

• Hermes Agent v0.13.0
• Config: multiple fallback providers via fallback_providers

Impact

Currently, a single unreliable API key in the fallback chain can block all other fallbacks, making the feature less robust than intended. This fix would improve resilience and user experience.