hermes - 💡(How to fix) Fix Support ordered failover chains for web search/extract backends

Feature description

Add first-class support for ordered web backend failover chains for web_search, web_extract, and optionally web_crawl.

Today Hermes supports a single active backend per capability via:

web:
  backend: firecrawl
  search_backend: brave-free
  extract_backend: tavily

This works, but it does not express a common real-world setup where a user has multiple configured providers and wants one provider to be primary while others are fallbacks.

Motivation

Some web providers can be temporarily unusable even though their API key is present and the provider passes the current availability check. For example:

• Firecrawl API key exists, so Hermes considers Firecrawl available.
• Firecrawl cloud quota/credits are exhausted and the actual call returns Payment Required / Insufficient credits.
• Brave Search and Tavily keys are also configured and working.
• The user wants Hermes to try the primary provider first, then fall back for this one call if the primary returns a quota/rate-limit/payment/transient error.
• The user does not want the primary provider to be permanently disabled; once quota is restored, it should naturally be tried again on future calls.

The current choices are either:

1. Explicitly set web.search_backend = brave-free to avoid Firecrawl, which prevents Firecrawl from being used again automatically after quota restoration; or
2. Leave backend selection automatic, where Firecrawl may keep being selected because credentials exist, even if current quota is exhausted.

A supported failover chain would make this behavior explicit and predictable without local source modifications.

Proposed solution

Support ordered backend chains in config, while preserving existing singular fields for backwards compatibility:

web:
  search_backends:
    - firecrawl
    - brave-free
    - tavily
    - exa
    - ddgs

  extract_backends:
    - firecrawl
    - tavily
    - exa
    - parallel

  crawl_backends:
    - firecrawl
    - tavily

Suggested behavior:

1. For each tool call, start from the first configured backend in the chain.
2. If the provider is unavailable due to missing credentials/package, skip to the next configured backend.
3. If the provider call fails with a retryable/failover-worthy error, try the next backend for that same call.
4. Do not permanently disable or demote the primary provider unless a separate explicit circuit-breaker feature is added later.
5. On the next independent tool call, start from the primary provider again.

Candidate failover-worthy errors:

• quota exhausted / insufficient credits
• Payment Required
• rate limit / HTTP 429
• transient 5xx responses
• provider timeout / temporary unavailable
• optionally authentication errors if another configured backend is available

Non-failover errors should still be surfaced directly, for example invalid input, unsafe/private URL blocking, or a provider that does not support the requested capability.

Backwards compatibility

Existing config should continue to work:

web:
  search_backend: brave-free
  extract_backend: tavily
  backend: firecrawl

Suggested precedence:

1. New plural capability-specific chain, e.g. web.search_backends
2. Existing singular capability-specific backend, e.g. web.search_backend
3. New shared plural chain, e.g. web.backends
4. Existing shared singular backend, e.g. web.backend
5. Existing legacy auto-detection order

This keeps current installs stable while enabling users with multiple providers to opt into failover explicitly.

Example user scenario

A user has these keys configured:

• FIRECRAWL_API_KEY
• BRAVE_SEARCH_API_KEY
• TAVILY_API_KEY

They want:

web:
  search_backends: [firecrawl, brave-free, tavily]
  extract_backends: [firecrawl, tavily]

If Firecrawl returns Payment Required: Insufficient credits, web_search should try Brave Search for that call. When Firecrawl quota is later restored, the next web_search call should try Firecrawl first again.

Alternatives considered

Manually switching web.search_backend

This is supported today but does not solve automatic recovery after the primary provider quota is restored.

Removing the exhausted provider API key

This is inconvenient and mixes credential management with runtime routing policy.

Local source patching

This is not desirable. Users should not need to modify Hermes source code to get predictable provider failover behavior.

Additional notes

This could also improve the user experience of hermes tools by allowing users to configure a provider preference order instead of a single provider.