Feature Request: Fallback Models for Auxiliary Tasks

Repo: NousResearch/hermes-agent

Title

feat: fallback models for auxiliary/web_extract

Body

Problem

The auxiliary config section supports only a single model per task (vision, web_extract, compression, etc.). If that model fails (rate limit, timeout, API error), there's no automatic fallback — the task returns an error or falls back to raw content (first 5K chars) without summarization.

The main conversation loop already has fallback_providers: [] for model-level fallback, but auxiliary tasks lack this entirely.

Proposed Solution

Add an optional fallback or backup field to each auxiliary task config, with the same structure as the primary config:

auxiliary:
  web_extract:
    provider: custom
    model: gemini-3.1-flash-lite-preview
    base_url: ...
    api_key: ...
    timeout: 120
    fallback:
      provider: custom
      model: gemma-4-26b-a4b-it
      base_url: ...
      api_key: ...
      timeout: 120

On failure (non-2xx, timeout, empty response), the system would automatically retry summarization with the fallback model before giving up.

Use Case

Users who pair a fast model (Gemini Flash Lite — ~19s/500 RPD) with a more resilient backup (Gemma 4 — ~33s/1.5K RPD) can get both speed and reliability without manual intervention.

Trade-offs Considered

• Adding complexity to the auxiliary dispatch path
• Must define what counts as a 'failure' worth falling back on (timeout, rate limit, any non-2xx)
• Fallback should not retry the fetch layer (Firecrawl/etc.) — only the LLM summarization phase
• Could chain multiple fallbacks (primary → secondary → tertiary) if needed