hermes - 💡(How to fix) Fix [Feature]: Global toggle to disable all provider fallbacks, prevent silent data leakage to third parties [1 participants]

Problem or Use Case

I like to have control over my data. Under no circumstances should my prompts, attached images, screenshots of browsers, or session data reach third-party APIs. Currently, ensuring this requires manually overriding every individual auxiliary task and hoping no code path misses a fallback edge case.

Hermes has multiple independent fallback mechanisms that activate when a provider fails or is unavailable. There is no single, auditable way to disable all of them. For users running Hermes under a strict privacy model (e.g., only local inference or a single trusted endpoint), this creates a risk of silent data leaks — prompts, images, and session context can be sent to third-party providers without explicit user consent.

Current behavior

Fallbacks are scattered across the codebase and controlled independently. In many cases, the correct method to disable a given fallback is either undocumented or requires reading source code to discover.

Note: The table below lists where fallbacks occur. The steps to disable them are not consistently documented in the official docs or CLI help, and some require inspecting agent/auxiliary_client.py to understand.

Note: Even where disable steps can be inferred (e.g., omitting fallback_model or setting explicit auxiliary.*.provider), there is no guarantee they are complete or forward-compatible. New auxiliary tasks may introduce new fallback paths that are not covered by existing documentation.

Why this is insufficient:

• Setting explicit auxiliary.vision.provider: custom blocks most fallback paths, but async_call_llm still contains a _try_payment_fallback() branch that fires on connection/payment errors when the resolved provider is "auto".
• A misconfigured env var or transient DNS failure could cause the resolved provider to flip to "auto", triggering fallback to an external aggregator without warning.
• There is no way to audit or prove that fallbacks are globally disabled — a single missed override in a new auxiliary task type defeats the purpose.

Environment

• Hermes version: v0.12.0
• Provider: Custom endpoint via model.provider: custom
• Use case: Privacy-sensitive, zero-third-party-exfiltration requirement

Proposed Solution

Add a single, top-level configuration switch that is respected by all fallback paths:

# ~/.hermes/config.yaml
fallback:
  enabled: false   # default: true

When fallback.enabled: false:

1. Primary model fallback — AIAgent never activates fallback_providers; hard-error on primary failure.
2. Auxiliary task fallback — resolve_vision_provider_client(), _resolve_auto(), _try_payment_fallback(), and all related auto-detection chains return None immediately instead of chaining.
3. Credential-pool rotation — still allowed (same provider, different keys), since this does not leak data to a different party.
4. Image size retry — unaffected (retries same endpoint).

This should be a single source of truth checked at the top of every fallback decision point, rather than the current per-task override whack-a-mole.

Alternatives Considered

Per-task explicit overrides (auxiliary.vision.provider: custom, etc.) — this is what I currently use, but:

• It is error-prone (easy to miss a task)
• It is not forward-compatible (new auxiliary tasks may add new fallback paths)
• It does not protect against the "auto" edge case in async_call_llm

Feature Type

Configuration option

Scope

Small (single file, < 50 lines)

Contribution

• I'd like to implement this myself and submit a PR

Debug Report (optional)