PR #12585: fix(models): accept Gemini + Anthropic in gateway /model picker (#12532)

• Repository: NousResearch/hermes-agent
• Author: briandevans
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/12585

Description (problem / solution / changelog)

Fixes #12532.

TL;DR

Gateway /model picker rejects both Gemini and Anthropic models while CLI hermes model works with the same credentials. Two distinct bugs in validate_requested_model:

• Gemini: /models returns IDs like models/gemini-2.5-flash — curated list and user input use bare gemini-2.5-flash → set membership drops every model.
• Anthropic: generic probe sends Authorization: Bearer without anthropic-version → Anthropic's native /v1/models returns 4xx → fetch_api_models yields None → lands in the generic "couldn't reach API" hard-reject.

Both fixes mirror existing patterns already in this function (Bedrock / Alibaba catalog fall-through for Anthropic; symmetric normalization for the Gemini prefix drift).

Root cause details

Gemini

Gemini's OpenAI-compat endpoint at generativelanguage.googleapis.com/v1beta/openai/models returns IDs in the native Gemini format (models/gemini-2.5-flash). The curated _PROVIDER_MODELS["gemini"] list and the picker UI both use the bare ID. requested_for_lookup in set(api_models) therefore fails for every known Gemini model.

Anthropic

probe_api_models constructs headers at hermes_cli/models.py:1776-1780:

headers: dict[str, str] = {}
if api_key:
    headers["Authorization"] = f"Bearer {api_key}"
if normalized.startswith(COPILOT_BASE_URL):
    headers.update(copilot_default_headers())

Anthropic's API requires x-api-key (for API keys) or Authorization: Bearer with anthropic-version header — never Bearer alone without a version. So the probe silently 4xxs, returning None, and the request lands in the "couldn't reach API" hard-reject branch.

_fetch_anthropic_models (line 1338) already does the right thing — it's called during provider_model_ids("anthropic") for picker population, but never for per-request validation.

Fix

Two surgical additions to validate_requested_model:

1. Gemini prefix strip — before the set-membership check:

if api_models is not None:
    if normalized == "gemini":
        api_models = [
            m[len("models/"):] if isinstance(m, str) and m.startswith("models/") else m
            for m in api_models
        ]
    if requested_for_lookup in set(api_models):
        ...

The normalized list also drives auto-correct + suggestions, so "Similar models: gemini-2.5-flash" surfaces bare IDs the user can actually type — not the useless models/... literals.

2. Anthropic catalog fall-through — after the existing Bedrock block, same pattern as the #12287 Alibaba fix:

if normalized == "anthropic":
    try:
        catalog = provider_model_ids("anthropic")
    except Exception:
        catalog = []
    if catalog:
        if requested in set(catalog) or requested_for_lookup in set(catalog):
            return {"accepted": True, "persist": True, "recognized": True, "message": None}
        suggestions = get_close_matches(requested, catalog, n=3, cutoff=0.4)
        return {
            "accepted": True,
            "persist": True,
            "recognized": False,
            "message": f"Note: `{requested}` was not found in the Anthropic catalog. "
                       f"It may still work if it's a newer model the catalog doesn't list yet. ...",
        }
    # empty / exception → fall through to the generic reject

provider_model_ids("anthropic") already handles live-fetch (via _fetch_anthropic_models with the correct x-api-key + anthropic-version headers) and falls back to the curated static list on failure.

Behaviour matrix

Narrow scope — explicitly not changed

• probe_api_models auth headers. Still Bearer-only. Teaching the generic probe about Anthropic's header requirements would entangle provider-specific details in the transport layer; the catalog fall-through is less invasive.
• Other providers that might return models/-prefixed IDs. The strip is gated on normalized == "gemini". Pinned by a custom canary.
• Reporter's Option A (skip validation entirely when the model was chosen from a curated picker). That's a gateway-side refactor of _on_model_selected/switch_model; this PR keeps the fix at the validator layer so it benefits CLI, gateway, and any future caller uniformly.

Regression coverage

Two new test classes, 11 cases:

TestValidateGeminiModelsPrefix (4 cases):

• test_bare_id_accepted_despite_models_prefix_from_api — reporter's repro
• test_all_curated_gemini_ids_resolve — 4 IDs across pro/flash/flash-lite/preview
• test_unknown_gemini_id_surfaces_suggestions_not_generic_reject — pins that suggestions don't leak the models/ prefix
• test_prefix_strip_limited_to_gemini_provider — custom provider canary

TestValidateAnthropicNoModelsEndpoint (7 cases):

• test_curated_claude_model_accepted — reporter's repro
• test_all_tiers_resolve — opus/sonnet/haiku
• test_unknown_model_accepted_with_warning
• test_empty_catalog_falls_through_to_generic_reject
• test_catalog_lookup_exception_falls_through
• test_unknown_model_includes_close_match_suggestion
• test_other_providers_still_hard_reject_when_api_unreachable — zai canary

6 of 11 fail on clean origin/main (6fb69229) with assert False is True on result["accepted"]. The 5 passing tests pin preserved behaviour (canaries + defensive fall-throughs).

Validation

source venv/bin/activate
python -m pytest \
  tests/hermes_cli/test_model_validation.py::TestValidateGeminiModelsPrefix \
  tests/hermes_cli/test_model_validation.py::TestValidateAnthropicNoModelsEndpoint -q
# 11 passed

Broader model-switch / normalize suites (6 files) → 159 passed, 0 failures.

Relation to prior PRs

This PR uses the exact same fall-through pattern as #12287 (Alibaba DashScope coding endpoint). That pattern in turn mirrors the pre-existing Bedrock branch. Each provider whose /models endpoint is structurally inaccessible gets its own small catalog fall-through — no broad widening of the validator's trust surface.

_{Co-authored via LLM assistance; I've reviewed every line and am responsible for correctness.}

Changed files

• hermes_cli/models.py (modified, +47/-0)
• tests/hermes_cli/test_model_validation.py (modified, +242/-0)