PR #23369: feat: hermes models / hermes providers — non-interactive inventory CLI (#23359)

• Repository: NousResearch/hermes-agent
• Author: kshitijk4poor
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/23369

Description (problem / solution / changelog)

What this PR does

Adds three scriptable, non-interactive CLI surfaces for inspecting Hermes' provider/model inventory:

hermes models list   [--provider NAME] [--json] [--all] [--no-live] [--offline]
hermes models status [--provider NAME] [--json] [--probe]            [--offline]
hermes providers list                  [--json] [--all]              [--offline]

Closes #23359 (jointly with #3503 once that PR migrates to consume the shared formatter).

The implementation is a thin façade over existing primitives — no new HTTP code, no parallel registry, no refactor of list_authenticated_providers. Default mode delegates to the same function the picker / dashboard / TUI already use; --offline synthesises from in-repo static data so cron and CI callers don't block on the network.

Why

Hermes can list providers/models four ways today (interactive picker via cli.py, TUI via tui_gateway/server.py:5158 model.options JSON-RPC, dashboard via hermes_cli/web_server.py:983 /api/model/options, internal switcher via hermes_cli/main.py:2250) — but none of them are scriptable. Every entry point requires a TTY or a running gateway. There is no way for a cron job, CI script, external dashboard, or Hermes-based agent introspecting its own environment to ask "what providers exist on this host, which are configured, what models can I switch to" without parsing free-form text or maintaining a reverse-engineered mental model of hermes_cli/auth.py:PROVIDER_REGISTRY.

Five open PRs have each reinvented this traversal for one narrow purpose with no shared function: #3503 (/models slash), #21695 (plugin provider runtime resolver), #19526 (model.picker_mode), #17994 (/free-models), #21785 (hermes doctor skip-by-endpoint). Each adds ~50-150 LOC of inventory walking that this PR consolidates.

Full motivation, pain points, and ecosystem evidence in the tracking issue #23359.

What this PR is NOT doing

• Not refactoring list_authenticated_providers — only delegating to it. No behaviour change for picker / dashboard / TUI.
• Not changing the existing hermes model interactive picker.
• Not adding write-side symmetry for hermes model → providers.<slug>.models — that's #16622 (separate ~50 LOC follow-up; schema fixtures already exist in tests/hermes_cli/test_model_catalog.py).
• Not implementing model.allowlist / model.picker_mode — that's #19526.
• Not adding model-discovery for built-in providers — that's #13055 / #21695.
• Not migrating dashboard or TUI to consume the new module — they continue to call list_authenticated_providers directly. Migration is a follow-up; the new module exposes the same primitives.

Modes / network behaviour

--offline bypasses list_authenticated_providers entirely and synthesises inventory from CANONICAL_PROVIDERS (universe of ~35) + _PROVIDER_MODELS (curated) + OPENROUTER_MODELS (local constant) + profile.fallback_models. Auth state is derived from environment variables alone — profile.env_vars ∪ PROVIDER_REGISTRY[slug].api_key_env_vars ∪ HERMES_OVERLAYS[slug].extra_env_vars. No consultation of auth.json, credential pool, Claude Code creds, or Hermes OAuth files.

JSON contract

{
  "schema_version": 1,
  "current": {"provider": "openrouter", "model": "anthropic/claude-sonnet-4.6"},
  "providers": [
    {
      "slug": "anthropic",
      "name": "Anthropic",
      "is_current": false,
      "is_user_defined": false,
      "auth_state": "configured",
      "api_mode": "anthropic_messages",
      "auth_type": "api_key",
      "base_url": "https://api.anthropic.com",
      "env_vars": ["ANTHROPIC_API_KEY", "ANTHROPIC_TOKEN", "CLAUDE_CODE_OAUTH_TOKEN"],
      "env_vars_present": [true, false, false],
      "models": ["claude-opus-4-7", "..."],
      "total_models": 8,
      "source": "canonical",
      "model_source": "curated"
    }
  ]
}

env_vars lists variable names (including HERMES_OVERLAYS bridges — e.g. OpenRouter accepts OPENAI_API_KEY). env_vars_present is the corresponding bool array. Values are never included in the payload — explicit invariant; canary-tested.

Substrate gotchas the implementation respects

These are documented in references/provider-inventory-surface.md (hermes-agent-dev skill) and were each verified against main at d62808c37 before implementation:

1. Four registries don't fully overlap. list_providers() (33), PROVIDER_REGISTRY (33), CANONICAL_PROVIDERS (35), _PROVIDER_MODELS (32) each have different exclusions. Inventory iterates CANONICAL_PROVIDERS (the universe). Naive iteration over list_providers() would silently drop lmstudio and tencent-tokenhub.
2. profile.fetch_models() returns models the runtime rejects. OpenRouter raw = 367, picker-filtered = 34. The plugin's own docstring says filtering happens in hermes_cli/models.py. This module never calls profile.fetch_models() directly — always through provider_model_ids(). Canary test enforces.
3. Custom providers split across two config shapes. Legacy custom_providers: [...] list vs v12+ keyed providers: {<key>: {...}} dict. Module reads via get_compatible_custom_providers(config) (the merged compatibility view), not raw cfg.get("custom_providers").
4. OpenRouter offline gotcha. _PROVIDER_MODELS["openrouter"] is None. Falling back to profile.fallback_models returns 5 entries; falling back to model_ids() does HTTP. Module uses the local OPENROUTER_MODELS constant directly (~31 entries, no HTTP). Canary test asserts equality against the constant length, not a magic number.
5. OAuth-external probe-skip predicate. Most oauth_external providers (gemini-cli, qwen-oauth, minimax-oauth) have a base_url but no usable /models endpoint — provider_model_ids falls back to curated lists, masking probe failure as "ok". Skip predicate uses an explicit allowlist frozenset({"openai-codex"}), applied before provider_model_ids is invoked.
6. CLI exit-code propagation. hermes_cli/main.py:11688 does args.func(args) and discards the return value. Handlers use parser.error() / required=True / SystemExit(2) instead of return 2.
7. Anthropic credential discovery has 5+ paths. A clean-env E2E asserting "anthropic is unconfigured" can fail on dev machines that have Claude Code installed (reads ~/.claude/credentials.json). E2E uses arcee (single env var, no extras, no auth_store seeding) as the canonical clean-baseline target.

Tests

The Concept B canary monkeypatches every per-provider live-fetch helper (fetch_anthropic_models, fetch_openrouter_models, fetch_ai_gateway_models, fetch_ollama_cloud_models, fetch_lmstudio_models, _fetch_github_models, fetch_api_models, fetch_nous_models, fetch_models_dev, model_ids, get_curated_nous_model_ids, bedrock_model_ids_or_none, list_authenticated_providers) to fail-on-call, then asserts none fire when --offline is passed. This is the load-bearing assertion — if a future change leaks any of these into the static path, the canary fires.

$ pytest tests/hermes_cli/test_inventory.py tests/hermes_cli/test_inventory_cli.py tests/hermes_cli/test_startup_plugin_gating.py -q
83 passed in 5.70s

Full tests/hermes_cli/ (4169 tests, ~5 min): all pre-existing failures verified to fail identically on origin/main (gateway systemd / WSL / kanban / plugins / web-server auth / update hangup — none related to this PR).

Lint diff

ruff — Total: 0 on HEAD, 0 on base (➖ 0)  🆕 New issues: none
ty   — 0 issues in hermes_cli/inventory.py

Files changed

hermes_cli/inventory.py                | 878 +++++++++++++++++++++++++++++++++ (new)
hermes_cli/main.py                     | 150 ++++++ (subparsers + 3 handlers + _BUILTIN_SUBCOMMANDS entry)
tests/hermes_cli/test_inventory.py     | 563 +++++++++++++++++++++ (new)
tests/hermes_cli/test_inventory_cli.py | 300 +++++++++++ (new)
website/docs/reference/cli-commands.md |  82 +++
5 files changed, 1973 insertions(+)

No setup.py files. No env-var-value logging (canary-tested). No changes to list_authenticated_providers, provider_model_ids, or any provider plugin. Single squashed commit.

How to verify

# Universe + offline correctness
hermes providers list --offline --all --json | jq '.providers | length'           # 35
hermes models list --provider=openrouter --offline --json | jq '.providers[0].total_models'   # ≈31

# OpenRouter regression check (the headline gotcha)
hermes models list --provider=openrouter --json | jq '.providers[0].total_models'  # ≤50, NOT 367

# Probe skip semantics
hermes models status --probe --provider=google-gemini-cli --json | jq '.providers[0].probe.status'  # "skipped"

# Mutex + exit codes
hermes models status --offline --probe; echo $?    # 2
hermes models list --provider=does-not-exist; echo $?   # 2
hermes models foo; echo $?    # 2

# Cron-safe inventory
hermes providers list --offline --all --json | jq '[.providers[] | select(.auth_state=="configured")] | length'

Out of scope (tracked separately)

• #16622 — write-side symmetry for hermes model writing to providers.<slug>.models (~50 LOC follow-up).
• Substrate audit — the empirical findings in #23359 (openrouter/custom excluded from PROVIDER_REGISTRY, profile.fetch_models() unfiltered for OpenRouter, provider_model_ids("nous") no-creds bypasses the remote catalog, list_authenticated_providers does unconditional ollama-cloud + lmstudio fetches) belong in a separate audit issue once this lands.
• #3503 — /models slash unification. After merge, comment on #3503 proposing migration to inventory.build_models_payload.
• TUI / dashboard migration — both currently inline their own canonical-merge logic. Module exposes everything needed for them to consume in a follow-up.

Refs

• Tracking issue: #23359
• Substrate: #20324 (merged 2026-05-05; this PR builds on top)
• Original refactor: #14424 (closed in favour of #20324)
• Issues this helps close (jointly with adjacent PRs): #3500, #13055
• Open PRs that should consume the new surface in follow-ups: #3503, #19526, #21695, #17994, #21785

Changed files

• hermes_cli/inventory.py (added, +917/-0)
• hermes_cli/main.py (modified, +152/-0)
• tests/hermes_cli/test_inventory.py (added, +600/-0)
• tests/hermes_cli/test_inventory_cli.py (added, +300/-0)
• website/docs/reference/cli-commands.md (modified, +82/-0)