hermes - ✅(Solved) Fix Named custom providers under providers: drop inline api_key during runtime resolution [1 pull requests, 2 comments, 2 participants]

hermes2026-04-22 17:02:28

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

NousResearch/hermes-agent#14065•Fetched 2026-04-23 07:46:59

View on GitHub

Comments

Participants

Timeline

Reactions

Author

NewTurn2017

Participants

alt-glitch

NewTurn2017

Timeline (top)

labeled ×4commented ×2

Named custom providers stored under the v12+ providers: dict lose their inline api_key during runtime resolution.

Root Cause

Additional evidence

The existing unit test already fails on main:

source venv/bin/activate
pytest -q tests/hermes_cli/test_runtime_provider_resolution.py::test_named_custom_provider_uses_providers_dict_when_list_missing

It fails at tests/hermes_cli/test_runtime_provider_resolution.py:610 because runtime resolution returns no-key-required instead of the configured key.

Fix Action

Fix / Workaround

Minimal reproduction

source venv/bin/activate
python - <<'PY'
import hermes_cli.runtime_provider as rp
from unittest.mock import patch

cfg = {
    'providers': {
        'openai-direct-primary': {
            'api': 'https://api.openai.com/v1',
            'api_key': 'dir-key',
            'default_model': 'gpt-5-mini',
            'name': 'OpenAI Direct (Primary)',
            'transport': 'codex_responses',
        }
    }
}
with patch.object(rp, 'load_config', lambda: cfg), \
     patch.object(rp, 'resolve_provider', lambda *a, **k: (_ for _ in ()).throw(AssertionError('should not resolve built-in'))):
    print(rp.resolve_runtime_provider(requested='openai-direct-primary'))
PY

PR fix notes

PR #14960: fix(cli): preserve api_mode/key_env/model for providers-dict named resolution

Repository: NousResearch/hermes-agent
Author: sliverp
State: open | merged: False
Link: https://github.com/NousResearch/hermes-agent/pull/14960

Description (problem / solution / changelog)

Summary

Fixes #14065. _get_named_custom_provider() had two resolution branches — an inline providers: dict handler and a legacy custom_providers: list handler — and they disagreed on which fields survive the trip to _resolve_named_custom_runtime().

The inline providers dict branch returned only name / base_url / api_key / model, silently dropping api_mode, transport, key_env and provider_key. When a user points at a non-OpenAI base URL (e.g. a LiteLLM proxy fronting GPT-5) with an explicit transport: codex_responses, _detect_api_mode_for_url() cannot rescue the missing field and the resolver falls back to chat_completions, pointing the runtime at the wrong API surface.

The fix

get_compatible_custom_providers(config) already merges the legacy list and the v12+ dict into a single normalized view via providers_dict_to_custom_providers() + _normalize_custom_provider_entry(), which preserves every field (api_mode, transport, key_env, provider_key, model, models, context_length, rate_limit_delay). The legacy branch below already consumed exactly that shape.

So the minimal, forward-compatible fix is to delete the duplicate inline parser and let the legacy branch handle both config shapes through the compatibility view. This is the approach NewTurn2017 suggested in the issue body.

Diff shape

hermes_cli/runtime_provider.py: −44 / +10. The inline providers dict loop is gone; the dict-vs-list misconfiguration warning is preserved.
tests/hermes_cli/test_runtime_provider_resolution.py: +132. Two regression tests.

Why the existing test kept passing on `main`

test_named_custom_provider_uses_providers_dict_when_list_missing uses api: "https://api.openai.com/v1" with transport: "codex_responses". That URL matches _detect_api_mode_for_url()'s OpenAI heuristic, which coincidentally returns codex_responses. The assertion passed because of the URL heuristic, not because transport was actually propagated — exactly the silent drop this PR fixes.

New tests

test_named_custom_provider_preserves_transport_for_non_openai_url — the core regression test. Uses a non-matching base URL (https://llm.internal.corp/v1) with transport: codex_responses. On main this asserts 'chat_completions' == 'codex_responses' → fails; with the fix applied → passes. (Verified locally by stashing the fix and running the test against main.)
test_named_custom_provider_uses_key_env_when_inline_api_key_absent — proves key_env now survives the trip into _resolve_named_custom_runtime() for providers-dict entries; previously the normalizer's key_env never reached the resolver because the inline handler returned api_key="" directly.

Also updated the docstring on test_named_custom_provider_key_env_overrides_inline_api_key to document the intended candidate order (inline api_key first, then os.getenv(key_env)) — that was not a new behavioral choice introduced here, but it deserved to be pinned down while we were in the area.

Risk / scope

The warning about custom_providers: <dict> misconfiguration is preserved (it now gates on custom_providers_raw before we reach get_compatible_custom_providers).
Dedup logic (seen_provider_keys / seen_name_url_pairs) in get_compatible_custom_providers is unchanged.
Match semantics for requested names are unchanged: the legacy branch matches name_norm, menu_key, provider_key_norm, provider_menu_key, all of which are populated by _normalize_custom_provider_entry when the entry originates from a providers dict.
Related sibling fix c449cd1a ("populate api_mode from provider_info in model switch") tackled the model-switch path; this PR completes the picture on the resolution path.

Test results

tests/hermes_cli/test_runtime_provider_resolution.py ........................ 74 passed
tests/hermes_cli/test_config.py                                               all pre-existing pass

Pre-existing failures in test_api_key_providers.py and test_custom_provider_model_switch.py are identical on main and on this branch (environment-dependent integration tests).

Credit

Thanks to @NewTurn2017 for the precise root-cause analysis with file paths and line numbers — the fix strategy in this PR is directly the one proposed in the issue body.

Changed files

hermes_cli/runtime_provider.py (modified, +10/-44)
tests/hermes_cli/test_runtime_provider_resolution.py (modified, +132/-0)

Code Example

source venv/bin/activate
python - <<'PY'
import hermes_cli.runtime_provider as rp
from unittest.mock import patch

cfg = {
    'providers': {
        'openai-direct-primary': {
            'api': 'https://api.openai.com/v1',
            'api_key': 'dir-key',
            'default_model': 'gpt-5-mini',
            'name': 'OpenAI Direct (Primary)',
            'transport': 'codex_responses',
        }
    }
}
with patch.object(rp, 'load_config', lambda: cfg), \
     patch.object(rp, 'resolve_provider', lambda *a, **k: (_ for _ in ()).throw(AssertionError('should not resolve built-in'))):
    print(rp.resolve_runtime_provider(requested='openai-direct-primary'))
PY

---

{'api_key': 'no-key-required', 'api_mode': 'codex_responses', ...}

---

source venv/bin/activate
pytest -q tests/hermes_cli/test_runtime_provider_resolution.py::test_named_custom_provider_uses_providers_dict_when_list_missing

RAW_BUFFERClick to expand / collapse

Summary

Named custom providers stored under the v12+ providers: dict lose their inline api_key during runtime resolution.

Affected files

hermes_cli/runtime_provider.py:279-315
hermes_cli/runtime_provider.py:392-408
hermes_cli/config.py:1571-1649
tests/hermes_cli/test_runtime_provider_resolution.py:575-613

Why this is a bug

providers_dict_to_custom_providers() and _normalize_custom_provider_entry() preserve api_key, api_mode, and model for providers: entries, but _get_named_custom_provider() re-parses config["providers"] manually and only resolves key_env. When key_env is absent, the inline api_key is dropped and _resolve_named_custom_runtime() falls back to "no-key-required".

That breaks direct custom endpoints defined in the new keyed schema even though the compatibility layer already knows how to normalize them.

Minimal reproduction

source venv/bin/activate
python - <<'PY'
import hermes_cli.runtime_provider as rp
from unittest.mock import patch

cfg = {
    'providers': {
        'openai-direct-primary': {
            'api': 'https://api.openai.com/v1',
            'api_key': 'dir-key',
            'default_model': 'gpt-5-mini',
            'name': 'OpenAI Direct (Primary)',
            'transport': 'codex_responses',
        }
    }
}
with patch.object(rp, 'load_config', lambda: cfg), \
     patch.object(rp, 'resolve_provider', lambda *a, **k: (_ for _ in ()).throw(AssertionError('should not resolve built-in'))):
    print(rp.resolve_runtime_provider(requested='openai-direct-primary'))
PY

Actual output includes:

{'api_key': 'no-key-required', 'api_mode': 'codex_responses', ...}

Expected: api_key should be dir-key (or the normalized providers-dict value), matching the compatibility normalization path.

Additional evidence

The existing unit test already fails on main:

source venv/bin/activate
pytest -q tests/hermes_cli/test_runtime_provider_resolution.py::test_named_custom_provider_uses_providers_dict_when_list_missing

It fails at tests/hermes_cli/test_runtime_provider_resolution.py:610 because runtime resolution returns no-key-required instead of the configured key.

Suggested investigation

Make _get_named_custom_provider() reuse providers_dict_to_custom_providers() / _normalize_custom_provider_entry() instead of maintaining a second partial parser for providers: entries. That should keep api_key, api_mode, model, and provider_key behavior consistent across legacy and v12+ config paths.

extent analysis

TL;DR

The most likely fix is to modify _get_named_custom_provider() to reuse providers_dict_to_custom_providers() and _normalize_custom_provider_entry() for consistent parsing of providers: entries.

Guidance

Investigate the _get_named_custom_provider() function to understand why it manually re-parses config["providers"] and drops the inline api_key when key_env is absent.
Consider reusing providers_dict_to_custom_providers() and _normalize_custom_provider_entry() in _get_named_custom_provider() to maintain consistency in parsing providers: entries.
Verify the fix by running the provided minimal reproduction script and checking if the api_key is correctly preserved.
Review the existing unit test test_named_custom_provider_uses_providers_dict_when_list_missing to ensure it passes after applying the fix.

Example

# Example of how _get_named_custom_provider() could be modified
def _get_named_custom_provider(config, name):
    # Reuse providers_dict_to_custom_providers() and _normalize_custom_provider_entry()
    custom_providers = providers_dict_to_custom_providers(config["providers"])
    return _normalize_custom_provider_entry(custom_providers.get(name))

Notes

The suggested investigation points towards maintaining consistency in parsing providers: entries, but the actual implementation may vary depending on the specific requirements and constraints of the project.

Recommendation

Apply workaround: Modify _get_named_custom_provider() to reuse providers_dict_to_custom_providers() and _normalize_custom_provider_entry() to ensure consistent parsing of providers: entries. This should fix the issue with named custom providers losing their inline api_key during runtime resolution.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #memory optimization #batch processing #GPU compatibility #latency issue

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

hermes - ✅(Solved) Fix Named custom providers under providers: drop inline api_key during runtime resolution [1 pull requests, 2 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Root Cause

Additional evidence

Fix Action

Fix / Workaround

Minimal reproduction

PR fix notes

PR #14960: fix(cli): preserve api_mode/key_env/model for providers-dict named resolution

Description (problem / solution / changelog)

Summary

The fix

Diff shape

Why the existing test kept passing on main

New tests

Risk / scope

Test results

Credit

Changed files

Code Example

Summary

Affected files

Why this is a bug

Minimal reproduction

Additional evidence

Suggested investigation

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

RELATED_DISCOVERY

TRENDING

Why the existing test kept passing on `main`