Update (2026-05-14) — original body claimed main also reproduces. That was wrong; I only diffed agent/auxiliary_client.py between v2026.5.7 and main and missed that run_agent.py had been patched separately. The real state, after re-checking, is:

• main (current HEAD 4fdfdf6) is already fixed by commit 6ddc48b0 ("fix(fallback): resolve api_key_env in fallback chain entries (carve-out of #22665)", 2026-05-09).
• But no released tag contains that commit yet — git tag --contains 6ddc48b0 returns only desktop-pr20059-installers. v2026.5.7 (the latest user-facing tag, released 2026-05-07) does not contain it.

So the issue is now scoped to "fix has merged on main but no release tag carries it." Suggested action below has been revised accordingly.

Summary

The documented api_key_env → key_env snake_case alias is recognized for entries under the top-level providers: section (via _normalize_custom_provider_entry in hermes_cli/config.py), but not for entries under fallback_providers: in the most recent published release tag v2026.5.7. As a result, when a fallback chain is activated (compression, title generation, etc.), entries written using api_key_env resolve with no API key and fall through to the "no-key-required" placeholder defined in agent/auxiliary_client.py, which is then sent as a Bearer token to the remote endpoint.

For remote providers that enforce auth (DeepSeek, Volcengine ARK, Moonshot, etc.) this produces a 401 with a confusingly masked key — e.g. DeepSeek returns Your api key: ****ired is invalid, where ired is simply the last 4 chars of the literal string no-key-required.

The failure is silent: no warning is emitted when the alias is dropped, and the upstream 401 message is the only signal — which is almost impossible to attribute without reading the source.

Versions

Reproduction (on v2026.5.7)

1. config.yaml:

   fallback_providers:
     - provider: custom
       model: kimi-k2.6
       base_url: https://ark.cn-beijing.volces.com/api/coding/v3
       api_key_env: ARK_API_KEY          # documented alias — silently dropped here

2. Have a session long enough to trigger compression (or otherwise force the fallback chain).
3. agent.log shows the chain activating the kimi fallback, the auxiliary client resolving via agent/auxiliary_client.py, and a 401 from https://ark.cn-beijing.volces.com/... with "API key format is incorrect". ARK_API_KEY is correctly set in .env.

Same reproduces for DeepSeek (api_key_env: DEEPSEEK_API_KEY) — the DeepSeek 401 echoes back the literal no-key-required masked as ****ired.

Expected

The api_key_env alias should resolve identically wherever it appears in config — under providers: and fallback_providers: both.

Actual (on v2026.5.7)

The alias is recognized in providers: but dropped in fallback_providers:, leading to silent auth failure downstream.

Root cause (code references, against v2026.5.7)

The alias normalization lives in hermes_cli/config.py:

# hermes_cli/config.py: _normalize_custom_provider_entry
"apiKeyEnv": "key_env",  # alias — OpenClaw-compatible + docs variant
...
if "api_key_env" in entry and "key_env" not in entry:
    entry["key_env"] = entry["api_key_env"]

But _normalize_custom_provider_entry is only invoked for:

• providers_dict_to_custom_providers(config.get("providers")) — providers: section
• _append_if_new(_normalize_custom_provider_entry(entry)) inside get_compatible_custom_providers — legacy custom_providers: list

The fallback_providers: consumer in run_agent.py (on v2026.5.7 only) reads the raw dict directly (≈ L7866):

fb_api_key_hint = (fb.get("api_key") or "").strip() or None
if not fb_api_key_hint:
    fb_key_env = (fb.get("key_env") or "").strip()    # api_key_env not checked
    if fb_key_env:
        fb_api_key_hint = os.getenv(fb_key_env, "").strip() or None

When fb_api_key_hint ends up None, the auxiliary client falls through to (agent/auxiliary_client.py ≈ L1519):

# Local servers (Ollama, llama.cpp, vLLM, LM Studio) don't require auth.
# Use a placeholder key — the OpenAI SDK requires a non-empty string but
# local servers ignore the Authorization header.
if not isinstance(custom_key, str) or not custom_key.strip():
    custom_key = "no-key-required"

That placeholder was added for local no-auth servers, but the same code path is reached for remote endpoints whose key resolution silently failed.

Status on main

Commit 6ddc48b0 on 2026-05-09 already applies the alias fix at both consumption sites in run_agent.py. The current main reads, with an explanatory comment:

# key_env and api_key_env are both documented aliases (see
# _normalize_custom_provider_entry in hermes_cli/config.py).
fb_key_env = (fb.get("key_env") or fb.get("api_key_env") or "").strip()

Suggested action

Since main is already fixed, this is a release-engineering rather than code change. Two options for maintainers:

1. Cut a patch release (v2026.5.8 or v2026.5.7.1) that includes 6ddc48b0. Users pinned to release tags will then pick it up via normal upgrade.
2. Cherry-pick 6ddc48b0 onto a release/v2026.5.x branch if one exists and re-tag.

In addition, to make the failure mode less hostile to debug if a similar slip happens in the future, please also consider:

• Emitting a WARNING when a fallback_providers entry ends up with neither api_key nor a resolvable key_env/api_key_env. Currently the only signal is the upstream provider's 401 message with ****ired — which is almost impossible to attribute without reading the source. Even a single log line like "fallback entry '<model>' has no resolvable api_key — request will be sent with placeholder 'no-key-required' and will 401 on auth-required endpoints" would have saved hours of forensics on the user side.
• (Optional) Pipe fallback_providers entries through _normalize_custom_provider_entry at config load time, the way providers: entries already are, so camelCase aliases (apiKeyEnv, keyEnv) and any future aliases work uniformly without duplicating the alias list at each consumption site.

Workaround for users on v2026.5.7

Rename api_key_env: → key_env: in every entry under fallback_providers:.

Related — distinct root causes, same "no-key-required" symptom

• #9318 — per-task auxiliary override (e.g. auxiliary.session_search) with empty api_key falling through. That path is fixable by inheriting model.api_key. Does not address fallback_providers because users intentionally specify different keys per fallback entry (e.g. ARK_API_KEY for kimi, DEEPSEEK_API_KEY for deepseek), which can't come from a single model.api_key.
• #16308 — fixes _resolve_task_provider_model() to consult PROVIDER_REGISTRY credentials when base_url overrides a named provider. Does not touch fallback_providers consumption in run_agent.py.
• #22665 — original PR; 6ddc48b0 is its carve-out for the alias fix.
• "no-key-required" placeholder introduced in PR #3842 (commit 3cc50532d, 2026-03-29). Intent was correct for local no-auth servers; this issue is about the placeholder being reachable from remote-endpoint code paths via the alias gap.
• Alias support added in commit c449cd1af (2026-04-13, "fix(config): restore custom providers after v11→v12 migration") — added the alias for providers: but did not extend it to fallback_providers: at the time, which is what 6ddc48b0 belatedly closed.