hermes - ✅(Solved) Fix [Feature]: Typed Config-Runtime Contract — eliminate silent config/state/hook binding gaps [3 pull requests, 1 participants]

hermes2026-05-19 23:38:04

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

NousResearch/hermes-agent#28984•Fetched 2026-05-20 04:00:47

View on GitHub

Comments

Participants

Timeline

Reactions

Author

zccyman

Participants

zccyman

Timeline (top)

cross-referenced ×3labeled ×3renamed ×1

Error Message

| Impact Severity | 7/10 | Silent failures, hard to debug, user sees wrong behavior with no error |

Root Cause

Dimension	Score	Rationale
Impact Breadth	9/10	17/100 open bugs (17%) share this root cause — #1 recurring pattern
Impact Severity	7/10	Silent failures, hard to debug, user sees wrong behavior with no error
Fix Leverage	9/10	One structural fix prevents entire bug class; individual ~30-line fixes don't
Implementation Feasibility	5/10	Touches core modules, but can be rolled out incrementally
Upstream Receptiveness	6/10	Architectural FRs are harder to accept, but 17-bug evidence is compelling
Proof of Concept	8/10	5 fixes already demonstrate the pattern; `get_custom_provider_model_field()` is a working micro-example

Fix Action

Fixed

Fixed by PR: fix: read max_tokens from custom_providers per-model config (https://github.com/NousResearch/hermes-agent/pull/28988)
Fixed by PR: feat: Config-Runtime Contract Registry (Phase 1) + fix #28046 + fix #28863 (https://github.com/NousResearch/hermes-agent/pull/28995)

PR fix notes

PR #28988: fix: read max_tokens from custom_providers per-model config

Repository: NousResearch/hermes-agent
Author: zccyman
State: closed | merged: False
Link: https://github.com/NousResearch/hermes-agent/pull/28988

Description (problem / solution / changelog)

Summary

Fixes #28046 — max_tokens configured under custom_providers[].models.<model>.max_tokens was silently ignored, always defaulting to 4096.

The codebase already had get_custom_provider_context_length() for reading per-model context_length from custom_providers, but no equivalent for max_tokens.

Changes

Extract get_custom_provider_model_field() — generic lookup helper that searches custom_providers entries for any per-model field. Replaces the inline logic in get_custom_provider_context_length().
Add get_custom_provider_max_tokens() — thin wrapper over the generic helper, symmetric to get_custom_provider_context_length().
Read max_tokens in agent_init.py — after the existing context_length custom_providers lookup, adds a symmetric block that calls get_custom_provider_max_tokens() when agent.max_tokens is None.
10 regression tests — full coverage for the new lookup: matching, trailing-slash insensitivity, zero/negative rejection, string coercion, coexistence with context_length, first-match-wins, None inputs.

Backward Compatibility

get_custom_provider_context_length() signature unchanged — custom_providers remains a positional arg (3 existing tests pass without modification)
New code only runs when agent.max_tokens is None (no override from top-level config or constructor) — pure additive fallback
22/22 tests pass (12 existing context_length + 10 new max_tokens)

Root Cause Pattern

This is the same pattern as #28961 (pre_tool_call missing session_id) and #28984 (Typed Plugin Hook Protocol FR): configuration/state flows through call chains without schema enforcement, so adding a new config field requires manually updating every bridge — and omissions are silent.

Changed files

.dev-workflow/code-graph.db (added, +0/-0)
agent/agent_init.py (modified, +14/-0)
hermes_cli/config.py (modified, +66/-25)
tests/hermes_cli/test_custom_provider_max_tokens.py (added, +158/-0)

PR #28995: feat: Config-Runtime Contract Registry (Phase 1) + fix #28046 + fix #28863

Repository: NousResearch/hermes-agent
Author: zccyman
State: open | merged: False
Link: https://github.com/NousResearch/hermes-agent/pull/28995

Description (problem / solution / changelog)

Summary

Implements Phase 1 of the Typed Config-Runtime Contract — a declarative config binding registry that catches orphan config fields at startup, before users discover them silently.

Along the way, fixes two concrete bugs that motivated this work.

Bug Fixes

Fix #28046: `max_tokens` silently ignored from `custom_providers` per-model config

Problem: Only context_length was read from custom_providers[].models.<model>. The max_tokens field was silently ignored, always defaulting to 4096.

Fix: Extract get_custom_provider_model_field() as a generic lookup helper, add get_custom_provider_max_tokens() symmetric to context_length, and read it in agent_init.py after the existing context_length lookup.

Fix #28863: `terminal.docker_extra_args` silently dropped

Problem: docker_extra_args was declared in DEFAULT_CONFIG but missing from the _terminal_env_map bridge in gateway/run.py, so it was never passed to the terminal tool.

Fix: Add the mapping entry.

Feature: Config-Runtime Contract Registry (Phase 1)

New module: `hermes_cli/config_contracts.py`

CONFIG_BINDINGS — declarative registry of 29 config→runtime bindings (all terminal.* env vars, model.max_tokens, model.context_length, custom_providers wildcards)
get_nested() — resolves dotted keys with wildcard support, including list-of-dicts (for custom_providers)
validate_config_bindings() — cross-checks loaded config against registry, returns warning list
get_binding_report() — human-readable status report for debugging

Gateway integration

At startup, after env var bridging, the gateway runs validate_config_bindings() and logs any warnings. This is purely additive — warnings only, no behavior change, wrapped in try/except so it can never block startup.

How this prevents future bugs

When a developer adds a new terminal.* key to DEFAULT_CONFIG, the contract registry immediately shows it as unregistered. The next person who reviews the code sees the gap before shipping.

Test Coverage

Test file	Tests	Status
`test_config_contracts.py`	17	All pass
`test_custom_provider_max_tokens.py`	10	All pass
`test_custom_provider_context_length.py`	12	All pass (existing, unchanged)
Total	39	All pass

Backward Compatibility

get_custom_provider_context_length() signature unchanged
get_custom_provider_max_tokens() is a new function (purely additive)
config_contracts.py is a new module (purely additive)
Gateway validation is wrapped in try/except (cannot break startup)
Zero behavior change to any existing code path

Implements Phase 1 of #28984
Complements #27342 (config schema validation — different scope)
Supersedes PR #28988 (this PR includes its fix plus the contract layer)

Files Changed

File	Change
`hermes_cli/config_contracts.py`	New — Contract registry module (235 lines)
`tests/hermes_cli/test_config_contracts.py`	New — 17 tests
`tests/hermes_cli/test_custom_provider_max_tokens.py`	New — 10 tests
`agent/agent_init.py`	+14 — Read max_tokens from custom_providers
`hermes_cli/config.py`	+66/-25 — Generic helper + max_tokens function
`gateway/run.py`	+14 — docker_extra_args mapping + contract validation

Changed files

.dev-workflow/code-graph.db (added, +0/-0)
agent/agent_init.py (modified, +14/-0)
gateway/run.py (modified, +13/-0)
hermes_cli/config.py (modified, +66/-25)
hermes_cli/config_contracts.py (added, +235/-0)
tests/hermes_cli/test_config_contracts.py (added, +142/-0)
tests/hermes_cli/test_custom_provider_max_tokens.py (added, +158/-0)

PR #29048: feat: Typed Hook Payloads (Phase 2) + Path Parity Tests (Phase 3)

Repository: NousResearch/hermes-agent
Author: zccyman
State: open | merged: False
Link: https://github.com/NousResearch/hermes-agent/pull/29048

Description (problem / solution / changelog)

Summary

Implements Phase 2 (Typed Hook Payloads) and Phase 3 (Path Parity Tests) of FR #28984.

Phase 1 was submitted separately as PR #28995.

Phase 2 — Typed Hook Payloads

New file: hermes_cli/hook_payloads.py (276 lines)

Defines frozen dataclass payloads for all 14 hook types:

Hook	Dataclass
`pre_tool_call`	`PreToolCallPayload`
`post_tool_call`	`PostToolCallPayload`
`transform_tool_result`	`TransformToolResultPayload`
`on_session_start`	`SessionStartPayload`
`on_session_end`	`SessionEndPayload`
`on_session_finalize`	`SessionFinalizePayload`
`on_session_reset`	`SessionResetPayload`
`pre_approval_request`	`PreApprovalRequestPayload`
`post_approval_response`	`PostApprovalResponsePayload`
`pre_llm_call`	`PreLlmCallPayload`
`post_llm_call`	`PostLlmCallPayload`
`pre_api_request`	`PreApiRequestPayload`
`post_api_request`	`PostApiRequestPayload`
`subagent_stop`	`SubagentStopPayload`

payload_to_kwargs() converts dataclass → plain dict, preserving cb(**kwargs) compatibility for all existing plugin callbacks.

5 call sites migrated (adds type safety, zero breaking changes):

File	Hook
`hermes_cli/plugins.py`	`get_pre_tool_call_block_message()`
`agent/conversation_loop.py`	`on_session_start`, `pre_llm_call`
`model_tools.py`	`post_tool_call`, `transform_tool_result`

New tests: tests/hermes_cli/test_hook_payloads.py — 21 tests covering all payloads, frozen immutability, round-trip, registry completeness.

Phase 3 — Path Parity Tests

New file: hermes_cli/path_parity.py (155 lines)

assert_field_parity(
    "fallback_model",
    {
        "gateway_run_agent": lambda: extract_gateway_fields(),
        "tui_make_agent": lambda: extract_tui_fields(),
    },
)

When any path does not consume the field, raises PathParityError with a diff showing which paths are missing it.

3 known divergences documented as failing tests:

#28753: TUI doesn't pass fallback_model to AIAgent
#28746: idle-expiry path doesn't emit session:end event
#28637: /model switch loses per-model token usage

New tests: tests/hermes_cli/test_path_parity.py — 13 tests covering the helper functions and documented divergence patterns.

What This Prevents

Future scenario	Without Phase 2	With Phase 2
New param added to `pre_tool_call`	2 of 3 call sites miss it	Dataclass forces all sites to provide it
Hook field renamed	Silent breakage in plugins	Frozen dataclass catches at construction

Future scenario	Without Phase 3	With Phase 3
`fallback_model` added to gateway but forgotten in TUI	Silent divergence (#28753)	`assert_field_parity` fails in CI
`session:end` forgotten in one exit path	Bug hidden until user hits expiry path (#28746)	Test documents expected parity

Test Results

tests/hermes_cli/test_hook_payloads.py  21 passed
tests/hermes_cli/test_path_parity.py   13 passed
tests/hermes_cli/test_plugins.py       105 passed
tests/test_model_tools.py               passed
tests/test_transform_tool_result_hook.py passed
tests/agent/test_plugin_llm.py          passed
tests/agent/test_shell_hooks.py         passed
─────────────────────────────────────────────
Total                                     139 passed

Refs: FR #28984 | PR #28995 (Phase 1)

Changed files

.dev-workflow/code-graph.db (added, +0/-0)
agent/conversation_loop.py (modified, +6/-4)
hermes_cli/hook_payloads.py (added, +276/-0)
hermes_cli/path_parity.py (added, +155/-0)
hermes_cli/plugins.py (modified, +7/-2)
hook_system_analysis.md (added, +348/-0)
model_tools.py (modified, +6/-4)
plans/path_parity_analysis.md (added, +135/-0)
tests/hermes_cli/test_hook_payloads.py (added, +245/-0)
tests/hermes_cli/test_path_parity.py (added, +198/-0)

Code Example

CONFIG_CONTRACTS = {
    "terminal.docker_extra_args": {
        "env_var": "TERMINAL_DOCKER_EXTRA_ARGS",
        "consumer": "gateway.run._terminal_env_map",
        "type": list[str],
    },
    "custom_providers.*.models.*.max_tokens": {
        "consumer": "agent.agent_init",
        "fallback": 4096,
        "type": int,
    },
}

---

@dataclass
class PreToolCallPayload:
    session_id: str
    tool_call_id: str
    tool_name: str
    tool_input: dict
    # Future fields added here automatically propagate

# Single entry point constructs the payload — all call sites get every field

---

def assert_config_path_parity(field: str, paths: list[str]):
    """Fail CI if any declared path doesn't consume the field."""

RAW_BUFFERClick to expand / collapse

Feature Request: Typed Config-Runtime Contract

Supersedes: This FR has been expanded from the original "Typed Plugin Hook Protocol" scope. Hook calls are one sub-pattern; this FR now covers the full contract gap across config→runtime, state→path, and interface→caller boundaries.

Value Score: 44/60 (73%) — HIGH VALUE

Dimension	Score	Rationale
Impact Breadth	9/10	17/100 open bugs (17%) share this root cause — #1 recurring pattern
Impact Severity	7/10	Silent failures, hard to debug, user sees wrong behavior with no error
Fix Leverage	9/10	One structural fix prevents entire bug class; individual ~30-line fixes don't
Implementation Feasibility	5/10	Touches core modules, but can be rolled out incrementally
Upstream Receptiveness	6/10	Architectural FRs are harder to accept, but 17-bug evidence is compelling
Proof of Concept	8/10	5 fixes already demonstrate the pattern; `get_custom_provider_model_field()` is a working micro-example

No existing FR covers this scope. Related:

#27342 (Config schema validation) is complementary — validates config shape at load time, doesn't ensure consumers read fields
#28621 (Declarative tool preview) is the same structural pattern in a different domain (display)

Problem

Hermes has no contract layer between config/state producers and consumers. When a new field is added to config.yaml, a hook signature, or a state object, there is no mechanism to ensure every downstream consumer picks it up. The result: silent failures that only surface when users hit the un-updated code path.

This is not a theory — it is the #1 recurring bug pattern in the open issue tracker.

Evidence: 17 bugs, one root cause, four sub-patterns

We analyzed 100 open bug issues and found 17 that follow the exact same structural defect. They break into four sub-patterns:

Sub-pattern 1: Config field declared, no consumer (4 bugs)

New config.yaml key added and documented, but the bridge code that maps it to a runtime variable/env-var was never updated.

Bug	Symptom
#28046 ✅	`custom_providers[].models.*.max_tokens` ignored, always defaults to 4096
#28863	`terminal.docker_extra_args` silently dropped — missing from `_terminal_env_map`
#28651	`web_tools` hardcodes provider list, ignores configured providers
#28034	`/model --global` doesn't persist when using the visual model picker

Sub-pattern 2: Path A works, path B broken (5 bugs)

Feature works correctly on one execution path (e.g., startup) but is broken on another (e.g., /model switch, gateway restart, fallback activation). The logic was copy-pasted or reimplemented instead of shared.

Bug	Symptom
#28753	TUI doesn't propagate `fallback_model`/`fallback_providers` — gateway path works
#28825	OpenAI-compatible API doesn't honour `tools` param — Anthropic path works
#28746	`session:end` event not emitted from idle-expiry/auto-reset path — normal close works
#28637	Per-model token usage lost during `/model` switch — init path records it
#28023	Credential pool strategy not honoured on fallback — startup path reads it

Sub-pattern 3: Interface expanded, caller missed (3 bugs)

A hook signature, callback, or protocol method was extended with new parameters, but not all call sites were updated.

Bug	Symptom
#28961 ✅	`pre_tool_call` hook missing `session_id`/`tool_call_id` in 2 of 3 call sites
#28296 ✅	OpenViking missing `on_session_switch()` — interface method added, implementation missed
#28662	`hermes cron list` crashes on MCP-created jobs — `schedule` field type assumption differs

Sub-pattern 4: State added, not propagated (3 bugs)

A new field was added to a state dict/object, but not all paths that serialize/deserialize/transform that state preserve the new field.

Bug	Symptom
#28841 ✅	Message timestamp lost during fork/compress/branch — always DB write-time
#28632	Gateway restart leaves launchd service unloaded — stop path cleans up, restart doesn't
#28489	Gateway persists invalid `/model` status override and keeps reusing it

(✅ = already fixed, listed here as evidence the pattern is real and fixable)

Also related (same structural pattern, different domain)

Bug	Domain	Same root cause
#28598 ✅	Display	`build_tool_preview()` hardcoded if-elif chain — new tool → forgotten entry → #28621
#28663 ✅	Gateway	Exec quick commands blocked during drain — one path checked, other didn't

Why individual fixes don't scale

Each of the 5 bugs we fixed required ~30 lines of targeted code. But fixing #28046 (max_tokens) did nothing to prevent #28863 (docker_extra_args) — they're the same structural defect manifesting in different config keys. Every new config field or code path is a ticking time bomb until someone reports it.

The pattern will keep recurring as long as producer→consumer bindings remain implicit.

Proposed Solution

A lightweight contract layer that makes bindings declarative and verifiable — not a big-bang rewrite, but an incremental rollout:

Phase 1: Config Field Registry + Startup Validator

Define a registry mapping config keys to their expected consumers:

CONFIG_CONTRACTS = {
    "terminal.docker_extra_args": {
        "env_var": "TERMINAL_DOCKER_EXTRA_ARGS",
        "consumer": "gateway.run._terminal_env_map",
        "type": list[str],
    },
    "custom_providers.*.models.*.max_tokens": {
        "consumer": "agent.agent_init",
        "fallback": 4096,
        "type": int,
    },
}

On startup, validate every declared field has at least one active consumer. Emit warnings (not errors) for:

Orphan fields (declared in config, no consumer registered)
Stale bindings (consumer references a field that no longer exists)

Zero behavior change — purely additive observability.

Phase 2: Typed Hook Payloads

Replace ad-hoc kwargs in plugin hook invocations with typed payloads:

@dataclass
class PreToolCallPayload:
    session_id: str
    tool_call_id: str
    tool_name: str
    tool_input: dict
    # Future fields added here automatically propagate

# Single entry point constructs the payload — all call sites get every field

This is the original scope from this FR (#28961, #25204, #7344). A Protocol-based approach ensures any new field added to the payload is structurally visible to all consumers.

Phase 3: Path Parity Tests

A test helper that verifies: if path A (startup) reads/configures field X, then path B (/model switch, gateway restart, fallback) must also read X.

def assert_config_path_parity(field: str, paths: list[str]):
    """Fail CI if any declared path doesn't consume the field."""

This turns "path divergence" bugs into CI failures before they reach users.

What this prevents

Future scenario	Without contract	With contract
New config field `terminal.gpu_layers` added	Developer forgets to add to env_map → silent drop until user reports	Registry shows orphan field at startup → caught immediately
New hook param `retry_count` added to `pre_tool_call`	Works in sequential path, forgotten in concurrent path	Typed payload means all paths get all fields
Model switch adds new state field	Init preserves it, switch loses it	Path parity test fails in CI
New tool added to plugin	`build_tool_preview()` shows generic output (#28598)	Declarative preview registered at tool definition time (#28621)

Existing proof of concept

The generic get_custom_provider_model_field() function introduced in PR #28988 (fixing #28046) is a working micro-example of this approach: instead of separate get_custom_provider_context_length() and get_custom_provider_max_tokens() functions with duplicated logic, we extracted a single generic lookup that any new per-model config field can use without writing new bridge code.

Fixes (as evidence): #28046, #28961, #28841, #28663, #28296
Would prevent: #28863, #28662, #28034, #28753, #28651, #28489, #28023, #28746, #28637, #28825, #28632, #28055
Related: #27342 (complementary), #28621 (same pattern, different domain)

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #tokenizer error #prompt formatting #chain error #conversation history

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

hermes - ✅(Solved) Fix [Feature]: Typed Config-Runtime Contract — eliminate silent config/state/hook binding gaps [3 pull requests, 1 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fixed

PR fix notes

PR #28988: fix: read max_tokens from custom_providers per-model config

Description (problem / solution / changelog)

Summary

Changes

Backward Compatibility

Root Cause Pattern

Changed files

PR #28995: feat: Config-Runtime Contract Registry (Phase 1) + fix #28046 + fix #28863

Description (problem / solution / changelog)

Summary

Bug Fixes

Fix #28046: max_tokens silently ignored from custom_providers per-model config

Fix #28863: terminal.docker_extra_args silently dropped

Feature: Config-Runtime Contract Registry (Phase 1)

New module: hermes_cli/config_contracts.py

Gateway integration

How this prevents future bugs

Test Coverage

Backward Compatibility

Related

Files Changed

Changed files

PR #29048: feat: Typed Hook Payloads (Phase 2) + Path Parity Tests (Phase 3)

Description (problem / solution / changelog)

Summary

Phase 2 — Typed Hook Payloads

Phase 3 — Path Parity Tests

What This Prevents

Test Results

Changed files

Code Example

Feature Request: Typed Config-Runtime Contract

Value Score: 44/60 (73%) — HIGH VALUE

Problem

Evidence: 17 bugs, one root cause, four sub-patterns

Sub-pattern 1: Config field declared, no consumer (4 bugs)

Sub-pattern 2: Path A works, path B broken (5 bugs)

Sub-pattern 3: Interface expanded, caller missed (3 bugs)

Sub-pattern 4: State added, not propagated (3 bugs)

Also related (same structural pattern, different domain)

Why individual fixes don't scale

Proposed Solution

Phase 1: Config Field Registry + Startup Validator

Phase 2: Typed Hook Payloads

Phase 3: Path Parity Tests

What this prevents

Existing proof of concept

Still need to ship something?

RELATED_DISCOVERY

TRENDING

Fix #28046: `max_tokens` silently ignored from `custom_providers` per-model config

Fix #28863: `terminal.docker_extra_args` silently dropped

New module: `hermes_cli/config_contracts.py`