hermes - ✅(Solved) Fix Tool-layer env vars defeat config.yaml auxiliary routing for vision and extraction models [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
NousResearch/hermes-agent#14693Fetched 2026-04-24 06:15:14
View on GitHub
Comments
0
Participants
1
Timeline
6
Reactions
0
Author
Participants
Timeline (top)
labeled ×6

Error Message

def _resolve_vision_model_for_handler() -> Optional[str]: try: cfg = read_raw_config() vision_cfg = cfg.get("auxiliary", {}).get("vision", {}) if isinstance(cfg, dict) else {} cfg_model = str(vision_cfg.get("model", "")).strip() if isinstance(vision_cfg, dict) else "" except Exception: cfg_model = "" if cfg_model: return None # let aux client resolve from config.yaml return os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None

Root Cause

Three helper functions follow the same broken pattern:

# tools/vision_tools.py (before fix)
def _handle_vision_analyze(args, **kw):
    model = os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None
    return vision_analyze_tool(image_url, full_prompt, model)  # explicit override

# tools/browser_tool.py (before fix)
def _get_vision_model() -> Optional[str]:
    return os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None

def _get_extraction_model() -> Optional[str]:
    return os.getenv("AUXILIARY_WEB_EXTRACT_MODEL", "").strip() or None

The same class of bug was previously reported and fixed at the provider-detection level (#4171, #4172) and env-cleanup level (#5161), but this variant lives at the tool layer and was not caught.

Fix Action

Fix / Workaround

  • Hermes Agent version: local-patches branch at commit c97fb6aa
  • OS: Ubuntu Linux (aarch64), Zoidberg VPS
  • Python: 3.11.15
  • Provider stack: Anthropic (main) → LiteLLM proxy (localhost:4000) → OpenRouter for auxiliary tasks
  • Auxiliary vision config:
    auxiliary:
      vision:
        model: hermes-aux-vision
        base_url: http://127.0.0.1:4000/v1

Tentative Workaround Applied

We patched the three helpers to consult config.yaml first via hermes_cli.config.read_raw_config(). When config specifies a model, the helper returns None (letting the aux client resolve routing from config). The env var is preserved as a fallback for env-only setups.

PR fix notes

PR #15009: fix(tools): respect config.yaml auxiliary routing in vision/browser model helpers

Description (problem / solution / changelog)

What does this PR do?

Three tool-layer helpers read AUXILIARY_*_MODEL env vars directly via os.getenv() and pass the value as an explicit model= override to the centralized auxiliary client. When the env var is stale (e.g. set at process startup from an earlier config.yaml and Hermes is reconfigured without a restart), this silently defeats the user's current config.yaml auxiliary.<task>.model routing and the vision/web-extract calls go to the wrong model.

The affected helpers:

  • tools/vision_tools.py::_handle_vision_analyze — reads AUXILIARY_VISION_MODEL and passes it to vision_analyze_tool(...).
  • tools/browser_tool.py::_get_vision_model() — reads AUXILIARY_VISION_MODEL, used by browser_vision.
  • tools/browser_tool.py::_get_extraction_model() — reads AUXILIARY_WEB_EXTRACT_MODEL, used by _extract_page_content_for_llm.

Each helper now consults hermes_cli.config.read_raw_config() first. When config.yaml specifies an auxiliary.<task>.model, the helper returns None so the centralized aux client resolves provider/base_url/model from config. The env var remains a fallback for env-only setups. A failure to read config logs a warning and falls back to the env var so the tool still works.

Related Issue

Fixes #14693

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • tools/vision_tools.py: added _resolve_vision_model_for_handler(); _handle_vision_analyze calls it instead of reading the env var directly.
  • tools/browser_tool.py: _get_vision_model() and _get_extraction_model() now consult read_raw_config() first and only fall back to env vars when config does not specify a model.
  • tests/tools/test_browser_tool_model_resolution.py: new test file with 13 tests covering both helpers (config beats env, env-only fallback, neither set, config-read failure, whitespace handling).
  • tests/tools/test_vision_tools.py: 2 new precedence tests and 2 existing tests updated to mock read_raw_config (the previous mocks targeted a symbol the module never imported).

Out of scope (noted from the issue)

The reporter mentioned a similar env-var-overrides-config pattern in tools/web_tools.py::_resolve_web_extract_client. This PR intentionally does not touch it — keeping the change focused. Happy to file a follow-up if maintainers want.

How to Test

  1. Set auxiliary.vision.model: hermes-aux-vision (or any non-default) in ~/.hermes/config.yaml.
  2. Export a stale env var: AUXILIARY_VISION_MODEL=stale-model.
  3. Before this change: vision calls go to stale-model. After this change: vision routes through the centralized client based on the config value.
  4. Run targeted tests:
    source .venv/bin/activate
    pytest tests/tools/test_browser_tool_model_resolution.py tests/tools/test_vision_tools.py -v
  5. Run the broader tool suite:
    pytest tests/tools/ -q

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes
  • I've tested on my platform: macOS (Darwin 25.4.0, Apple Silicon), Python 3.11

Documentation & Housekeeping

  • Updated relevant documentation — N/A
  • Updated cli-config.yaml.example — N/A
  • Updated contributing / agents docs — N/A
  • Considered cross-platform impact — N/A (pure Python config-read change)
  • Updated tool descriptions/schemas — N/A

Changed files

  • tests/tools/test_browser_tool_model_resolution.py (added, +141/-0)
  • tests/tools/test_vision_tools.py (modified, +41/-1)
  • tools/browser_tool.py (modified, +26/-2)
  • tools/vision_tools.py (modified, +13/-1)

Code Example

auxiliary:
    vision:
      model: hermes-aux-vision
      base_url: http://127.0.0.1:4000/v1

---

# tools/vision_tools.py (before fix)
def _handle_vision_analyze(args, **kw):
    model = os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None
    return vision_analyze_tool(image_url, full_prompt, model)  # explicit override

# tools/browser_tool.py (before fix)
def _get_vision_model() -> Optional[str]:
    return os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None

def _get_extraction_model() -> Optional[str]:
    return os.getenv("AUXILIARY_WEB_EXTRACT_MODEL", "").strip() or None

---

def _resolve_vision_model_for_handler() -> Optional[str]:
    try:
        cfg = read_raw_config()
        vision_cfg = cfg.get("auxiliary", {}).get("vision", {}) if isinstance(cfg, dict) else {}
        cfg_model = str(vision_cfg.get("model", "")).strip() if isinstance(vision_cfg, dict) else ""
    except Exception:
        cfg_model = ""
    if cfg_model:
        return None  # let aux client resolve from config.yaml
    return os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None

---

def _get_vision_model() -> Optional[str]:
    try:
        cfg = read_raw_config()
        vision_cfg = cfg.get("auxiliary", {}).get("vision", {}) if isinstance(cfg, dict) else {}
        cfg_model = str(vision_cfg.get("model", "")).strip() if isinstance(vision_cfg, dict) else ""
    except Exception:
        cfg_model = ""
    if cfg_model:
        return None
    return os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None

def _get_extraction_model() -> Optional[str]:
    try:
        cfg = read_raw_config()
        we_cfg = cfg.get("auxiliary", {}).get("web_extract", {}) if isinstance(cfg, dict) else {}
        cfg_model = str(we_cfg.get("model", "")).strip() if isinstance(we_cfg, dict) else ""
    except Exception:
        cfg_model = ""
    if cfg_model:
        return None
    return os.getenv("AUXILIARY_WEB_EXTRACT_MODEL", "").strip() or None

---

"""Tests for browser_tool model resolution: config.yaml > env var precedence."""

import os
from unittest.mock import patch

from tools.browser_tool import _get_vision_model, _get_extraction_model


class TestGetVisionModel:
    def test_config_yaml_vision_model_beats_env_var(self):
        """Regression: stale AUXILIARY_VISION_MODEL must NOT override config.yaml."""
        with (
            patch.dict(os.environ, {"AUXILIARY_VISION_MODEL": "stale-env-model"}),
            patch(
                "tools.browser_tool.read_raw_config",
                return_value={"auxiliary": {"vision": {"model": "hermes-aux-vision"}}},
            ),
        ):
            result = _get_vision_model()
        assert result != "stale-env-model"
        assert result in (None, "hermes-aux-vision")

    def test_env_var_used_when_config_empty(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_VISION_MODEL": "env-only/model"}),
            patch("tools.browser_tool.read_raw_config", return_value={"auxiliary": {"vision": {}}}),
        ):
            assert _get_vision_model() == "env-only/model"

    def test_returns_none_when_both_empty(self):
        with (
            patch.dict(os.environ, {}, clear=False),
            patch("tools.browser_tool.read_raw_config", return_value={"auxiliary": {"vision": {}}}),
        ):
            os.environ.pop("AUXILIARY_VISION_MODEL", None)
            assert _get_vision_model() is None

    def test_config_read_failure_falls_back_to_env(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_VISION_MODEL": "env-fallback"}),
            patch("tools.browser_tool.read_raw_config", side_effect=RuntimeError("config broken")),
        ):
            assert _get_vision_model() == "env-fallback"


class TestGetExtractionModel:
    def test_config_yaml_extraction_model_beats_env_var(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_WEB_EXTRACT_MODEL": "stale-env-model"}),
            patch(
                "tools.browser_tool.read_raw_config",
                return_value={"auxiliary": {"web_extract": {"model": "hermes-aux-web-extract"}}},
            ),
        ):
            result = _get_extraction_model()
        assert result != "stale-env-model"
        assert result in (None, "hermes-aux-web-extract")

    def test_env_var_used_when_config_empty(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_WEB_EXTRACT_MODEL": "env-only/extract"}),
            patch("tools.browser_tool.read_raw_config", return_value={"auxiliary": {"web_extract": {}}}),
        ):
            assert _get_extraction_model() == "env-only/extract"

    def test_returns_none_when_both_empty(self):
        with (
            patch.dict(os.environ, {}, clear=False),
            patch("tools.browser_tool.read_raw_config", return_value={"auxiliary": {"web_extract": {}}}),
        ):
            os.environ.pop("AUXILIARY_WEB_EXTRACT_MODEL", None)
            assert _get_extraction_model() is None

    def test_config_read_failure_falls_back_to_env(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_WEB_EXTRACT_MODEL": "env-fallback"}),
            patch("tools.browser_tool.read_raw_config", side_effect=RuntimeError("config broken")),
        ):
            assert _get_extraction_model() == "env-fallback"

---

$ pytest tests/tools/test_browser_tool_model_resolution.py tests/tools/test_vision_tools.py -q
113 passed

$ pytest tests/agent/test_auxiliary_config_bridge.py tests/agent/test_vision_resolved_args.py -q
20 passed
RAW_BUFFERClick to expand / collapse

Bug Report: Tool-layer env vars (AUXILIARY_VISION_MODEL / AUXILIARY_WEB_EXTRACT_MODEL) defeat config.yaml auxiliary routing

Bug Description

Several tool-layer helpers read auxiliary model names directly from environment variables and pass them as explicit model= overrides to the centralized LLM client (call_llm / async_call_llm). When the Hermes process has been running across config changes, those env vars become stale (set at startup from the previous config.yaml) and silently override the current config.yaml settings.

The specific symptoms we hit:

  • tools/vision_tools.py::_handle_vision_analyze reads AUXILIARY_VISION_MODEL directly via os.getenv() and passes it as the model= arg to vision_analyze_tool.
  • tools/browser_tool.py::_get_vision_model() reads AUXILIARY_VISION_MODEL the same way, used by browser_vision screenshot analysis.
  • tools/browser_tool.py::_get_extraction_model() reads AUXILIARY_WEB_EXTRACT_MODEL the same way, used by _extract_page_content_for_llm.

All three follow the same pattern: env var → explicit model= override → defeats routing configured in config.yaml.

Environment

  • Hermes Agent version: local-patches branch at commit c97fb6aa
  • OS: Ubuntu Linux (aarch64), Zoidberg VPS
  • Python: 3.11.15
  • Provider stack: Anthropic (main) → LiteLLM proxy (localhost:4000) → OpenRouter for auxiliary tasks
  • Auxiliary vision config:
    auxiliary:
      vision:
        model: hermes-aux-vision
        base_url: http://127.0.0.1:4000/v1

Steps to Reproduce

  1. Start Hermes CLI or gateway with auxiliary.vision.model: hermes-aux-vision in config.yaml, routing through LiteLLM to openrouter/qwen/qwen3.6-plus.
  2. Change config.yaml to use a different model (e.g., switch the provider or model name).
  3. Without restarting Hermes, run vision_analyze or browser_vision.
  4. Observe that requests go to the old model, not the one specified in the updated config.yaml.
    • In our case: LiteLLM logged Received Model Group=gpt-5.4-mini even though config.yaml said hermes-aux-vision, producing "No endpoints found that support image input".

Expected Behavior

config.yaml should be authoritative. Tool-layer helpers should let the centralized auxiliary client resolve the model, base_url, and api_key from config, and only fall back to env vars when config does not specify a model (backward compat for env-only setups).

Actual Behavior

The tool layer reads AUXILIARY_VISION_MODEL from env and passes it as an explicit model= override to call_llm, bypassing the centralized config-based routing. The stale env value wins.

Root Cause

Three helper functions follow the same broken pattern:

# tools/vision_tools.py (before fix)
def _handle_vision_analyze(args, **kw):
    model = os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None
    return vision_analyze_tool(image_url, full_prompt, model)  # explicit override

# tools/browser_tool.py (before fix)
def _get_vision_model() -> Optional[str]:
    return os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None

def _get_extraction_model() -> Optional[str]:
    return os.getenv("AUXILIARY_WEB_EXTRACT_MODEL", "").strip() or None

The same class of bug was previously reported and fixed at the provider-detection level (#4171, #4172) and env-cleanup level (#5161), but this variant lives at the tool layer and was not caught.

Tentative Workaround Applied

We patched the three helpers to consult config.yaml first via hermes_cli.config.read_raw_config(). When config specifies a model, the helper returns None (letting the aux client resolve routing from config). The env var is preserved as a fallback for env-only setups.

tools/vision_tools.py — new helper _resolve_vision_model_for_handler():

def _resolve_vision_model_for_handler() -> Optional[str]:
    try:
        cfg = read_raw_config()
        vision_cfg = cfg.get("auxiliary", {}).get("vision", {}) if isinstance(cfg, dict) else {}
        cfg_model = str(vision_cfg.get("model", "")).strip() if isinstance(vision_cfg, dict) else ""
    except Exception:
        cfg_model = ""
    if cfg_model:
        return None  # let aux client resolve from config.yaml
    return os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None

tools/browser_tool.py_get_vision_model() and _get_extraction_model() patched with the same pattern:

def _get_vision_model() -> Optional[str]:
    try:
        cfg = read_raw_config()
        vision_cfg = cfg.get("auxiliary", {}).get("vision", {}) if isinstance(cfg, dict) else {}
        cfg_model = str(vision_cfg.get("model", "")).strip() if isinstance(vision_cfg, dict) else ""
    except Exception:
        cfg_model = ""
    if cfg_model:
        return None
    return os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None

def _get_extraction_model() -> Optional[str]:
    try:
        cfg = read_raw_config()
        we_cfg = cfg.get("auxiliary", {}).get("web_extract", {}) if isinstance(cfg, dict) else {}
        cfg_model = str(we_cfg.get("model", "")).strip() if isinstance(we_cfg, dict) else ""
    except Exception:
        cfg_model = ""
    if cfg_model:
        return None
    return os.getenv("AUXILIARY_WEB_EXTRACT_MODEL", "").strip() or None

Tests

New file: tests/tools/test_browser_tool_model_resolution.py (8 tests, covers both helpers):

"""Tests for browser_tool model resolution: config.yaml > env var precedence."""

import os
from unittest.mock import patch

from tools.browser_tool import _get_vision_model, _get_extraction_model


class TestGetVisionModel:
    def test_config_yaml_vision_model_beats_env_var(self):
        """Regression: stale AUXILIARY_VISION_MODEL must NOT override config.yaml."""
        with (
            patch.dict(os.environ, {"AUXILIARY_VISION_MODEL": "stale-env-model"}),
            patch(
                "tools.browser_tool.read_raw_config",
                return_value={"auxiliary": {"vision": {"model": "hermes-aux-vision"}}},
            ),
        ):
            result = _get_vision_model()
        assert result != "stale-env-model"
        assert result in (None, "hermes-aux-vision")

    def test_env_var_used_when_config_empty(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_VISION_MODEL": "env-only/model"}),
            patch("tools.browser_tool.read_raw_config", return_value={"auxiliary": {"vision": {}}}),
        ):
            assert _get_vision_model() == "env-only/model"

    def test_returns_none_when_both_empty(self):
        with (
            patch.dict(os.environ, {}, clear=False),
            patch("tools.browser_tool.read_raw_config", return_value={"auxiliary": {"vision": {}}}),
        ):
            os.environ.pop("AUXILIARY_VISION_MODEL", None)
            assert _get_vision_model() is None

    def test_config_read_failure_falls_back_to_env(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_VISION_MODEL": "env-fallback"}),
            patch("tools.browser_tool.read_raw_config", side_effect=RuntimeError("config broken")),
        ):
            assert _get_vision_model() == "env-fallback"


class TestGetExtractionModel:
    def test_config_yaml_extraction_model_beats_env_var(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_WEB_EXTRACT_MODEL": "stale-env-model"}),
            patch(
                "tools.browser_tool.read_raw_config",
                return_value={"auxiliary": {"web_extract": {"model": "hermes-aux-web-extract"}}},
            ),
        ):
            result = _get_extraction_model()
        assert result != "stale-env-model"
        assert result in (None, "hermes-aux-web-extract")

    def test_env_var_used_when_config_empty(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_WEB_EXTRACT_MODEL": "env-only/extract"}),
            patch("tools.browser_tool.read_raw_config", return_value={"auxiliary": {"web_extract": {}}}),
        ):
            assert _get_extraction_model() == "env-only/extract"

    def test_returns_none_when_both_empty(self):
        with (
            patch.dict(os.environ, {}, clear=False),
            patch("tools.browser_tool.read_raw_config", return_value={"auxiliary": {"web_extract": {}}}),
        ):
            os.environ.pop("AUXILIARY_WEB_EXTRACT_MODEL", None)
            assert _get_extraction_model() is None

    def test_config_read_failure_falls_back_to_env(self):
        with (
            patch.dict(os.environ, {"AUXILIARY_WEB_EXTRACT_MODEL": "env-fallback"}),
            patch("tools.browser_tool.read_raw_config", side_effect=RuntimeError("config broken")),
        ):
            assert _get_extraction_model() == "env-fallback"

Existing tests updated: tests/tools/test_vision_tools.py — 3 handler tests corrected from load_config to read_raw_config mocks (the previous mocks were broken — patch("tools.vision_tools.load_config", ...) raised AttributeError because the module does not import load_config at top level).

All pass:

$ pytest tests/tools/test_browser_tool_model_resolution.py tests/tools/test_vision_tools.py -q
113 passed

$ pytest tests/agent/test_auxiliary_config_bridge.py tests/agent/test_vision_resolved_args.py -q
20 passed

Suggested Fix

Apply the read_raw_config-first pattern to all tool-layer helpers that read AUXILIARY_*_MODEL env vars. The key principle: config.yaml is authoritative; env vars are a fallback only when config does not specify the model. This matches how agent/auxiliary_client.py::_resolve_task_provider_model already works — the problem is that tool-layer helpers bypass it by passing an explicit model= override.

A broader audit may be warranted for other env-var-driven overrides in the tool layer (e.g., AUXILIARY_WEB_EXTRACT_MODEL in tools/web_tools.py::_resolve_web_extract_client).


This issue was written using Xiaomi Mimo-V2.5-Pro (via OpenRouter) based on debugging work by Claude Opus 4.7 (via Anthropic), all within Hermes Agent itself.

extent analysis

TL;DR

Apply the read_raw_config-first pattern to all tool-layer helpers that read AUXILIARY_*_MODEL env vars to ensure config.yaml is authoritative.

Guidance

  • Identify all tool-layer helpers reading AUXILIARY_*_MODEL env vars and update them to consult config.yaml first via hermes_cli.config.read_raw_config().
  • Verify the fix by testing the updated helpers with different config.yaml settings and environment variables.
  • Consider a broader audit of env-var-driven overrides in the tool layer to ensure consistency with the agent/auxiliary_client.py::_resolve_task_provider_model behavior.
  • Review the provided test cases (tests/tools/test_browser_tool_model_resolution.py and updated tests/tools/test_vision_tools.py) to ensure the fix covers all scenarios.

Example

The updated _resolve_vision_model_for_handler() function in tools/vision_tools.py demonstrates the read_raw_config-first pattern:

def _resolve_vision_model_for_handler() -> Optional[str]:
    try:
        cfg = read_raw_config()
        vision_cfg = cfg.get("auxiliary", {}).get("vision", {}) if isinstance(cfg, dict) else {}
        cfg_model = str(vision_cfg.get("model", "")).strip() if isinstance(vision_cfg, dict) else ""
    except Exception:
        cfg_model = ""
    if cfg_model:
        return None  # let aux client resolve from config.yaml
    return os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None

Notes

The provided fix and test cases should address the specific issue with AUXILIARY_VISION_MODEL and AUXILIARY_WEB_EXTRACT_MODEL env vars. However, a more comprehensive review of the tool layer may be necessary to ensure all env-var-driven overrides are handled correctly.

Recommendation

Apply the suggested fix to

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

hermes - ✅(Solved) Fix Tool-layer env vars defeat config.yaml auxiliary routing for vision and extraction models [1 pull requests, 1 participants]