hermes - 💡(How to fix) Fix [Bug]: `vision_analyze` fails on Ollama Cloud because vision auto-detection falls through to non-vision models [1 comments, 2 participants]

Bug Description

When using ollama-cloud as the main provider with a vision-capable model (kimi-k2.6), the vision_analyze tool fails with Error code: 400 - this model does not support image input.

The root cause is in resolve_vision_provider_client() in agent/auxiliary_client.py. When the main provider (ollama-cloud) is NOT in _VISION_AUTO_PROVIDER_ORDER (which only contains ("openrouter", "nous")), the code tries resolve_provider_client(main_provider, vision_model) at line ~2060. It then uses whatever model is configured as default in config.yaml instead of the actually active model.

Environment

• Hermes Agent v0.10.0 (2026.4.16)
• macOS
• Provider: ollama-cloud
• Main model: kimi-k2.6
• Config default: minimax-m2.7 (a text-only model)

Relevant Config

model:
  api_key: ollama
  base_url: https://ollama.com/v1
  default: minimax-m2.7    # <-- This is what gets used for vision
  provider: ollama-cloud

auxiliary:
  vision:
    provider: auto         # DEFAULT — causes the bug
    model: ''

Root Cause Analysis

In agent/auxiliary_client.py, line ~2059:

1. main_provider = "ollama-cloud"
2. main_model = "minimax-m2.7" (from config.yaml model.default)
3. vision_model = _PROVIDER_VISION_MODELS.get(main_provider, main_model) -> "minimax-m2.7" (because ollama-cloud is not in _PROVIDER_VISION_MODELS)
4. The non-vision model minimax-m2.7 gets used for the vision API call
5. Ollama Cloud returns 400 because minimax-m2.7 does not support image input

Evidence

Direct API tests with openai SDK against https://ollama.com/v1:

The user's main chat model is kimi-k2.6 (as shown in models_dev_cache.json and the active model header), but _read_main_model() reads config.yaml's model.default: minimax-m2.7.

Expected Behavior

The vision tool should respect the ACTUAL currently active model, not the static model.default from config. When a user has switched to kimi-k2.6 via hermes model, vision should use kimi-k2.6.

Current Workaround

Explicitly configure vision auxiliary to use the correct model:

auxiliary:
  vision:
    provider: ollama-cloud
    model: kimi-k2.6

However, this is fragile — users must manually sync their vision model with their active chat model.

Suggested Fix

One of two approaches:

Option A (preferred): In _resolve_task_provider_model() for task="vision", fallback to the runtime active model instead of config default when auto-detecting.

Option B: Add ollama-cloud to _VISION_AUTO_PROVIDER_ORDER and/or add a vision-aware default model mapping for ollama-cloud in _API_KEY_PROVIDER_AUX_MODELS.

Option C: In resolve_vision_provider_client(), when the main provider is ollama-cloud, check models_dev_cache.json for the actually active model instead of _read_main_model().

Related Code

• agent/auxiliary_client.py line ~2059: vision_model = _PROVIDER_VISION_MODELS.get(main_provider, main_model)
• agent/auxiliary_client.py line ~985: def _read_main_model() reads config.yaml model.default (not the runtime active model)
• agent/auxiliary_client.py line ~1939: _VISION_AUTO_PROVIDER_ORDER = ("openrouter", "nous") — ollama-cloud is not included
• agent/auxiliary_client.py line ~146: "ollama-cloud": "nemotron-3-nano:30b" in _API_KEY_PROVIDER_AUX_MODELS

Checklist

• I can reproduce this consistently with the steps above
• I have tested the workaround
• I have searched existing issues for duplicates

Steps to Reproduce

See description

Expected Behavior

Auto detection should work

Actual Behavior

See desc

Affected Component

CLI (interactive chat)

Messaging Platform (if gateway-related)

N/A (CLI only)

Debug Report

N/A

Operating System

MacOS 26.4.1

Python Version

No response

Hermes Version

No response

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

No response

Proposed Fix (optional)

No response

Are you willing to submit a PR for this?

• I'd like to fix this myself and submit a PR