hermes - 💡(How to fix) Fix vision_analyze and browser_vision ignore auxiliary.vision — route images to main model

Bug Description

vision_analyze and browser_vision tools fail with unknown variant image_url, expected text even when auxiliary.vision is explicitly configured to point to a vision-capable model (OpenAI GPT-4o-mini).

The tools appear to route images to the main model (e.g. DeepSeek, which only accepts text messages) instead of using the configured auxiliary.vision provider/model.

Steps to Reproduce

1. Configure a main model that does NOT support images (e.g. DeepSeek V4)
2. Set up a vision-capable auxiliary model:

   hermes config set auxiliary.vision.provider openai
   hermes config set auxiliary.vision.model gpt-4o-mini

3. Ensure OPENAI_API_KEY is set in .env
4. Restart the gateway (hermes gateway restart)
5. Send an image via Telegram DM
6. Call vision_analyze(image_url=...) or open image in browser then browser_vision()

Expected Behavior

The image analysis request should be routed to the configured auxiliary.vision model (GPT-4o-mini), which supports image_url messages.

Actual Behavior

Both tools return HTTP 400:

unknown variant `image_url`, expected `text`

The error indicates the image is being sent to the main model (DeepSeek), which does not support image inputs.

Workaround

Direct OpenAI API call via Python with GPT-4o-mini works fine — same image, same API key, just routed directly instead of through auxiliary.vision.

Environment

• Hermes version: latest (May 2026)
• OS: Linux (Raspbian Trixie, Raspberry Pi 5, arm64)
• Main model: deepseek-v4-pro via DeepSeek provider
• Auxiliary vision config:

  auxiliary:
    vision:
      provider: openai
      model: gpt-4o-mini

• OPENAI_API_KEY valid (confirmed by direct GPT-4o-mini call)
• vision toolset enabled

Additional Context

After gateway restart, vision_analyze disappeared from available tools — only browser_vision remained (fails same way).