hermes - 💡(How to fix) Fix Add config option to pass inbound images directly to multimodal main models [1 comments, 2 participants]

Problem

Telegram and CLI image attachments are currently always preprocessed through the auxiliary vision pipeline before reaching the main agent model:

• Telegram media is cached locally and stored on event.media_urls / event.media_types.
• gateway/run.py::_prepare_inbound_message_text() detects image media and calls _enrich_message_with_vision().
• _enrich_message_with_vision() calls tools.vision_tools.vision_analyze_tool() with a broad "Describe everything visible..." prompt.
• The resulting text description plus local image path is prepended to the user's message.
• CLI follows the same pattern in cli.py::_preprocess_images_with_vision().

This means capable multimodal main models never receive the original image content directly from Telegram/CLI. They receive a lossy textual description produced by a separate auxiliary vision model.

Why this is a design issue

The current design is useful as a compatibility fallback for text-only models, but it is a poor default for multimodal-capable main models:

1. Lossy information transfer

   • Screenshots, UI details, diagrams, code blocks, tables, layout, colors, and small visual cues can be lost or distorted during the auxiliary description step.

2. Wrong model does the important perceptual work

   • If the user deliberately selected a strong multimodal main model, the image should ideally be interpreted by that model, not by an auxiliary model that may be weaker, cheaper, slower, or differently configured.

3. More latency and cost

   • Every image-bearing message triggers an extra vision call before the real model call.
   • Existing related issue: #10809 shows that the default vision preprocessing can produce very long descriptions and slow down image-bearing requests.

4. Behavior is surprising

   • Users expect a multimodal main model to receive the image directly.
   • Instead Hermes silently converts image input into text, which changes the semantics of the interaction.

5. Hard to evaluate multimodal models accurately

   • When benchmarking model quality in Hermes, the main model is not actually being tested on image understanding if the image has already been summarized by another model.

Proposed solution

Add a config-gated inbound image mode, for example:

vision:
  inbound_mode: preprocess          # current behavior / safest default
  # inbound_mode: direct_if_supported
  # inbound_mode: always_direct

Suggested semantics:

• preprocess

  • Current behavior.
  • Analyze image with auxiliary vision model and inject text description.
  • Best compatibility with text-only models and prompt caching.

• direct_if_supported

  • If the configured main provider/model supports multimodal image content, pass the cached local image as an OpenAI-style image content part (image_url / base64 data URL) into the main conversation.
  • If unsupported or the image is too large / incompatible, fall back to the current preprocessing path.

• always_direct

  • Always pass image content directly to the main model.
  • If the provider/model cannot handle images, return a clear error rather than silently falling back.

Implementation notes

There already appears to be lower-level multimodal support in run_agent.py for image content parts (image_url / input_image handling). The missing piece is that Telegram/CLI inbound paths normally convert images to text before the main conversation sees them.

Likely touch points:

• gateway/run.py::_prepare_inbound_message_text()
• gateway/run.py::_enrich_message_with_vision()
• cli.py::_preprocess_images_with_vision()
• possibly the message construction path before AIAgent.run_conversation()

A minimal first version could keep preprocess as the default and add direct_if_supported only for providers/models known to accept OpenAI-style image content.

Expected behavior

When using a multimodal-capable main model, sending a Telegram/CLI image should allow the main model to inspect the original image directly, preserving the current auxiliary vision preprocessing only as an explicit mode or fallback.