hermes - 💡(How to fix) Fix Pass native images to multimodal models instead of always going through vision enrichment pipeline [2 comments, 3 participants]

Problem When a user sends an image through any gateway platform (Telegram, Feishu, WeChat, Discord, etc.), the gateway always runs it through _enrich_message_with_vision() — a separate vision analysis tool call — and replaces the original image with a text description before passing it to the LLM:

# gateway/run.py: _enrich_message_with_vision() line 8237-8280

result_json = await vision_analyze_tool(image_url=path, user_prompt=analysis_prompt)

description = result.get("analysis", "")

# Result: "[The user sent an image~ Here's what I can see:\n{description}]"

The agent receives only text like:

[The user sent an image~ Here's what I can see:
This image shows a screenshot of a dashboard with...
]

This means: Multimodal models never actually see the original image — they only get a text description from a separate (often weaker) vision model. Information loss — details the multimodal model could pick up directly (fine print, layout, color, specific icons) are lost or distorted in the vision-to-text roundtrip. Extra latency & cost — every user image requires an extra API call to the vision analysis tool before the main LLM even begins reasoning. Expected Behavior For multimodal models (GPT-4o, Claude 3.5 Sonnet/Opus, Gemini 2.0, GLM-4V, Qwen-VL, etc.), the cached local image should be passed directly to the LLM as a native image_url content part in the OpenAI-style messages array:

# What should happen for multimodal models:

{

    "role": "user",

    "content": [

        {"type": "text", "text": user_text or ""},

        {"type": "image_url", "image_url": {"url": f"file://{cached_path}"}}

    ]

}

Only fall back to the current _enrich_message_with_vision() text-description approach for text-only models. Proposed Implementation The key change is in gateway/run.py inside _handle_message_with_agent(), around line 3910-3924: Detect whether the current model supports multimodal (vision) input — possibly from agent/model_metadata.py If yes → build a content array with type: image_url pointing to the cached local file path, instead of enriching the message string If no → keep existing behavior (use _enrich_message_with_vision() as fallback) AIAgent.run_conversation() currently takes user_message: str (line 8630). It would also need to accept structured messages (content arrays with image_url entries), or a parallel interface like conversation_history-style message dicts. Related #15576: Feishu/WeChat images fail with "Invalid image source" — a different bug in the same enrichment pipeline (empty paths not filtered)