hermes - 💡(How to fix) Fix Gemini vision provider fails — `image_url` not accepted by gemini-2.0-flash[Bug]:

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Error Message

2026-05-31 14:30:42 WARNING agent.tool_executor: Tool vision_analyze returned error (1.86s)
2026-05-31 14:31:18 WARNING agent.auxiliary_client: Auxiliary: marking gemini unhealthy for 600s (payment / credit error)
2026-05-31 14:34:12 ERROR tools.vision_tools: Error analyzing image: Error code: 400 - 
  unknown variant `image_url`, expected `text` at line 1 column 178509

Root Cause

vision_tools.py and/or auxiliary_client.py builds messages with OpenAI-compatible image_url content blocks. When the provider is gemini with gemini-2.0-flash, the message format is not converted to Gemini's native format.

The fix should be in agent/gemini_native_adapter.py or agent/auxiliary_client.py: when the vision provider is Gemini, convert image_url blocks to Gemini-native inline_data format, OR use Gemini's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai/) which does support image_url.

Also, the error classification is wrong — "payment / credit error" should be "invalid request / format error". This causes Gemini to be unnecessarily marked unhealthy for 10 minutes.

Code Example

Error code: 400 - unknown variant `image_url`, expected `text`

---

2026-05-31 14:30:42 WARNING agent.tool_executor: Tool vision_analyze returned error (1.86s)
2026-05-31 14:31:18 WARNING agent.auxiliary_client: Auxiliary: marking gemini unhealthy for 600s (payment / credit error)
2026-05-31 14:34:12 ERROR tools.vision_tools: Error analyzing image: Error code: 400 - 
  unknown variant `image_url`, expected `text` at line 1 column 178509

---

File "vision_tools.py", line 840, in vision_analyze_tool
    response = await async_call_llm(**call_kwargs)

Error code: 400 - Failed to deserialize the JSON body into the target type:
messages[0]: unknown variant `image_url`, expected `text`

---

Hermes Agent v0.15.1 (2026.5.29)
2026-05-31 14:25:58,213 INFO gateway.platforms.telegram: [Telegram] Cached user photo at C:\Hermes\image_cache\img_888c4b1230ba.jpg
2026-05-31 14:25:59,025 INFO gateway.platforms.telegram: [Telegram] Flushing photo batch agent:main:telegram:dm:1599593749:photo-burst with 1 image(s)
2026-05-31 14:25:59,029 INFO gateway.run: inbound message: platform=telegram user=Axl Gozincky chat=1599593749 msg=''
2026-05-31 14:25:59,597 INFO gateway.run: Image routing: text (mode=text). Pre-analyzing 1 image(s) via vision_analyze.
2026-05-31 14:26:22,908 INFO gateway.run: response ready: platform=telegram chat=1599593749 time=23.9s api_calls=2 response=559 chars
2026-05-31 14:26:22,932 INFO gateway.platforms.base: [Telegram] Sending response (559 chars) to 1599593749
2026-05-31 14:27:28,285 INFO gateway.platforms.telegram: [Telegram] Flushing text batch agent:main:telegram:dm:1599593749 (74 chars)
2026-05-31 14:27:28,291 INFO gateway.run: inbound message: platform=telegram user=Axl Gozincky chat=1599593749 msg='Puedes configurar solo el géminis o preciso hacer algo con la computadora?'
2026-05-31 14:27:40,534 INFO gateway.run: response ready: platform=telegram chat=1599593749 time=12.2s api_calls=2 response=467 chars
2026-05-31 14:27:40,560 INFO gateway.platforms.base: [Telegram] Sending response (467 chars) to 1599593749
2026-05-31 14:28:04,129 INFO gateway.memory_monitor: [MEMORY] rss=85MB gc=(94, 0, 2) threads=9 uptime=7500s
2026-05-31 14:28:15,535 INFO gateway.platforms.telegram: [Telegram] Flushing text batch agent:main:telegram:dm:1599593749 (15 chars)
2026-05-31 14:28:15,540 INFO gateway.run: inbound message: platform=telegram user=Axl Gozincky chat=1599593749 msg='Ya te di la api'
2026-05-31 14:29:34,040 INFO gateway.run: response ready: platform=telegram chat=1599593749 time=78.5s api_calls=6 response=619 chars
2026-05-31 14:29:34,056 INFO gateway.platforms.base: [Telegram] Sending response (619 chars) to 1599593749
2026-05-31 14:29:55,250 INFO gateway.platforms.telegram: [Telegram] Flushing text batch agent:main:telegram:dm:1599593749 (11 chars)
2026-05-31 14:29:55,254 INFO gateway.run: inbound message: platform=telegram user=Axl Gozincky chat=1599593749 msg='Ok reinicia'
2026-05-31 14:30:04,551 INFO gateway.platforms.telegram: Telegram button resolved 1 approval(s) for session agent:main:telegram:dm:1599593749 (choice=once, user=Cristian)
2026-05-31 14:30:07,284 INFO gateway.run: Received UNKNOWN as a planned gateway stop — exiting cleanly
2026-05-31 14:30:07,286 WARNING gateway.run: Shutdown context: signal=UNKNOWN under_systemd=no parent_pid=31804 parent_name=? loadavg_1m=? parent_cmdline='(unknown)'
2026-05-31 14:30:07,289 INFO gateway.run: Stopping gateway...
2026-05-31 14:30:07,925 INFO gateway.run: Sent shutdown notification to active chat telegram:1599593749
2026-05-31 14:30:07,928 INFO gateway.run: Shutdown phase: notify_active_sessions done at +0.62s
2026-05-31 14:31:04,788 INFO gateway.platforms.telegram: Telegram button resolved 1 approval(s) for session agent:main:telegram:dm:1599593749 (choice=once, user=Cristian)
2026-05-31 14:32:36,923 INFO gateway.platforms.telegram: Telegram button resolved 1 approval(s) for session agent:main:telegram:dm:1599593749 (choice=once, user=Cristian)
2026-05-31 14:32:56,333 INFO gateway.run: response ready: platform=telegram chat=1599593749 time=181.1s api_calls=9 response=939 chars
2026-05-31 14:32:56,352 INFO gateway.platforms.base: [Telegram] Sending response (939 chars) to 1599593749
2026-05-31 14:32:56,361 INFO gateway.run: Shutdown phase: drain done at +169.06s (drain took 168.42s, timed_out=False, active_at_start=1, active_now=0)
2026-05-31 14:32:59,906 INFO gateway.platforms.telegram: [Telegram] Disconnected from Telegram
2026-05-31 14:32:59,907 INFO gateway.run: ✓ telegram disconnected (3.55s)
2026-05-31 14:32:59,909 INFO gateway.run: Shutdown phase: all adapters disconnected at +172.61s
2026-05-31 14:32:59,913 INFO gateway.run: Shutdown phase: final-cleanup tool kill done at +172.61s
2026-05-31 14:32:59,925 INFO gateway.run: Shutdown phase: SessionDB close done at +172.62s
2026-05-31 14:32:59,936 INFO gateway.run: Gateway stopped (total teardown 172.64s)
2026-05-31 14:32:59,937 INFO gateway.run: Cron ticker stopped
2026-05-31 14:32:59,941 INFO gateway.memory_monitor: [MEMORY] shutdown rss=83MB gc=(20, 4, 5) threads=14 uptime=7796s
2026-05-31 14:32:59,942 INFO gateway.memory_monitor: [MEMORY] Periodic memory monitoring stopped
2026-05-31 14:33:11,062 INFO gateway.memory_monitor: [MEMORY] baseline rss=102MB gc=(496, 9, 7) threads=1 uptime=0s
2026-05-31 14:33:11,065 INFO gateway.memory_monitor: [MEMORY] Periodic memory monitoring started (interval: 300s)
2026-05-31 14:33:11,421 INFO gateway.run: Starting Hermes Gateway...
2026-05-31 14:33:11,424 INFO gateway.run: Session storage: C:\Hermes\sessions
2026-05-31 14:33:11,426 INFO gateway.run: Agent budget: max_iterations=90 (agent.max_turns from config.yaml, or HERMES_MAX_ITERATIONS from .env, or default 90)
2026-05-31 14:33:11,427 INFO gateway.run: Secret redaction: ENABLED (tool output, logs, and chat responses are scrubbed before delivery)
2026-05-31 14:33:11,450 INFO gateway.run: Previous gateway exited cleanly — skipping session suspension
2026-05-31 14:33:11,958 INFO gateway.run: Connecting to telegram...
2026-05-31 14:33:12,300 INFO gateway.platforms.telegram: [Telegram] Auto-discovered Telegram fallback IPs: 149.154.166.110
2026-05-31 14:33:12,302 INFO gateway.platforms.telegram: [Telegram] Telegram fallback IPs active: 149.154.166.110
2026-05-31 14:33:14,021 INFO gateway.platforms.telegram: [Telegram] set_my_commands OK for scope BotCommandScopeDefault (30 cmds)
2026-05-31 14:33:14,333 INFO gateway.platforms.telegram: [Telegram] set_my_commands OK for scope BotCommandScopeAllPrivateChats (30 cmds)
2026-05-31 14:33:14,640 INFO gateway.platforms.telegram: [Telegram] set_my_commands OK for scope BotCommandScopeAllGroupChats (30 cmds)
2026-05-31 14:33:14,642 INFO gateway.platforms.telegram: [Telegram] Telegram menu: 30 commands registered, 15 hidden (over 30 limit). Use /commands for full list.
2026-05-31 14:33:14,671 INFO gateway.platforms.telegram: [Telegram] Connected to Telegram (polling mode)
2026-05-31 14:33:14,685 INFO gateway.run: ✓ telegram connected

---
RAW_BUFFERClick to expand / collapse

Bug Description

[Bug]: Gemini vision provider fails — image_url not accepted by gemini-2.0-flash

Bug Description

Configuring auxiliary.vision.provider = gemini with model = gemini-2.0-flash causes all vision/image analysis to fail. Images sent via Telegram are cached correctly by the gateway and the vision_analyze tool is invoked, but the API call fails with:

Error code: 400 - unknown variant `image_url`, expected `text`

The root cause: Hermes sends vision requests in OpenAI-compatible format (image_url message content type), but the Gemini API endpoint being used doesn't accept this variant. Gemini's native API expects images via inline_data or file_data, not image_url.

After the first failure, Gemini is marked unhealthy for 600s with misleading error: "payment / credit error" (should be "format/API error").

Affected Component

  • Gateway (Telegram)
  • Agent Core (auxiliary_client, vision tools)

Root Cause Analysis

vision_tools.py and/or auxiliary_client.py builds messages with OpenAI-compatible image_url content blocks. When the provider is gemini with gemini-2.0-flash, the message format is not converted to Gemini's native format.

The fix should be in agent/gemini_native_adapter.py or agent/auxiliary_client.py: when the vision provider is Gemini, convert image_url blocks to Gemini-native inline_data format, OR use Gemini's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai/) which does support image_url.

Also, the error classification is wrong — "payment / credit error" should be "invalid request / format error". This causes Gemini to be unnecessarily marked unhealthy for 10 minutes.

Logs

2026-05-31 14:30:42 WARNING agent.tool_executor: Tool vision_analyze returned error (1.86s)
2026-05-31 14:31:18 WARNING agent.auxiliary_client: Auxiliary: marking gemini unhealthy for 600s (payment / credit error)
2026-05-31 14:34:12 ERROR tools.vision_tools: Error analyzing image: Error code: 400 - 
  unknown variant `image_url`, expected `text` at line 1 column 178509

Steps to Reproduce

  1. Set auxiliary.vision.provider = gemini and auxiliary.vision.model = gemini-2.0-flash
  2. Set GEMINI_API_KEY in .env
  3. Restart gateway
  4. Send an image via Telegram
  5. Agent receives image, calls vision_analyze
  6. Error: unknown variant 'image_url', expected 'text'

Expected Behavior

Gemini should process the image and return a description/analysis.

Actual Behavior

File "vision_tools.py", line 840, in vision_analyze_tool
    response = await async_call_llm(**call_kwargs)

Error code: 400 - Failed to deserialize the JSON body into the target type:
messages[0]: unknown variant `image_url`, expected `text`

Then Gemini is marked unhealthy for 600s.

Affected Component

Gateway (Telegram/Discord/Slack/WhatsApp), Agent Core (conversation loop, context compression, memory)

Messaging Platform (if gateway-related)

Telegram

Debug Report

Hermes Agent v0.15.1 (2026.5.29)
2026-05-31 14:25:58,213 INFO gateway.platforms.telegram: [Telegram] Cached user photo at C:\Hermes\image_cache\img_888c4b1230ba.jpg
2026-05-31 14:25:59,025 INFO gateway.platforms.telegram: [Telegram] Flushing photo batch agent:main:telegram:dm:1599593749:photo-burst with 1 image(s)
2026-05-31 14:25:59,029 INFO gateway.run: inbound message: platform=telegram user=Axl Gozincky chat=1599593749 msg=''
2026-05-31 14:25:59,597 INFO gateway.run: Image routing: text (mode=text). Pre-analyzing 1 image(s) via vision_analyze.
2026-05-31 14:26:22,908 INFO gateway.run: response ready: platform=telegram chat=1599593749 time=23.9s api_calls=2 response=559 chars
2026-05-31 14:26:22,932 INFO gateway.platforms.base: [Telegram] Sending response (559 chars) to 1599593749
2026-05-31 14:27:28,285 INFO gateway.platforms.telegram: [Telegram] Flushing text batch agent:main:telegram:dm:1599593749 (74 chars)
2026-05-31 14:27:28,291 INFO gateway.run: inbound message: platform=telegram user=Axl Gozincky chat=1599593749 msg='Puedes configurar solo el géminis o preciso hacer algo con la computadora?'
2026-05-31 14:27:40,534 INFO gateway.run: response ready: platform=telegram chat=1599593749 time=12.2s api_calls=2 response=467 chars
2026-05-31 14:27:40,560 INFO gateway.platforms.base: [Telegram] Sending response (467 chars) to 1599593749
2026-05-31 14:28:04,129 INFO gateway.memory_monitor: [MEMORY] rss=85MB gc=(94, 0, 2) threads=9 uptime=7500s
2026-05-31 14:28:15,535 INFO gateway.platforms.telegram: [Telegram] Flushing text batch agent:main:telegram:dm:1599593749 (15 chars)
2026-05-31 14:28:15,540 INFO gateway.run: inbound message: platform=telegram user=Axl Gozincky chat=1599593749 msg='Ya te di la api'
2026-05-31 14:29:34,040 INFO gateway.run: response ready: platform=telegram chat=1599593749 time=78.5s api_calls=6 response=619 chars
2026-05-31 14:29:34,056 INFO gateway.platforms.base: [Telegram] Sending response (619 chars) to 1599593749
2026-05-31 14:29:55,250 INFO gateway.platforms.telegram: [Telegram] Flushing text batch agent:main:telegram:dm:1599593749 (11 chars)
2026-05-31 14:29:55,254 INFO gateway.run: inbound message: platform=telegram user=Axl Gozincky chat=1599593749 msg='Ok reinicia'
2026-05-31 14:30:04,551 INFO gateway.platforms.telegram: Telegram button resolved 1 approval(s) for session agent:main:telegram:dm:1599593749 (choice=once, user=Cristian)
2026-05-31 14:30:07,284 INFO gateway.run: Received UNKNOWN as a planned gateway stop — exiting cleanly
2026-05-31 14:30:07,286 WARNING gateway.run: Shutdown context: signal=UNKNOWN under_systemd=no parent_pid=31804 parent_name=? loadavg_1m=? parent_cmdline='(unknown)'
2026-05-31 14:30:07,289 INFO gateway.run: Stopping gateway...
2026-05-31 14:30:07,925 INFO gateway.run: Sent shutdown notification to active chat telegram:1599593749
2026-05-31 14:30:07,928 INFO gateway.run: Shutdown phase: notify_active_sessions done at +0.62s
2026-05-31 14:31:04,788 INFO gateway.platforms.telegram: Telegram button resolved 1 approval(s) for session agent:main:telegram:dm:1599593749 (choice=once, user=Cristian)
2026-05-31 14:32:36,923 INFO gateway.platforms.telegram: Telegram button resolved 1 approval(s) for session agent:main:telegram:dm:1599593749 (choice=once, user=Cristian)
2026-05-31 14:32:56,333 INFO gateway.run: response ready: platform=telegram chat=1599593749 time=181.1s api_calls=9 response=939 chars
2026-05-31 14:32:56,352 INFO gateway.platforms.base: [Telegram] Sending response (939 chars) to 1599593749
2026-05-31 14:32:56,361 INFO gateway.run: Shutdown phase: drain done at +169.06s (drain took 168.42s, timed_out=False, active_at_start=1, active_now=0)
2026-05-31 14:32:59,906 INFO gateway.platforms.telegram: [Telegram] Disconnected from Telegram
2026-05-31 14:32:59,907 INFO gateway.run: ✓ telegram disconnected (3.55s)
2026-05-31 14:32:59,909 INFO gateway.run: Shutdown phase: all adapters disconnected at +172.61s
2026-05-31 14:32:59,913 INFO gateway.run: Shutdown phase: final-cleanup tool kill done at +172.61s
2026-05-31 14:32:59,925 INFO gateway.run: Shutdown phase: SessionDB close done at +172.62s
2026-05-31 14:32:59,936 INFO gateway.run: Gateway stopped (total teardown 172.64s)
2026-05-31 14:32:59,937 INFO gateway.run: Cron ticker stopped
2026-05-31 14:32:59,941 INFO gateway.memory_monitor: [MEMORY] shutdown rss=83MB gc=(20, 4, 5) threads=14 uptime=7796s
2026-05-31 14:32:59,942 INFO gateway.memory_monitor: [MEMORY] Periodic memory monitoring stopped
2026-05-31 14:33:11,062 INFO gateway.memory_monitor: [MEMORY] baseline rss=102MB gc=(496, 9, 7) threads=1 uptime=0s
2026-05-31 14:33:11,065 INFO gateway.memory_monitor: [MEMORY] Periodic memory monitoring started (interval: 300s)
2026-05-31 14:33:11,421 INFO gateway.run: Starting Hermes Gateway...
2026-05-31 14:33:11,424 INFO gateway.run: Session storage: C:\Hermes\sessions
2026-05-31 14:33:11,426 INFO gateway.run: Agent budget: max_iterations=90 (agent.max_turns from config.yaml, or HERMES_MAX_ITERATIONS from .env, or default 90)
2026-05-31 14:33:11,427 INFO gateway.run: Secret redaction: ENABLED (tool output, logs, and chat responses are scrubbed before delivery)
2026-05-31 14:33:11,450 INFO gateway.run: Previous gateway exited cleanly — skipping session suspension
2026-05-31 14:33:11,958 INFO gateway.run: Connecting to telegram...
2026-05-31 14:33:12,300 INFO gateway.platforms.telegram: [Telegram] Auto-discovered Telegram fallback IPs: 149.154.166.110
2026-05-31 14:33:12,302 INFO gateway.platforms.telegram: [Telegram] Telegram fallback IPs active: 149.154.166.110
2026-05-31 14:33:14,021 INFO gateway.platforms.telegram: [Telegram] set_my_commands OK for scope BotCommandScopeDefault (30 cmds)
2026-05-31 14:33:14,333 INFO gateway.platforms.telegram: [Telegram] set_my_commands OK for scope BotCommandScopeAllPrivateChats (30 cmds)
2026-05-31 14:33:14,640 INFO gateway.platforms.telegram: [Telegram] set_my_commands OK for scope BotCommandScopeAllGroupChats (30 cmds)
2026-05-31 14:33:14,642 INFO gateway.platforms.telegram: [Telegram] Telegram menu: 30 commands registered, 15 hidden (over 30 limit). Use /commands for full list.
2026-05-31 14:33:14,671 INFO gateway.platforms.telegram: [Telegram] Connected to Telegram (polling mode)
2026-05-31 14:33:14,685 INFO gateway.run: ✓ telegram connected

Operating System

Windows 11

Python Version

3.11.15

Hermes Version

Hermes Agent v0.15.1 (2026.5.29)

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

vision_tools.py and/or auxiliary_client.py builds messages with OpenAI-compatible image_url content blocks. When the provider is gemini with gemini-2.0-flash, the message format is not converted to Gemini's native format.

The fix should be in agent/gemini_native_adapter.py or agent/auxiliary_client.py: when the vision provider is Gemini, convert image_url blocks to Gemini-native inline_data format, OR use Gemini's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai/) which does support image_url.

Also, the error classification is wrong — "payment / credit error" should be "invalid request / format error". This causes Gemini to be unnecessarily marked unhealthy for 10 minutes.

Proposed Fix (optional)

No response

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

hermes - 💡(How to fix) Fix Gemini vision provider fails — `image_url` not accepted by gemini-2.0-flash[Bug]: