hermes - 💡(How to fix) Fix [Bug] Ollama Cloud vision models return 500 on /v1/chat/completions — need native /api/chat adapter [1 comments, 2 participants]

Bug Description

When using Ollama Cloud as the vision provider (auxiliary.vision.provider: ollama-cloud), vision-capable models (e.g. qwen3-vl:235b) return 500 Internal Server Error when images are included in the request payload. Text-only requests to the same models work fine.

Root Cause

Hermes sends all API calls through the OpenAI-compatible endpoint (/v1/chat/completions) using the OpenAI Python SDK. The DEFAULT_OLLAMA_CLOUD_BASE_URL is set to https://ollama.com/v1 (in hermes_cli/auth.py:74).

For vision, vision_tools.py constructs messages in OpenAI format:

"content": [
    {"type": "text", "text": "..."},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
]

However, Ollama Cloud's /v1/chat/completions endpoint has inconsistent vision support. The native Ollama API (/api/chat) works reliably with all vision models using a simpler format:

{"role": "user", "content": "Describe this image.", "images": ["<base64>"]}

This is a known Ollama upstream issue:

• ollama/ollama#13462 — Feature request to enable image input for all vision-capable cloud models (closed Dec 2025)
• ollama/ollama#13464 — Cloud gemma3 models fail on /v1/chat/completions with images but work on /api/chat (fixed Jan 2026 for gemma3 only)
• ollama/ollama#13468 — Library incorrectly showed "Text" only for vision-capable cloud models (fixed Mar 2026)

Other models (e.g. qwen3-vl:235b) may still return 500 on the OpenAI endpoint despite being reported as working.

Steps to Reproduce

1. Configure auxiliary.vision.provider: ollama-cloud and auxiliary.vision.model: qwen3-vl:235b in config.yaml
2. Have a valid OLLAMA_API_KEY set
3. Call vision_analyze with any image
4. Observe 500 Internal Server Error from Ollama Cloud

Expected Behavior

Vision analysis should work with Ollama Cloud vision models, either by:

1. Using the native /api/chat endpoint with the images array format for Ollama Cloud vision requests
2. Or correctly formatting the OpenAI-compatible request so Ollama Cloud accepts it

Proposed Solution

Add an Ollama native API adapter for vision requests in auxiliary_client.py:

1. In call_llm/async_call_llm, detect when resolved_provider == "ollama-cloud" and task == "vision"
2. Instead of calling client.chat.completions.create() (which hits /v1/chat/completions), make a direct HTTP POST to https://ollama.com/api/chat with the native payload:

   {
     "model": "qwen3-vl:235b",
     "messages": [{"role": "user", "content": "Describe this image.", "images": ["<base64>"]}],
     "stream": false
   }

3. Convert the OpenAI-format image_url content blocks to the native images array
4. Parse the native response back to a format extract_content_or_reasoning() can handle

Key files to modify:

• agent/auxiliary_client.py — resolve_vision_provider_client(), call_llm(), async_call_llm()
• tools/vision_tools.py — message construction (lines 532-548)

Environment

• Hermes Agent: latest (v0.8+)
• Provider: Ollama Cloud
• Models tested: qwen3-vl:235b, qwen3.5:397b
• Result: 500 Internal Server Error on image payloads, text-only works fine

Related

• ollama/ollama#13462, ollama/ollama#13464, ollama/ollama#13468