hermes - 💡(How to fix) Fix Feature: First-class native vision support for vision-capable main models (with reference implementation + bug findings)

Feature Request: First-Class Native Vision Support for Vision-Capable Main Models

Summary

Hermes currently routes ALL image analysis through an auxiliary vision model (qwen3-vl or configured alternative), even when the main model is natively vision-capable (e.g., gpt-4o, glm-5v-turbo, claude-sonnet-4). This adds unnecessary latency, cost, and information loss (text description ≠ seeing the image).

We implemented a working native vision bypass (native_vision: true config toggle) that sends images directly to the main model as multimodal content blocks — and in doing so, we discovered several underlying issues in Hermes's conversation pipeline that would affect ANY multimodal content, not just our hack.

This issue serves dual purpose:

1. Feature request: Upstream native vision support (with our implementation as reference)
2. Bug findings: Pipeline issues that should be fixed regardless

Our Reference Implementation

We patched 4 files to add native vision support:

Full patch code: see native-vision-patch skill (community-maintained)

Bugs We Discovered (Affect All Multimodal Content)

These are issues in Hermes's core pipeline that would impact any attempt to send image data through the conversation system:

1. Missing detail: "auto" on Image Injection (Token Bloat)

When constructing OpenAI-format image_url content blocks, there's no detail parameter. This means images are tokenized at full resolution (detail: "high" implicit default).

Impact: A typical phone photo = ~121-230K tokens at full res vs ~2-5K with detail: "auto". That's a 25-60× reduction.

Fix location (3 injection points):

• run_agent.py — tool result injection (~L8465)
• run_agent.py — user upload injection (~L8852)
• tools/browser_tool.py — auxiliary vision fallback (~L2155)

# Before:
{"type": "image_url", "image_url": {"url": data_url}}

# After:
{"type": "image_url", "image_url": {"url": data_url, "detail": "auto"}}

Reference: OpenAI Vision API docs

2. Context Compressor Explicitly Skips Multimodal Content

agent/context_compressor.py — all 3 compression passes have this guard:

if isinstance(content, list):
    continue  # ← skips EVERY multimodal block including images

Impact: Image data is immortal. After N turns with M images = unbounded context growth. The compressor reports "failed after 3 attempts" when 60%+ of context is uncompressible image data.

Fix: Add a Pass 4 that replaces old image blocks (outside protected tail) with lightweight text placeholders. Follows same pattern as OpenClaw (images are per-turn only).

3. Session Persistence Bakes Raw Content Verbatim

Large string content (including base64 image data) is stored directly into session JSON files without size awareness or sanitization. On session reload, raw data is sent back to the API.

Impact: A single 1.97 MB base64 blob in a user message causes context overflow on every reload until manually cleaned up.

Fix: Override persist to store text-only references for image content; strip base64 from messages before session save.

4. Gateway Preambles Break JSON Message Parsing

The gateway prepends notices (e.g., model-switch notifications) before message content. If that content is JSON (like our vision markers), json.loads() fails on the combined string.

Fix: Find JSON start position before parsing; preserve preamble text separately.

5. Short Sessions Bypass Compression Entirely

When compress() sees fewer than _min_for_compress messages, it returns immediately. For fresh sessions replaying a poisoned upload (e.g., after auto-reset), NO compression runs — the base64 survives every retry until crash.

Fix: Run base64 stripping unconditionally at the start of compress(), before the early-return guard.

Token Math: Why Native Vision Matters

Proposed Approach

1. Add auxiliary.native_vision config toggle (upstream our hack)
2. Apply detail: "auto" to all image_url injection points
3. Add context compressor Pass 4 for image eviction
4. Fix session persistence to not bake base64
5. Fix preamble-aware JSON parsing
6. Add pre-compress base64 strip for short sessions
7. Graceful fallback: if native vision fails → fall through to auxiliary path

Workaround (Current)

Currently users must use auxiliary vision. Set auxiliary.vision to a vision model config. No native option exists upstream.

References

• OpenAI Vision API — Low or High Fidelity
• OpenClaw implementation — reference architecture for per-turn-only images with detail:"auto"
• Our full implementation + troubleshooting guide: native-vision-patch skill

Note: Posted by a Hermes user who built and battle-tested a native vision implementation. Happy to contribute as a PR if maintainers are interested.