Native multimodal mode: expose image path in user text for tool-parameter use

Problem

When a vision-capable main model (e.g. gpt-4o, gpt-5.5, claude-sonnet-4) receives an image via the gateway in image_input_mode: native (image bytes inlined as image_url content part), the local file path is never exposed as plain text in the conversation context.

This means the model can see the image (full native vision) but cannot reference the file as a string parameter for tools that expect a path or URL — for example, a custom MCP tool that takes an image_url: str argument and attaches the image to a downstream tracker / OCR endpoint / issue tracker.

The text-mode path (_enrich_message_with_vision) already injects this hint ("...use vision_analyze with image_url: /local/path..."), but the native-mode path that replaced it for vision-capable models in v0.12.0 has no equivalent.

Reproduction

1. Configure Hermes with a vision-capable main model (e.g. gpt-5.5 via openai-codex, or claude-sonnet-4).
2. Leave agent.image_input_mode unset (default auto → resolves to native for vision-capable models).
3. Register a custom MCP tool that expects an image URL/path:

   @mcp.tool()
   def task_attach_image(task_id: int, image_url: str) -> dict: ...

4. From Slack, send an image with caption: "Bunu task #123'e ekle".
5. Model sees the image (correct OCR/description on demand) but responds:

   "I see the image, but the file URL is not in my context, so I cannot pass it to task_attach_image."

Verified live in Pi 5 production deployment with Slack gateway serving a non-English operations team — v0.12.0 (commit 73bf3ab1b), gpt-5.5 via openai-codex, 2026-05-02. The use case is real day-to-day workflow (operations manager attaching screenshots/photos to internal tracker tasks via Slack), not a synthetic local-dev reproduction.

Code Pointers

• agent/image_routing.py::build_native_content_parts — only constructs {"type": "image_url", ...} parts; no companion text injection of the source path. The function's input user_text is the original message; the path string never enters the parts list.
• gateway/run.py::_enrich_message_with_vision — TEXT mode already injects something equivalent for the auxiliary path:

  [... use vision_analyze using image_url: /local/path ...]

  Native mode skips this branch entirely (see _decide_image_input_mode at gateway/run.py:9407 and the if _img_mode == "native" short-circuit at gateway/run.py:4691).
• gateway/run.py near line 11604 — build_native_content_parts is invoked, but the user-text part it produces does not include any path reference.

Expected Behaviour

In native mode, after the image_url content part is constructed, optionally append a brief text note like [Image attached at: /local/path] to the user text part of the same multimodal turn. This parallels the text-mode injection but for native mode where vision_analyze is not called.

The model then has both:

1. The pixels (native vision quality preserved).
2. A string reference to use as a tool argument when needed.

Workaround (Plugin-Based)

We have implemented a Hermes plugin (ocp-image-path-inject) that hooks into pre_llm_call and prepends [Image attached at: <path>] to the user message when native image parts are present. The workaround is functional but:

• Requires per-installation setup (we run Slack gateway on Raspberry Pi 5 in production).
• Touches a hot path that is already under careful prompt-cache discipline (per AGENTS.md "Prompt Caching Must Not Break"); a first-party config flag would let upstream guarantee correctness for everyone.
• Adds friction for any user with a custom image-handling tool, which is a growing class as MCP adoption increases.

The plugin source is shareable on request — comment if interested, I'll publish a standalone repo (hermes-plugin-image-path-inject). It contains only the Hermes plugin pattern + cache path whitelist, no application-internal code.

Proposed Solution

Add an opt-in config flag:

agent:
  native_vision_path_hint: true  # default: false

When true, native mode also injects [Image attached at: <path>] (one line per attached image) into the text part of the multimodal user turn.

Default false to preserve current prompt-cache behaviour — the path string changes per upload, and would invalidate cached prefixes for repeated images. Users who need tool-parameter access opt in and accept the trade-off.

Why This Matters

Custom MCP tools that operate on images (issue trackers, OCR-to-endpoint, image-to-DB) cannot work cleanly in native mode without this path exposure. Today the alternatives are:

• Force image_input_mode: text: doubles the LLM call (auxiliary vision + main) and loses the native vision quality the user paid for.
• Custom plugin per installation (works, not portable).

A first-party opt-in flag unifies the experience and lives under the same agent.* namespace as the existing image_input_mode.

Related

• #15288 (open, P2, comp/gateway) — Adds inbound mode (preprocess vs direct). Adjacent but orthogonal: that issue is about whether to send pixels natively; this one is about what extra metadata to expose alongside the pixels in native mode. Both can ship independently.
• #13065 / #7641 (open) — Native vision support feature requests; v0.12.0 / PR #16506 partially addresses by making native routing automatic for vision-capable models. Tool-parameter path exposure is not covered there.
• #5661 (closed, not planned) — Earlier passthrough proposal; superseded.

Environment

• Hermes v0.12.0 (commit 73bf3ab1b22314ed9dfecbb59242c03742fe72af)
• Slack gateway on Raspberry Pi 5, systemd --user service
• Main model: gpt-5.5 via openai-codex provider
• Custom MCP server with image-handling tools registered via ~/.hermes/config.yaml → mcp block

Default-Value Rationale

Default false is deliberate:

• Preserves prompt-cache behaviour for the majority of users who only need vision (not tool-parameter access). Image paths are per-upload unique strings and would invalidate cache hits.
• Opt-in users explicitly accept the trade-off in exchange for tool-parameter functionality.
• Matches the existing pattern of conservative defaults in agent.* config keys (e.g. image_input_mode: auto, busy_ack_enabled opt-in).

A follow-up PR is being prepared with the implementation, tests, and docs updates for review.