PR #14964: feat(api-server): serve generated images over HTTP for Open WebUI

• Repository: NousResearch/hermes-agent
• Author: sunxyless
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/14964

Description (problem / solution / changelog)

What does this PR do?

Adds HTTP delivery for images produced by the image_generate tool on the api_server platform, so Open WebUI (and any other HTTP client) can actually render generated images inline.

Today, the tool returns the absolute filesystem path of the saved PNG and the model faithfully renders ![desc](/Users/…/cache/images/foo.png) per its own schema. Platforms like Telegram and Discord turn that path into an upload; api_server does neither — the path hits the browser and 404s. Full problem statement + SSE evidence: #14959.

Two coupled, opt-in changes — the rewrite only activates when the user sets IMAGE_SERVE_BASE_URL (or image_gen.serve_base_url in config), so CLI / Telegram / Discord / feishu paths are byte-identical when unconfigured.

Related Issue

Fixes #14959

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• tools/image_generation_tool.py (+58)

  • New _resolve_image_serve_base_url() — reads IMAGE_SERVE_BASE_URL env var, then falls back to image_gen.serve_base_url in config.yaml. Returns None when unset so callers are no-ops.
  • New _maybe_rewrite_image_to_url(result_json) — post-processes the provider's JSON result. If it's a success with a local filesystem path in image (i.e. not an existing URL or data: URI), rewrites image to {base}/images/{basename}. Leaves every other field untouched.
  • _handle_image_generate now funnels both dispatch paths (plugin provider and in-tree FAL) through the post-processor.

• gateway/platforms/api_server.py (+17)

  • During _setup_app, register an aiohttp static route /images/* serving IMAGE_CACHE_DIR (already the canonical cache path in gateway/platforms/base.py).
  • Auth-free by design: <img> tags don't send Authorization headers. Security relies on the random hex suffix in save_b64_image filenames. Failure to register logs a warning and keeps the server up — not fatal.

• tests/tools/test_image_generation_url_rewrite.py (+154, new)

  • 18 tests covering both helpers: env-vs-config precedence, trailing-slash/whitespace handling, pass-through for failures, already-URL / data-URI / empty / missing image fields, malformed JSON, basename() leak-proofing.

How to Test

# 1. Enable a plugin image-gen provider (openai-codex works well — no API key needed)
hermes -p <profile> config set image_gen.provider openai-codex

# 2. Configure the public base for generated images
echo 'IMAGE_SERVE_BASE_URL=http://127.0.0.1:8642' >> ~/.hermes/profiles/<profile>/.env

# 3. Enable api_server + restart
# (.env should already have API_SERVER_ENABLED=true and API_SERVER_KEY=<key>)
hermes -p <profile> gateway restart

# 4. Trigger an image over /v1/chat/completions and watch the SSE stream
curl -sN -X POST http://127.0.0.1:8642/v1/chat/completions \
  -H "Authorization: Bearer <key>" -H 'Content-Type: application/json' \
  -d '{"model":"<profile>","stream":true,"messages":[{"role":"user","content":"Use image_generate to draw a red apple. Render inline."}]}'

Before this change the delta.content would include ![apple](/abs/path/cache/images/foo.png). With this change it includes ![apple](http://127.0.0.1:8642/images/foo.png), and a browser GET on that URL returns HTTP 200, image/png with the exact cache file bytes.

Verified end-to-end on two deployments:

• Mac + Open WebUI in Docker with IMAGE_SERVE_BASE_URL=http://127.0.0.1:8642
• VPS + Caddy reverse proxy with IMAGE_SERVE_BASE_URL=https://chat.example.com, Caddy routing /images/* through to the gateway

Tests: pytest tests/tools/test_image_generation_url_rewrite.py -q → 18 passed. Broader sweep pytest tests/tools/test_image_generation*.py tests/plugins/image_gen/ -q → 121 passed (no regressions).

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (feat(api-server): …)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this feature (no unrelated commits)
• I've run pytest tests/tools/test_image_generation*.py tests/plugins/image_gen/ -q and all 121 tests pass
• I've added tests for my changes (18 new tests)
• I've tested on my platform: macOS 15 + Ubuntu 24.04 (VPS)

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — can update website/docs/user-guide/messaging/open-webui.md with a one-line note about the env var if maintainers prefer
• I've updated cli-config.yaml.example if I added/changed config keys — N/A (feature is gated entirely on env var or config.yaml addition; the config key is optional)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — os.path.basename and aiohttp.add_static are both platform-agnostic; IMAGE_CACHE_DIR already uses get_hermes_dir
• I've updated tool descriptions/schemas if I changed tool behavior — the existing schema ("Returns either a URL or an absolute file path in the \image` field"`) already covers the URL case; no change needed

Open questions for maintainers (also listed in #14959)

1. Auth on /images/*. I went with public + random-hex filename. Happy to switch to signed URLs if you prefer — that's a self._sign_image_url(path) helper and a query-string check in the static handler.
2. Route naming. /images/ vs OpenAI-style /v1/images/{filename}? I picked the shorter path because it's not an OpenAI-spec endpoint and doesn't belong under /v1/.
3. Data-URI fallback. Some deployments can't easily expose a URL (private VPS without reverse proxy). Interested if you'd accept a complementary IMAGE_SERVE_MODE=data_uri option as a follow-up.

Happy to iterate on any of these.

Changed files

• gateway/platforms/api_server.py (modified, +17/-0)
• tests/tools/test_image_generation_url_rewrite.py (added, +154/-0)
• tools/image_generation_tool.py (modified, +54/-4)