hermes - ✅(Solved) Fix vision_analyze tool: cannot analyze local image files in CLI despite claiming to support local paths [1 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
NousResearch/hermes-agent#14955Fetched 2026-04-24 10:44:01
View on GitHub
Comments
1
Participants
2
Timeline
4
Reactions
0
Timeline (top)
labeled ×3commented ×1

Error Message

  1. The tool returns an error or silently fails.
  • Treats the path as an invalid URL and raises a validation error

Root Cause

Actual behavior: Local file paths are not handled correctly. The tool either: - Treats the path as an invalid URL and raises a validation error - Fails because the tool runs in an environment that cannot access the user's local filesystem - Returns empty/incomplete results

Fix Action

Fixed

PR fix notes

PR #15092: fix: clarify vision_analyze image_url parameter accepts file:// URIs …

Description (problem / solution / changelog)

…and absolute paths

What does this PR do?

<!-- Describe the change clearly. What problem does it solve? Why is this approach the right one? -->

Related Issue

<!-- Link the issue this PR addresses. If no issue exists, consider creating one first. -->

Fixes #

Type of Change

<!-- Check the one that applies. -->
  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

<!-- List the specific changes. Include file paths for code changes. -->

How to Test

<!-- Steps to verify this change works. For bugs: reproduction steps + proof that the fix works. -->

Checklist

<!-- Complete these before requesting review. -->

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: <!-- e.g. Ubuntu 24.04, macOS 15.2, Windows 11 -->

Documentation & Housekeeping

<!-- Check all that apply. It's OK to check "N/A" if a category doesn't apply to your change. -->
  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

<!-- Only fill this out if you're adding a skill. Delete this section otherwise. -->
  • This skill is broadly useful to most users (if bundled) — see Contributing Guide
  • SKILL.md follows the standard format (frontmatter, trigger conditions, steps, pitfalls)
  • No external dependencies that aren't already available (prefer stdlib, curl, existing Hermes tools)
  • I've tested the skill end-to-end: hermes --toolsets skills -q "Use the X skill to do Y"

Screenshots / Logs

<!-- If applicable, add screenshots or log output showing the fix/feature in action. -->

Fix

Clarify the image_url parameter description for vision_analyze tool to accurately document supported formats: http://, https://, file:// URIs, and absolute local paths.

Closes #14955

Changed files

  • tools/vision_tools.py (modified, +1/-1)
RAW_BUFFERClick to expand / collapse

The vision_analyze tool description states it accepts "Image URL (http/https) or local file path to analyze," but in practice local file paths do not work reliably in the CLI context.

Steps to reproduce:
1. In a CLI session, call vision_analyze with a local file path:

   vision_analyze(image_url="/Users/username/screenshot.png", question="What is shown?")

2. The tool returns an error or silently fails.

Expected behavior:
The tool should be able to analyze locally stored image files that the user can access on their machine.

Actual behavior:
Local file paths are not handled correctly. The tool either:
- Treats the path as an invalid URL and raises a validation error
- Fails because the tool runs in an environment that cannot access the user's local filesystem
- Returns empty/incomplete results

Root Cause Analysis

Looking at tools/vision_tools.py, the implementation has two issues:

1. Schema/description mismatch: The tool parameter description mentions "local file path" support, but the internal _validate_image_url() only accepts http:// and https:// URLs. Local paths only work if prefixed with file://, and even then only when the tool process has access to the filesystem at that path.

2. No file upload mechanism: The vision tool works by having the tool itself download or read the image. This requires the tool's runtime environment to have access to the file. When Hermes runs in a CLI session on a user's machine, this may fail if the tool process is sandboxed or running in a different context than the terminal.

The correct architecture would be for the Agent to first receive the file from the user (via an upload mechanism), store it in a location the vision service can access, then pass that URL to the vision tool.

Proposed Solution

Option A — Quick fix (documentation):
Update the tool description to be explicit: only http:///https:// URLs and file:// URIs are supported. Add examples showing the exact format required.

Option B — Proper fix (recommended):
Implement a file upload mechanism for the vision toolchain:
1. Add a tool (e.g., upload_image or attach_file) that accepts a local path, copies the file to a temp location accessible by the auxiliary vision service, and returns a URL
2. Have vision_analyze accept only accessible HTTP URLs (removing the confusing local path claim from its description)
3. The Agent would then use upload_image first, then pass the resulting URL to vision_analyze

This is the only approach that works when the Agent runs in remote, sandboxed, or containerized environments.

Environment

- Hermes Agent version: (latest)
- OS: macOS (likely affects all desktop users)
- CLI context
- Provider: (any — vision is a separate auxiliary service)

Screenshots / Additional Context

Tool description in schema:
> "Image URL (http/https) or local file path to analyze."

This claim is not reliably fulfilled in practice for CLI users with local files.

extent analysis

TL;DR

Update the vision_analyze tool to either only accept HTTP/HTTPS URLs or implement a file upload mechanism to handle local file paths.

Guidance

  • Verify that the issue is due to the tool's inability to access local file paths by checking the tool's runtime environment and permissions.
  • Consider implementing a file upload mechanism, such as the proposed upload_image tool, to handle local file paths.
  • Update the tool description to reflect the supported input formats, such as only HTTP/HTTPS URLs or file:// URIs.
  • Test the tool with different input formats, including local file paths and HTTP/HTTPS URLs, to ensure correct behavior.

Example

# Proposed upload_image tool
def upload_image(local_path):
    # Copy the file to a temp location accessible by the vision service
    temp_url = copy_file_to_temp_location(local_path)
    return temp_url

# Updated vision_analyze tool
def vision_analyze(image_url):
    # Check if the input is a valid HTTP/HTTPS URL
    if not is_valid_url(image_url):
        raise ValueError("Invalid image URL")
    # Analyze the image using the vision service
    analyze_image(image_url)

Notes

The proposed solution requires changes to the tool's implementation and may involve additional dependencies or infrastructure. The exact implementation details may vary depending on the specific requirements and constraints of the project.

Recommendation

Apply the workaround by implementing a file upload mechanism, such as the proposed upload_image tool, to handle local file paths. This approach provides a more robust and reliable solution for handling local file paths in the CLI context.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING