hermes - 💡(How to fix) Fix [Feature]: Add native vision tool-result support for Alibaba/Qwen vision models

Problem or Use Case

I'm trying to use Hermes Agent with Alibaba/Qwen vision-capable models for visual desktop automation workflows, especially computer_use screenshots and vision_analyze.

For example, qwen3.6-plus supports image input, but in Hermes Agent v0.14.0 the current native multimodal tool-result path does not appear to cover the Alibaba/Qwen provider stack.

As a result, when using Qwen vision models for desktop/browser visual tasks, Hermes may treat the active model/provider as not supporting image input and fall back to text-only behavior or report that the model cannot process screenshots.

I understand that my local setup uses a custom provider alias named alibaba-coding-plan, which may not be an official Hermes provider name. That part may be local configuration. However, even the standard alibaba provider does not appear to be included in _supports_media_in_tool_results(), so Alibaba/Qwen vision models cannot use the native vision tool-result fast path.

Proposed Solution

Add native vision tool-result support for Alibaba/Qwen vision-capable models.

Specifically:

1. Update tools/vision_tools.py

_supports_media_in_tool_results(provider, model) currently explicitly supports providers such as OpenAI, Anthropic, some Gemini variants, and several aggregators.

Please consider adding support for the standard Alibaba/Qwen provider stack, gated by known vision-capable Qwen model names or model capabilities.

Example model markers:

qwen_vision_markers = (
    "qwen3.6-plus",
    "qwen3.5-plus",
    "qwen3-vl",
    "qwen-vl",
    "qvq",
    "qwen-omni",
    "qwen3-omni",
    "qwen2.5-vl",
    "qwen2-5-vl",
)

The goal is not to enable image input for all Qwen models. Text/code-only models such as qwen3-coder should still fall back to the text path.

Consider a config-level capability source for custom provider aliases Hermes currently resolves model capabilities through agent/models_dev.py and PROVIDER_TO_MODELS_DEV.

If a user defines a custom provider alias that points to an Alibaba/Qwen-compatible backend, Hermes cannot reuse alibaba models.dev capability metadata unless the alias is hardcoded.

A possible config option could be:

custom_providers:
  - name: my-qwen-provider
    models_dev_provider: alibaba

or similar.

This would let custom provider aliases inherit capability metadata from a known provider without adding every local alias to PROVIDER_TO_MODELS_DEV.

### Alternatives Considered

1. Switch to OpenAI, Anthropic, Gemini, or OpenRouter for `computer_use`

This works around the issue because those provider stacks are already covered by the native vision tool-result path. However, it does not help users who want to use Qwen vision models directly.

2. Use `vision_analyze` as a text fallback

This can work for simple screenshots, but it is less reliable for desktop automation because the model does not receive the screenshot as native visual context.

3. Hardcode my local alias in `PROVIDER_TO_MODELS_DEV`

I tested this locally for `alibaba-coding-plan`, but this is probably not appropriate as a built-in upstream fix because the alias is local to my setup.

A config-level capability mapping would be cleaner.

### Feature Type

Configuration option

### Scope

Small (single file, < 50 lines)

### Contribution

- [x] I'd like to implement this myself and submit a PR

### Debug Report (optional)

```shell
Local validation:

With a local patch:

- `get_model_capabilities("alibaba-coding-plan", "qwen3.6-plus").supports_vision == True`
- `qwen3-coder` remains treated as non-vision for native tool-result purposes
- Related tests passed locally:


tests/agent/test_models_dev.py
tests/tools/test_vision_native_fast_path.py

51 passed
Note: alibaba-coding-plan is my local provider alias and may not be an official Hermes provider name.

I'm happy to help test this locally and submit a PR if the maintainers agree with the proposed direction.