openclaw - ✅(Solved) Fix image tool returns 'Unknown model' for configured ollama models with image input [3 pull requests, 2 comments, 3 participants]

clawd-bot-ai · 2026-04-02T23:02:51Z

[openclaw] PR 66165: fix tools : normalize model reference before registry lookup - Repository: openclaw/openclaw - Author: yqli2420 - State: closed | merged:… # PR #66165: fix(tools): normalize model reference before registry lookup - Repository: openclaw/openclaw - Author: yqli2420 - State: closed | merged: False - Link: https://github.com/openclaw/openclaw/pull/66165 ## Description (problem / solution / changelog) ## Summary Describe the problem and fix in 2–5 bullets: - Problem: The `image` and `pdf` tools failed with "Unknown model" errors for configured Ollama models because they performed registry lookups using raw, non-normalized model references. - Why it matters: This prevented users from using valid, configured vision-capable models via the built-in tools, blocking a core workflow for image analysis. - What changed: Modified `resolveModelFromRegistry` to normalize provider/model IDs using `normalizeModelRef` before lookup. Refactored `resolveModelRuntimeApiKey` to accept normalized references and simplify API key retrieval. - What did NOT change: The underlying model registry implementation, configuration loading, or the actual model inference logic. ## Change Type (select all) - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Docs - [ ] Security hardening - [ ] Chore/infra ## Scope (select all touched areas) - [ ] Gateway / orchestration - [x] Skills / tool execution - [ ] Auth / tokens - [ ] Memory / storage - [ ] Integrations - [ ] API / contracts - [ ] UI / DX - [ ] CI/CD / infra ## Linked Issue/PR - Closes #59943 ## User-visible / Behavior Changes The `image` tool now successfully processes requests using configured Ollama models (e.g., `ollama/qwen3.5:397b-cloud`) instead of returning an "Unknown model" error. ## Security Impact (required) - New permissions/capabilities? (`No`) - Secrets/tokens handling changed? (`No`) - New/changed network calls? (`No`) - Command/tool execution surface changed? (`No`) - Data access scope changed? (`No`) - If any `Yes`, explain risk + mitigation: ## Repro + Verification ### Environment - OS: Linux (Ubuntu) - Runtime/container: Node v25.9.0 - Model/provider: ollama/qwen3.5:397b-cloud - Integration/channel (if any): CLI - Relevant config (redacted): `agents.defaults.imageModel.primary` set to `ollama/qwen3.5:397b-cloud` ### Steps 1. Configure `agents.defaults.imageModel.primary` to an Ollama model with image support (e.g., `ollama/qwen3.5:397b-cloud`). 2. Verify model status via `openclaw models status`. 3. Execute the `image` tool with a prompt and image URL. ### Expected The tool analyzes the image using the configured model and returns a description. ### Actual The tool returns a successful response with the image analysis (previously returned `{"status": "error", "tool": "image", "error": "Unknown model: ..."}`). ## Evidence - [ ] Failing test/log before + passing after - [x] Trace/log snippets - [ ] Screenshot/recording - [ ] Perf numbers (if relevant) ## Human Verification (required) - Verified scenarios: local scoped validation and targeted checks for the changed area passed - Edge cases checked: relevant changed-path scenarios covered by selected validation - What you did **not** verify: full repository integration coverage beyond the selected validation scope ## Compatibility / Migration - Backward compatible? (`Yes`) - Config/env changes? (`No`) - Migration needed? (`No`) - If yes, exact upgrade steps: N/A ## Failure Recovery (if this breaks) - How to disable/revert this change quickly: Revert the commit modifying `src/agents/tools/media-tool-shared.ts`. - Files/config to restore: `src/agents/tools/media-tool-shared.ts`. - Known bad symptoms reviewers should watch for: If `normalizeModelRef` behaves unexpectedly, valid models might fail to resolve. ## Risks and Mitiggedations The change relies on the `normalizeModelRef` utility. If this utility handles edge cases differently than the previous raw lookup, it could theoretically affect resolution. However, since it is a shared utility imported from `model-selection.js`, it aligns behavior with the rest of the system and reduces inconsistency. ## Changed files - `src/agents/tools/media-tool-shared.ts` (modified, +11/-13) --- # PR #66422: fix(tools): normalize media model lookups - Repository: openclaw/openclaw - Author: vincentkoc - State: closed | merged: True - Link: https://github.com/openclaw/openclaw/pull/66422 ## Description (problem / solution / changelog) ## Summary - Problem: the image and PDF tools looked up models in the Pi registry using raw provider/model refs instead of the normalized refs the rest of the runtime uses. - Why it matters: valid configured Ollama vision models could be rejected as `Unknown model`, which broke the built-in image tool and the PDF path that shares the same lookup helper. - What changed: `resolveModelFromRegistry()` now normalizes the provider/model pair before registry lookup and error reporting, and a focused unit test locks that seam in. - What did NOT change

openclaw2026-04-02 23:02:51

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

openclaw/openclaw#59943•Fetched 2026-04-08 02:38:29

View on GitHub

Comments

Participants

Timeline

Reactions

Author

Participants

Timeline (top)

commented ×2subscribed ×2

Error Message

The built-in image tool fails with Unknown model error when using ollama models that are configured with image input support. "status": "error", "error": "Unknown model: ollama/qwen3.5:397b-cloud"

Also tested with ollama/kimi-k2.5:cloud — same error The imageModel.primary value of qwen3.5:397b-cloud (without ollama/ prefix) matches the status output but the error shows ollama/qwen3.5:397b-cloud (with prefix), suggesting a normalization mismatch.

Fix Action

Workaround

Using MCP Vision with Gemini:

mcporter call gemini.analyze_media --args '{"media_source": "<URL>", "prompt": "..."}'

PR fix notes

PR #66165: fix(tools): normalize model reference before registry lookup

Repository: openclaw/openclaw
Author: yqli2420
State: closed | merged: False
Link: https://github.com/openclaw/openclaw/pull/66165

Description (problem / solution / changelog)

Summary

Describe the problem and fix in 2–5 bullets:

Problem: The image and pdf tools failed with "Unknown model" errors for configured Ollama models because they performed registry lookups using raw, non-normalized model references.
Why it matters: This prevented users from using valid, configured vision-capable models via the built-in tools, blocking a core workflow for image analysis.
What changed: Modified resolveModelFromRegistry to normalize provider/model IDs using normalizeModelRef before lookup. Refactored resolveModelRuntimeApiKey to accept normalized references and simplify API key retrieval.
What did NOT change: The underlying model registry implementation, configuration loading, or the actual model inference logic.

Change Type (select all)

Scope (select all touched areas)

Linked Issue/PR

Closes #59943

User-visible / Behavior Changes

The image tool now successfully processes requests using configured Ollama models (e.g., ollama/qwen3.5:397b-cloud) instead of returning an "Unknown model" error.

Security Impact (required)

New permissions/capabilities? (No)
Secrets/tokens handling changed? (No)
New/changed network calls? (No)
Command/tool execution surface changed? (No)
Data access scope changed? (No)
If any Yes, explain risk + mitigation:

Repro + Verification

Environment

OS: Linux (Ubuntu)
Runtime/container: Node v25.9.0
Model/provider: ollama/qwen3.5:397b-cloud
Integration/channel (if any): CLI
Relevant config (redacted): agents.defaults.imageModel.primary set to ollama/qwen3.5:397b-cloud

Steps

Configure agents.defaults.imageModel.primary to an Ollama model with image support (e.g., ollama/qwen3.5:397b-cloud).
Verify model status via openclaw models status.
Execute the image tool with a prompt and image URL.

Expected

The tool analyzes the image using the configured model and returns a description.

Actual

The tool returns a successful response with the image analysis (previously returned {"status": "error", "tool": "image", "error": "Unknown model: ..."}).

Evidence

Failing test/log before + passing after
Trace/log snippets
Screenshot/recording
Perf numbers (if relevant)

Human Verification (required)

Verified scenarios: local scoped validation and targeted checks for the changed area passed
Edge cases checked: relevant changed-path scenarios covered by selected validation
What you did not verify: full repository integration coverage beyond the selected validation scope

Compatibility / Migration

Backward compatible? (Yes)
Config/env changes? (No)
Migration needed? (No)
If yes, exact upgrade steps: N/A

Failure Recovery (if this breaks)

How to disable/revert this change quickly: Revert the commit modifying src/agents/tools/media-tool-shared.ts.
Files/config to restore: src/agents/tools/media-tool-shared.ts.
Known bad symptoms reviewers should watch for: If normalizeModelRef behaves unexpectedly, valid models might fail to resolve.

Risks and Mitiggedations

The change relies on the normalizeModelRef utility. If this utility handles edge cases differently than the previous raw lookup, it could theoretically affect resolution. However, since it is a shared utility imported from model-selection.js, it aligns behavior with the rest of the system and reduces inconsistency.

Changed files

src/agents/tools/media-tool-shared.ts (modified, +11/-13)

PR #66422: fix(tools): normalize media model lookups

Repository: openclaw/openclaw
Author: vincentkoc
State: closed | merged: True
Link: https://github.com/openclaw/openclaw/pull/66422

Description (problem / solution / changelog)

Summary

Problem: the image and PDF tools looked up models in the Pi registry using raw provider/model refs instead of the normalized refs the rest of the runtime uses.
Why it matters: valid configured Ollama vision models could be rejected as Unknown model, which broke the built-in image tool and the PDF path that shares the same lookup helper.
What changed: resolveModelFromRegistry() now normalizes the provider/model pair before registry lookup and error reporting, and a focused unit test locks that seam in.
What did NOT change (scope boundary): I did not carry the auth helper refactor from #66165, because it introduced real PDF-tool breakage and was unrelated to the actual lookup bug.

Change Type (select all)

Scope (select all touched areas)

Linked Issue/PR

Closes #59943
Related #66165
This PR fixes a bug or regression

Root Cause (if applicable)

Root cause: src/agents/tools/media-tool-shared.ts called modelRegistry.find() with raw tool-path refs, while the discovered-model registry expects the same normalized provider/model identity used elsewhere.
Missing detection / guardrail: there was no direct unit test around the shared media-tool registry lookup helper, so a normalization mismatch could slip through without tripping tool tests.
Contributing context (if known): src/media-understanding/image.ts already normalized before registry lookup; the shared media-tool helper had drifted from that working pattern.

Regression Test Plan (if applicable)

Coverage level that should have caught this:
- Unit test
- Seam / integration test
- End-to-end test
- Existing coverage already sufficient
Target test or file: src/agents/tools/media-tool-shared.model-resolution.test.ts
Scenario the test should lock in: the helper normalizes the provider/model ref before registry lookup and reports misses with the normalized ref.
Why this is the smallest reliable guardrail: it exercises the exact shared seam used by both image and PDF tools without needing full tool/runtime setup.
Existing test that already covers this (if any): nearby image/PDF tool suites verify the surrounding paths still stay green.
If no new test is added, why not: N/A

User-visible / Behavior Changes

Configured Ollama vision models stop failing the built-in image/PDF tool lookup with Unknown model when the ref only differs by normalization.

Diagram (if applicable)

Before:
[tool config ref] -> resolveModelFromRegistry -> raw modelRegistry.find(provider, model) -> Unknown model

After:
[tool config ref] -> resolveModelFromRegistry -> normalizeModelRef(...) -> modelRegistry.find(normalized provider, normalized model)

Security Impact (required)

New permissions/capabilities? (No)
Secrets/tokens handling changed? (No)
New/changed network calls? (No)
Command/tool execution surface changed? (No)
Data access scope changed? (No)
If any Yes, explain risk + mitigation:

Repro + Verification

Environment

OS: macOS 26.4 local maintainer worktree
Runtime/container: Node 22+, pnpm
Model/provider: Ollama vision-model lookup path
Integration/channel (if any): built-in image/PDF tools
Relevant config (redacted): configured image-model / PDF shared lookup path

Steps

Resolve a configured image/PDF tool model through the shared media-tool helper.
Ensure the helper normalizes the ref before hitting the registry.
Verify the surrounding image/PDF tool suites still pass.

Expected

Valid configured models resolve from the registry after normalization.

Actual

Before this change, the shared helper used the raw ref and could throw Unknown model even when the configured model existed.

Evidence

Attach at least one:

Failing test/log before + passing after
Trace/log snippets
Screenshot/recording
Perf numbers (if relevant)

Human Verification (required)

Verified scenarios: pnpm test:serial src/agents/tools/media-tool-shared.test.ts src/agents/tools/media-tool-shared.model-resolution.test.ts; pnpm test:serial src/agents/tools/image-tool.test.ts src/agents/tools/pdf-tool.test.ts src/agents/tools/pdf-tool.model-config.test.ts src/agents/tools/pdf-tool.model-catalog.test.ts src/agents/tools/pdf-tool.helpers.test.ts; pnpm build
Edge cases checked: the shared helper now reports misses with the normalized ref, and nearby image/PDF suites stayed green.
What you did not verify: I did not run a live Ollama image-tool invocation end-to-end.

Review Conversations

I replied to or resolved every bot review conversation I addressed in this PR.
I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

Backward compatible? (Yes)
Config/env changes? (No)
Migration needed? (No)
If yes, exact upgrade steps:

Risks and Mitigations

Risk: the helper now relies on the standard model-ref normalization path before lookup.
- Mitigation: this mirrors the existing media-understanding lookup path and is covered by both a new unit seam test and the surrounding tool suites.

Changed files

CHANGELOG.md (modified, +1/-0)
src/agents/tools/media-tool-shared.model-resolution.test.ts (added, +62/-0)
src/agents/tools/media-tool-shared.ts (modified, +7/-2)

PR #67634: fix: describe images via imageModel when primary model is text-only

Repository: openclaw/openclaw
Author: Joel-Claw
State: open | merged: False
Link: https://github.com/openclaw/openclaw/pull/67634

Description (problem / solution / changelog)

Problem

When the primary model does not support images (e.g. ollama/glm-5.1:cloud, ollama/glm-4:cloud), sending a photo via Telegram crashes the session and blocks all subsequent messages, even pure text.

Root cause: the text-only model rejects the image with an HTTP 400 → classified as "format" failure → auth profile cooldown activated (30s → 5min escalating) → all messages to that profile are rejected during cooldown, including normal text messages.

Fixes #67606

Related: #51392, #59943, #66095, #39690

Root Cause

Three issues stack:

parseMessageWithAttachments (src/gateway/chat-attachments.ts): When supportsImages=false, all attachments are dropped with a log warning. The images never reach the media store, so no downstream pipeline can process them.
No imageModel fallback in Gateway path: The media-understanding pipeline (runCapability in runner.ts) already knows how to describe images using imageModel when the primary model is text-only. But this pipeline runs in the getReplyFromConfig (auto-reply) path. The Gateway chat path never calls it, so images are just gone.
Format error triggers profile-wide cooldown: When the image is (incorrectly) sent to a text-only model, the provider rejects with HTTP 400 → classified as "format" → auth profile cooldown → ALL subsequent messages fail, even pure text. This is what makes the session "stuck" — one bad image blocks everything.

Fix

1. Offload instead of drop (`chat-attachments.ts`)

When supportsImages=false, parseMessageWithAttachments now offloads all image attachments to the media store and injects media:// markers into the prompt. The images array is empty (no inline image blocks), but offloadedRefs and imageOrder are populated.

2. Describe offloaded images via imageModel (`describeOffloadedImagesForTextOnlyModel`)

New exported function in chat-attachments.ts that:

Resolves the imageModel from config using resolveAutoImageModel
For each offloaded ref, resolves the physical file path from the media store
Calls describeImageFileWithModel to describe the image via the vision-capable imageModel
Replaces [media attached: media://inbound/<id>] markers with [attached image: <description>]

If no imageModel is configured, or if description fails, the original media:// marker is preserved (graceful fallback).

3. Call sites updated

All three parseMessageWithAttachments call sites now call describeOffloadedImagesForTextOnlyModel when supportsImages=false and there are offloaded refs.

Example

Before: Send photo → image dropped OR leaks through → HTTP 400 "format" → profile cooldown → session stuck, ALL messages fail

After: Send photo → offloaded → described via imageModel → [attached image: A cat sitting on a windowsill] → text-only model reasons about content → no format error → session works normally

Checklist

Code compiles without new TypeScript errors
Tests added for new behavior
Graceful fallback when imageModel is not configured
Graceful fallback when image description fails (original marker preserved)
No changes to behavior when supportsImages=true

Changed files

src/gateway/chat-attachments.test.ts (modified, +72/-0)
src/gateway/chat-attachments.ts (modified, +136/-23)
src/gateway/media-understanding-describe.runtime.ts (added, +11/-0)
src/gateway/server-methods/agent.ts (modified, +25/-5)
src/gateway/server-methods/chat.ts (modified, +22/-5)
src/gateway/server-node-events.runtime.ts (modified, +2/-2)
src/gateway/server-node-events.ts (modified, +20/-3)

Code Example

"imageModel": {
     "primary": "ollama/qwen3.5:397b-cloud"
   }

---

$ openclaw models status
   Image model   : qwen3.5:397b-cloud
   
   $ openclaw models list --all | grep qwen
   ollama/qwen3.5:397b-cloud                  text+image 256k     yes   no    fallback#1,configured,alias:qwen

---

image prompt="Describe this image" image="https://example.com/image.jpg"

---

{
  "status": "error",
  "tool": "image",
  "error": "Unknown model: ollama/qwen3.5:397b-cloud"
}

---

mcporter call gemini.analyze_media --args '{"media_source": "<URL>", "prompt": "..."}'

---

"models": {
  "providers": {
    "ollama": {
      "models": [
        {
          "id": "qwen3.5:397b-cloud",
          "input": ["text", "image"],
          ...
        }
      ]
    }
  }
}

RAW_BUFFERClick to expand / collapse

Bug Description

The built-in image tool fails with Unknown model error when using ollama models that are configured with image input support.

Environment

OpenClaw Version: 2026.4.2
OS: Linux (Ubuntu)
Node: v25.9.0

Steps to Reproduce

Configure agents.defaults.imageModel.primary to an ollama model with image support:
```
"imageModel": {
  "primary": "ollama/qwen3.5:397b-cloud"
}
```

Verify model is configured and supports images:

$ openclaw models status
Image model   : qwen3.5:397b-cloud

$ openclaw models list --all | grep qwen
ollama/qwen3.5:397b-cloud                  text+image 256k     yes   no    fallback#1,configured,alias:qwen

Use the image tool with an image URL or file:

image prompt="Describe this image" image="https://example.com/image.jpg"

Expected Behavior

The image tool should analyze the image using the configured model.

Actual Behavior

{
  "status": "error",
  "tool": "image",
  "error": "Unknown model: ollama/qwen3.5:397b-cloud"
}

Additional Context

Also tested with ollama/kimi-k2.5:cloud — same error
Both models show text+image input support in models list --all
MCP Vision (gemini.analyze_media) works correctly for image analysis
Model is listed as configured in models status

Workaround

Using MCP Vision with Gemini:

mcporter call gemini.analyze_media --args '{"media_source": "<URL>", "prompt": "..."}'

Investigation

The issue appears to be in how the image tool resolves the model reference. The model is properly configured in openclaw.json and shows up in models status, but the tool doesn't recognize it.

"models": {
  "providers": {
    "ollama": {
      "models": [
        {
          "id": "qwen3.5:397b-cloud",
          "input": ["text", "image"],
          ...
        }
      ]
    }
  }
}

The imageModel.primary value of qwen3.5:397b-cloud (without ollama/ prefix) matches the status output but the error shows ollama/qwen3.5:397b-cloud (with prefix), suggesting a normalization mismatch.

extent analysis

TL;DR

The most likely fix is to update the imageModel.primary configuration to include the full model reference, including the provider prefix.

Guidance

Verify that the model reference in imageModel.primary matches the format expected by the image tool, including any required prefixes.
Check the models configuration in openclaw.json to ensure the model is properly defined with the correct id and input fields.
Test using the full model reference, including the provider prefix, in the imageModel.primary configuration, e.g., "ollama/qwen3.5:397b-cloud".
If the issue persists, try using the workaround provided, which utilizes MCP Vision with Gemini for image analysis.

Example

"imageModel": {
  "primary": "ollama/qwen3.5:397b-cloud"
}

Notes

The investigation suggests a normalization mismatch between the model reference in the configuration and the format expected by the image tool. Updating the imageModel.primary configuration to include the full model reference may resolve the issue.

Recommendation

Apply workaround: Utilize MCP Vision with Gemini for image analysis, as it is reported to work correctly, while further investigating the model reference normalization issue.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#agent execution #callback error #memory management #API rate limit #retriever error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

openclaw - ✅(Solved) Fix image tool returns 'Unknown model' for configured ollama models with image input [3 pull requests, 2 comments, 3 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Fix Action

Workaround

PR fix notes

PR #66165: fix(tools): normalize model reference before registry lookup

Description (problem / solution / changelog)

Summary

Change Type (select all)

Scope (select all touched areas)

Linked Issue/PR

User-visible / Behavior Changes

Security Impact (required)

Repro + Verification

Environment

Steps

Expected

Actual

Evidence

Human Verification (required)

Compatibility / Migration

Failure Recovery (if this breaks)

Risks and Mitiggedations

Changed files

PR #66422: fix(tools): normalize media model lookups

Description (problem / solution / changelog)

Summary

Change Type (select all)

Scope (select all touched areas)

Linked Issue/PR

Root Cause (if applicable)

Regression Test Plan (if applicable)

User-visible / Behavior Changes

Diagram (if applicable)

Security Impact (required)

Repro + Verification

Environment

Steps

Expected

Actual

Evidence

Human Verification (required)

Review Conversations

Compatibility / Migration

Risks and Mitigations

Changed files

PR #67634: fix: describe images via imageModel when primary model is text-only

Description (problem / solution / changelog)

Problem

Root Cause

Fix

1. Offload instead of drop (chat-attachments.ts)

2. Describe offloaded images via imageModel (describeOffloadedImagesForTextOnlyModel)

3. Call sites updated

Example

Checklist

Changed files

Code Example

Bug Description

Environment

Steps to Reproduce

Expected Behavior

Actual Behavior

Additional Context

Workaround

Investigation

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

RELATED_DISCOVERY

TRENDING

1. Offload instead of drop (`chat-attachments.ts`)

2. Describe offloaded images via imageModel (`describeOffloadedImagesForTextOnlyModel`)