PR #67634: fix: describe images via imageModel when primary model is text-only

• Repository: openclaw/openclaw
• Author: Joel-Claw
• State: open | merged: False
• Link: https://github.com/openclaw/openclaw/pull/67634

Description (problem / solution / changelog)

Problem

When the primary model does not support images (e.g. ollama/glm-5.1:cloud, ollama/glm-4:cloud), sending a photo via Telegram crashes the session and blocks all subsequent messages, even pure text.

Root cause: the text-only model rejects the image with an HTTP 400 → classified as "format" failure → auth profile cooldown activated (30s → 5min escalating) → all messages to that profile are rejected during cooldown, including normal text messages.

Fixes #67606

Related: #51392, #59943, #66095, #39690

Root Cause

Three issues stack:

1. parseMessageWithAttachments (src/gateway/chat-attachments.ts): When supportsImages=false, all attachments are dropped with a log warning. The images never reach the media store, so no downstream pipeline can process them.

2. No imageModel fallback in Gateway path: The media-understanding pipeline (runCapability in runner.ts) already knows how to describe images using imageModel when the primary model is text-only. But this pipeline runs in the getReplyFromConfig (auto-reply) path. The Gateway chat path never calls it, so images are just gone.

3. Format error triggers profile-wide cooldown: When the image is (incorrectly) sent to a text-only model, the provider rejects with HTTP 400 → classified as "format" → auth profile cooldown → ALL subsequent messages fail, even pure text. This is what makes the session "stuck" — one bad image blocks everything.

Fix

1. Offload instead of drop (chat-attachments.ts)

When supportsImages=false, parseMessageWithAttachments now offloads all image attachments to the media store and injects media:// markers into the prompt. The images array is empty (no inline image blocks), but offloadedRefs and imageOrder are populated.

2. Describe offloaded images via imageModel (describeOffloadedImagesForTextOnlyModel)

New exported function in chat-attachments.ts that:

1. Resolves the imageModel from config using resolveAutoImageModel
2. For each offloaded ref, resolves the physical file path from the media store
3. Calls describeImageFileWithModel to describe the image via the vision-capable imageModel
4. Replaces [media attached: media://inbound/<id>] markers with [attached image: <description>]

If no imageModel is configured, or if description fails, the original media:// marker is preserved (graceful fallback).

3. Call sites updated

All three parseMessageWithAttachments call sites now call describeOffloadedImagesForTextOnlyModel when supportsImages=false and there are offloaded refs.

Example

Before: Send photo → image dropped OR leaks through → HTTP 400 "format" → profile cooldown → session stuck, ALL messages fail

After: Send photo → offloaded → described via imageModel → [attached image: A cat sitting on a windowsill] → text-only model reasons about content → no format error → session works normally

Checklist

• Code compiles without new TypeScript errors
• Tests added for new behavior
• Graceful fallback when imageModel is not configured
• Graceful fallback when image description fails (original marker preserved)
• No changes to behavior when supportsImages=true

Changed files

• src/gateway/chat-attachments.test.ts (modified, +72/-0)
• src/gateway/chat-attachments.ts (modified, +136/-23)
• src/gateway/media-understanding-describe.runtime.ts (added, +11/-0)
• src/gateway/server-methods/agent.ts (modified, +25/-5)
• src/gateway/server-methods/chat.ts (modified, +22/-5)
• src/gateway/server-node-events.runtime.ts (modified, +2/-2)
• src/gateway/server-node-events.ts (modified, +20/-3)