PR #58017: fix(xai): normalize image tool results for responses

• Repository: openclaw/openclaw
• Author: neeravmakwana
• State: open | merged: False
• Link: https://github.com/openclaw/openclaw/pull/58017

Description (problem / solution / changelog)

Summary

• Problem: xAI openai-responses requests could replay image-bearing tool results as array-valued function_call_output.output, which xAI rejects with a 422 deserialization error.
• Why it matters: read(image) and similar tool flows could succeed on the tool call itself, then crash on the next model turn instead of continuing the session.
• What changed: the xAI stream payload compatibility wrapper now rewrites array-valued function_call_output items into string outputs and, when the model explicitly supports image input, emits the image blocks as a following user message instead.
• What did NOT change (scope boundary): core transcript structures, non-xAI providers, and upstream pi-ai response conversion logic.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes #57981
• Related #57981
• This PR fixes a bug or regression

Root Cause / Regression History (if applicable)

• Root cause: the xAI provider uses the OpenAI Responses transport, and the current payload builder can emit image-bearing tool results as function_call_output.output = [input_text, input_image]; xAI rejects that shape.
• Missing detection / guardrail: the xAI compatibility wrapper stripped unsupported reasoning/schema fields but did not normalize image-bearing tool-result payloads.
• Prior context (git blame, prior PR, issue, or refactor if known): issue #57981 captured the failing payload shape and 422 symptom for read(image) flows.
• Why this regressed now: the provider path preserved structured image tool results into the Responses payload, but xAI's endpoint is stricter than direct OpenAI here.
• If unknown, what was ruled out: verified current code still reproduces the array-valued function_call_output.output payload; this is not already normalized in the current xAI wrapper layer.

Regression Test Plan (if applicable)

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file: extensions/xai/stream.test.ts
• Scenario the test should lock in: image-bearing tool results sent through the xAI openai-responses wrapper must become string function_call_output.output plus a following user image message.
• Why this is the smallest reliable guardrail: the bug lives inside the xAI payload compatibility wrapper, so a focused wrapper test exercises the exact failing shape without depending on live provider calls.
• Existing test that already covers this (if any): none for this payload shape.
• If no new test is added, why not: N/A

User-visible / Behavior Changes

• xAI-backed sessions can continue after tool results that include image blocks instead of failing on the follow-up turn with a 422.

Diagram (if applicable)

Before:
[tool returns text + image blocks] -> [function_call_output.output contains input_image] -> [xAI 422]

After:
[tool returns text + image blocks] -> [function_call_output.output becomes text] -> [image blocks replayed as user message] -> [session continues]

Security Impact (required)

• New permissions/capabilities? (Yes/No) No
• Secrets/tokens handling changed? (Yes/No) No
• New/changed network calls? (Yes/No) No
• Command/tool execution surface changed? (Yes/No) No
• Data access scope changed? (Yes/No) No
• If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

• OS: macOS (local verification) and issue repro from Ubuntu in #57981
• Runtime/container: local repo checkout
• Model/provider: xAI / openai-responses
• Integration/channel (if any): agent session / tool replay
• Relevant config (redacted): xAI provider using https://api.x.ai/v1

Steps

1. Configure an xAI model on the openai-responses path.
2. Produce a tool result with text plus image blocks, such as read on a PNG.
3. Replay that result into the next model turn.

Expected

• The follow-up request stays xAI-compatible and the session continues.

Actual

• Before this change, the request can include array-valued function_call_output.output with input_image, which causes xAI to fail with a 422.

Evidence

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Human Verification (required)

• Verified scenarios: reproduced the current payload shape locally, added a regression test for the xAI wrapper, ran pnpm test -- extensions/xai/stream.test.ts, ran pnpm test:extension xai, and ran pnpm build.
• Edge cases checked: tool results with text plus image blocks; models that do not explicitly advertise image input keep a string-only tool output.
• What you did not verify: live xAI API execution in this environment.

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? (Yes/No) Yes
• Config/env changes? (Yes/No) No
• Migration needed? (Yes/No) No
• If yes, exact upgrade steps: N/A

Risks and Mitigations

• Risk: xAI models without explicit image input support could still receive invalid image blocks.
  • Mitigation: the wrapper only forwards image blocks when the resolved model explicitly declares image input support; otherwise it keeps a string-only tool output.

Notes

• AI-assisted: yes.
• pnpm check currently stops on unrelated existing tsgo errors in extensions/diffs/src/language-hints.test.ts on top of origin/main; the touched xAI lane and build both passed.

Made with Cursor

Changed files

• extensions/xai/stream.test.ts (modified, +174/-0)
• extensions/xai/stream.ts (modified, +81/-0)