openclaw - 💡(How to fix) Fix codex-app-server rollout: exec_command stdout leaks into input_image base64, breaks all subsequent API calls

Summary

On 2026-05-25 around 22:33 UTC, the default-persona session on a Mac Mini deployment of OC entered a permanently-broken state. Every subsequent ChatGPT-OAuth call failed with:

Invalid input[N].output[K].image_url. Expected a base64-encoded data URL ... but got an invalid base64-encoded value.

Codex CLI surfaces this generically as:

Something went wrong while processing your request. Please try again, or use /new to start a fresh session.

…making the underlying bug invisible to the user.

Root cause: exec_command output text (specifically the formatted_truncate_text framing strings — Total output lines: N header and …N tokens truncated… middle marker) has leaked inside the base64 payload of an input_image entry written to the codex-app-server rollout JSONL. Once written, the corruption is replayed on every API call because the rollout history is included in each request.

Reproduction evidence (post-mortem)

• OC version: 2026.5.20 (downgraded from 2026.5.22 earlier the same day, well before this session was created)
• codex CLI: 0.125.0 (installed 2026-04-25, unchanged across the OC downgrade)
• Rollout file: ~/.openclaw/agents/main/agent/codex-home/sessions/2026/05/25/rollout-2026-05-25T22-32-44-019e610d-f389-7840-9f5e-643c5709c568.jsonl
• Corrupted entry: line 4774, type: response_item, payload.type: custom_tool_call_output, call_id: call_TE1GZjc3Qr8VLD3ouRJiBjkp

The payload.output array looked like:

[
  { "type": "input_text", "text": "Script completed\nWall time 1.6 seconds\nOutput:\n" },
  {
    "type": "input_image",
    "image_url": "data:image/png;base64,Totaloutputlines:1iVBORw0KGgoAAAA…"
                                       // ^^^^^^^^^^^^^^^^^^^
                                       //  the "Total output lines: 1" header
                                       //  prepended directly to the PNG base64,
                                       //  with ASCII whitespace stripped.
                                                                  
  // …~20kb of base64 later…
  // "…IFcygghZo8RGjs…52309tokenstruncated…53KX7Lh7y/mLl+w9o+PZ…"
  //                  ^^^^^^^^^^^^^^^^^^^^^
  //   the truncate_middle marker injected mid-base64 (~52309 tokens removed).
    "detail": "low"
  },
  { "type": "input_text", "text": "folder screenshot" }
]

Two distinct contamination patterns are present:

1. Header prepended: Totaloutputlines:1 at the very start of image_url (right after data:image/png;base64,). The original Rust format!("Total output lines: {total_lines}\n\n{result}") produces "Total output lines: 1\n\n…" with whitespace + newlines. The corrupted version has whitespace and the \n\n stripped but the : preserved — consistent with the bytes being run through a base64-canonicalization step (code <= 32 chars dropped) AFTER the contamination occurred but BEFORE write.

2. Mid-stream marker: …52309tokenstruncated… injected ~20kb into the image_url base64 — clearly the upstream …{removed_count} tokens truncated… marker from codex-rs/utils/string/src/truncate.rs:133 format_truncation_marker (with the ellipsis multi-byte UTF-8 preserved, but the inner whitespace stripped).

Both markers are produced by upstream Rust code (strings confirmed present in aarch64-apple-darwin/codex/codex Mach-O binary):

$ strings /opt/homebrew/lib/node_modules/@openai/codex/.../codex/codex | grep -E "Total output lines|tokens truncated"
Total output lines:
Total output lines:
 tokens truncated

…and the contaminant source files referenced in the binary are:

core/src/tools/mod.rs          — has both `Total output lines:` format calls
codex-rs/utils/output-truncation/src/lib.rs   — formatted_truncate_text / formatted_truncate_text_content_items_with_policy
codex-rs/utils/string/src/truncate.rs         — `…{removed_count} tokens truncated…`
codex-rs/core/src/tools/code_mode/mod.rs      — prepend_script_status (`{status}\nWall time {wall_time:.1} seconds\nOutput:\n`)

The "Script completed\nWall time 1.6 seconds\nOutput:\n" framing in the corrupted entry's text field is verbatim from codex-rs/core/src/tools/code_mode/mod.rs prepend_script_status, confirming the bug occurred in the code-mode tool execution path (V8 sandbox returning a script result with a screenshot + stdout, packaged into a custom_tool_call_output).

Where the bug lives (best hypothesis)

OC's installed /opt/homebrew/lib/node_modules/openclaw/dist/ JS bundle does NOT write rollout files and does NOT contain either contaminant string — verified via:

grep -rE "Total output lines|Script completed|Wall time" /opt/homebrew/lib/node_modules/openclaw/dist/
# (returns nothing)

OC's rollout files are written entirely by the upstream @openai/codex Rust binary (codex-app-server mode) which OC spawns via the shim at ~/.openclaw/bin/codex-app-server-real-home. The bug is therefore upstream-of-OC, in the codex CLI / app-server, specifically in the code-mode result assembly path where:

• exec stdout is formatted via format_exec_output_str / formatted_truncate_text (producing the Total output lines: N\n\n… header) and
• screenshot bytes are packaged as FunctionCallOutputContentItem::InputImage { image_url: "data:image/png;base64,…", … }

…and one of these two paths writes into the OTHER's allocated string buffer instead of producing a separate FunctionCallOutputContentItem::InputText entry next to the image. The exact call site that fuses the formatted-text framing into the image_url base64 needs source-side debugging at OpenAI/codex; OC cannot fix it because OC is downstream of the JSONL write.

Why this is severe

• Permanent thread breakage from a single corrupted line. Codex replays the full rollout on every API call to ChatGPT-OAuth — once line N is corrupted, every subsequent call fails with the same Invalid input[N].output[K].image_url error.
• Generic user-facing error. Codex CLI surfaces only "Something went wrong, try /new" — zero pointer to the rollout file, the corrupted line, or the underlying cause. Users will lose hours of thread context to /new without understanding why.
• OC's own image sanitizer can't help. sanitizeResponsesImagePayload in OC's dist/openai-transport-stream-…js (canonicalizeBase64 + sniffImageMime) WOULD detect this and replace the broken image with [omitted image payload]. But it only runs on OC's direct OpenAI/Azure Responses API paths (call sites at the bundle's openai-transport-stream-C2y68vfH.js:2201 and :2474). The codex-app-server path bypasses OC's transport entirely — codex's own Rust HTTP client talks to chatgpt.com/backend-api/codex/responses directly.
• Manual recovery is destructive. Only options: (a) edit the rollout JSONL by hand and replace the corrupted image_url with a 1×1 PNG placeholder, OR (b) /new and lose the thread.

Suggested fix paths (upstream codex CLI)

1. Audit codex-rs/core/src/tools/code_mode/mod.rs::prepend_script_status + truncate_code_mode_result for the specific case where content_items contains a mix of InputText + InputImage. The truncate_function_output_items_with_policy function in utils/output-truncation/src/lib.rs looks correct in isolation (it clones image_url untouched), so the contamination likely happens during prior assembly — possibly during V8-to-Rust marshalling in codex-rs/code-mode/src/runtime/value.rs::normalize_output_image, or during whatever path constructs the per-cell stdout buffer that the formatter then frames.

2. Validate input_image.image_url at rollout-write time. Cheap regex / base64 round-trip check before writing. Refuse to persist a corrupted line — fail loud rather than silently producing a thread that's permanently broken on the next call.

3. Apply the same sanitizer to codex-app-server's outbound responses-API request. Mirror OC's sanitizeResponsesImagePayload (sniff magic bytes, drop the image entry if base64 round-trip fails) inside the Rust HTTP client. Even if the JSONL is corrupted, the API call could be saved.

4. Surface a more actionable error from codex-app-server when the API returns the Invalid input[N].output[K].image_url error — point at the rollout file path + line number so users can either patch or /new knowing exactly what they're losing.

Defensive workaround we deployed (in this OC fleet, NOT in upstream)

A LaunchAgent-backed scanner watchdog walks all recent OC rollout JSONLs every 5 minutes, detects image_url payloads where Total output lines: or tokens truncated appears inside the base64, replaces them with a 1×1 transparent PNG + a sibling input_text explaining the substitution, and backs up the original. Code lives in our personal config repo:

• Script: ~/.claude/scripts/openclaw-rollout-corruption-scanner.sh
• Skill: ~/.claude/skills/openclaw-rollout-corruption-fix/SKILL.md
• Repo: https://github.com/EthanSK/dot-claude

This works around the symptom (thread continues, no more Invalid image_url errors). It does not fix the root cause.

Environment

• macOS: 26.4.1 (Build 25E253), Apple Silicon (arm64)
• OC: 2026.5.20 (installed 2026-05-25 21:21 local; downgraded from 2026.5.22 earlier same day)
• codex CLI: 0.125.0 (installed 2026-04-25, unchanged across the OC downgrade)
• Backend: ChatGPT-OAuth (https://chatgpt.com/backend-api/codex)

Mixed-state hypothesis (ruled out)

We investigated whether the 5.22 → 5.20 downgrade left files in a mixed state contributing to the bug. Ruled out by evidence:

• The OC install completed at 21:21:05 on 2026-05-25; the only file in /opt/homebrew/lib/node_modules/openclaw/ modified AFTER that timestamp is dist/fetch-DjYEgUxg.js (modified 22:31:58, an unrelated Telegram-fetch UA patch from a different watchdog).
• The corrupted session began at 22:32:44 — 71 minutes after the OC install completed, with no OC code changes in the intervening window.
• The codex Rust binary (@openai/codex 0.125.0) was installed 2026-04-25 and was completely untouched by the OC downgrade.
• The contaminant strings live exclusively in the codex Rust binary (core/src/tools/mod.rs, utils/string/src/truncate.rs, tools/code_mode/mod.rs); none of them appear in OC's dist bundle.

The corruption is reproducible-in-principle on any OC version that uses codex-app-server with @openai/codex 0.125.0; the downgrade was coincidental, not causal.