PR #71608: fix(agents): prevent Bedrock replay death loop on empty assistant content

• Repository: openclaw/openclaw
• Author: openperf
• State: closed | merged: False
• Link: https://github.com/openclaw/openclaw/pull/71608

Description (problem / solution / changelog)

Summary

• Problem: When a stream/provider error fires before any assistant content block is produced, OpenClaw persists {"role":"assistant","content":[],...} to the session JSONL via buildStreamErrorAssistantMessage (src/agents/stream-message-shared.ts:75-90). On the next turn, AWS Bedrock Converse rejects the replay with "The content field in the Message object at messages.N is empty. Add a ContentBlock object to the content field and try again." Each subsequent user message replays the same poisoned turn, so the session enters a death loop and a gateway restart does not help (the JSONL is already corrupt). Affected sessions reported in #71572 cover both embedded agent runs and Feishu channel delivery.
• Root Cause: Three transcript-hygiene gaps line up to make the failure permanent:
  1. buildStreamErrorAssistantMessage creates the error turn with content: [] (line 84). For Bedrock Converse, an assistant message must contain at least one ContentBlock, so this entry is invalid the moment it is persisted.
  2. normalizeAssistantReplayContent in src/agents/pi-embedded-runner/replay-history.ts (line 230) only normalises legacy string content; it skips empty arrays entirely. It also forwards openclaw transcript-only mirror entries (provider:"openclaw", model:"delivery-mirror" from src/config/sessions/transcript.ts:128 and model:"gateway-injected" from src/gateway/server-methods/chat-transcript-inject.ts:89) to the actual provider, which duplicates content and, on strict providers, breaks turn-ordering rules.
  3. repairSessionFileIfNeeded in src/agents/session-file-repair.ts only drops malformed JSONL lines. The poisoned line is valid JSON, so repair leaves it untouched and the death loop survives every restart.
• Fix: Address each gap at its origin so a poisoned session cannot form, an in-flight session cannot replay poison even from older builds, and a session already poisoned on disk recovers without manual surgery.
  1. buildStreamErrorAssistantMessage now writes [{type:"text", text: errorMessage || sentinel}] so the persisted turn is replay-valid for Bedrock and remains useful for clients reading content.
  2. normalizeAssistantReplayContent now (a) drops openclaw delivery-mirror / gateway-injected assistant entries from the in-memory replay copy (the persisted JSONL is unchanged, so user-facing transcript surfaces stay intact) and (b) substitutes a sentinel text block for any assistant entry whose content is an empty array. Existing string-content wrapping behavior is preserved.
  3. repairSessionFileIfNeeded now also rewrites persisted assistant entries that have content: [] to the sentinel text block, in addition to the existing malformed-JSONL drop. The rewrite is idempotent: a session with no empty-content entries returns repaired:false and bytes on disk are unchanged.
  • The sentinel string [assistant turn failed before producing content] is reused in all three call sites so a session repaired offline reads identically to a live stream-error turn. It is kept as a module-local constant in each file to avoid creating a cross-module coupling for one literal.
• What changed:
  • src/agents/stream-message-shared.ts — export STREAM_ERROR_FALLBACK_TEXT; buildStreamErrorAssistantMessage returns a non-empty text block instead of content: [].
  • src/agents/pi-embedded-runner/replay-history.ts — extend normalizeAssistantReplayContent to drop transcript-only openclaw assistant entries and to substitute the sentinel for empty content arrays; preserve original-array-reference fast path when nothing changes.
  • src/agents/session-file-repair.ts — extend repairSessionFileIfNeeded to detect-and-rewrite assistant entries with empty content; add rewrittenAssistantMessages to RepairReport (additive, optional); update warn message; combine drop and rewrite signals into the repaired flag.
  • src/agents/stream-message-shared.test.ts (new, co-located) — covers: never returns empty content; carries the error message verbatim in a single text block; falls back to sentinel when errorMessage is blank.
  • src/agents/pi-embedded-runner/replay-history.test.ts (new, co-located) — covers: empty assistant content array becomes sentinel; legacy string-content wrapping still works; delivery-mirror and gateway-injected are filtered from replay; original array reference returned when nothing changes.
  • src/agents/session-file-repair.test.ts — adds two cases: persisted assistant content: [] is rewritten with backup; an already-repaired session is a no-op (idempotent).
• What did NOT change (scope boundary):
  • No persisted JSONL entry is dropped from disk by repair. Mirror filtering happens only in the in-memory replay copy in normalizeAssistantReplayContent; UI/transcript history surfaces remain unaffected.
  • errorMessage field on the assistant message is preserved verbatim, including blank/whitespace input, so any client that reads errorMessage separately still sees the original payload.
  • RepairReport only gains an optional field (rewrittenAssistantMessages?: number); existing callers that ignore the return value or only read repaired/droppedLines are unaffected.
  • No changes to provider plugins, transcript-policy, image sanitization, tool-call repair, compaction, or session-file format/version.
  • The Ollama plugin has its own local buildStreamErrorAssistantMessage (extensions/ollama/src/stream.ts:289) that is intentionally left untouched here; it is provider-scoped and not in the Bedrock replay path. A separate change can mirror this fix once the plugin contract owners agree.

Reproduction

1. Run an OpenClaw 2026.4.23 / 2026.4.25 gateway against AWS Bedrock Converse (e.g. anthropic.claude-3-haiku or any other Bedrock-hosted model).
2. From an embedded agent or Feishu channel, send a prompt that triggers a stream/provider error before the model emits any content block (network drop, transient ValidationException, or any error path that flows through buildStreamErrorAssistantMessage).
3. Observe the session JSONL: the new assistant entry has "content": [], "stopReason": "error".
4. Send any next user message in the same session. The Bedrock SDK rejects with "The content field in the Message object at messages.N is empty. Add a ContentBlock object to the content field and try again."
5. Restart the gateway and retry — the same validation error fires because the JSONL is unchanged.

After this fix:

• Step 3 produces "content": [{"type":"text","text": "<errorMessage or sentinel>"}], so the persisted turn is replay-valid.
• For sessions already poisoned by older builds, the session-file repair pass on the next load rewrites the empty-content entry to a sentinel text block (with backup) and the session resumes; rerunning repair is a no-op.
• Replay drops openclaw delivery-mirror / gateway-injected mirror entries from the provider request, removing a separate class of strict-provider rejections (see replay-history.ts line ~568 comment for the related assistant-first ordering case).

Risk / Mitigation

• Risk: Changing buildStreamErrorAssistantMessage shape might affect downstream consumers that special-case content.length === 0.
  • Mitigation: All call sites surveyed (src/agents/openai-ws-stream.ts:1284, src/agents/pi-embedded-runner/run/attempt.stop-reason-recovery.ts:49,78,119) pass the value through to the agent loop's error event, which already iterates content defensively. The errorMessage field is preserved verbatim for clients that surface it separately. Unit test stream-message-shared.test.ts pins both behaviors.
• Risk: Filtering delivery-mirror / gateway-injected from replay could remove user-visible turns from the model's context.
  • Mitigation: These entries are transcript-only by design — transcript.ts:264 isRedundantDeliveryMirror and chat-transcript-inject.ts document them as channel-delivery / gateway-injection records, not model output. Persisted JSONL is unchanged, so transcript history surfaces (WebChat, TUI, REST, SSE) keep showing them. The strict-provider ordering fallback at replay-history.ts:564-574 is unchanged and remains the safety net for any provider hooks that already rewrite ordering.
• Risk: Repair rewriting on-disk content could damage a session if the rewrite path crashes.
  • Mitigation: Repair already writes a .bak-${pid}-${ts} backup before any rewrite and uses an atomic tmp → rename swap. The new rewrite path reuses the same write/backup machinery; failure paths still clean up the tmp file and return repaired:false with a reason. Idempotency test ensures a second call is a no-op on a healed session.
• Risk: Adding rewrittenAssistantMessages to RepairReport could be a typed-API regression for external callers.
  • Mitigation: Field is optional (?:) and additive; all in-tree callers either ignore the return value or only read repaired/droppedLines/reason. No external consumers found in the repo.

Change Type (select all)

• Bug fix

Scope (select all touched areas)

• Gateway
• Agents (embedded runner / replay)
• Session storage (file repair)
• Provider replay (Bedrock Converse)

Changed files

• src/agents/pi-embedded-runner/replay-history.test.ts (added, +85/-0)
• src/agents/pi-embedded-runner/replay-history.ts (modified, +56/-10)
• src/agents/session-file-repair.test.ts (modified, +108/-0)
• src/agents/session-file-repair.ts (modified, +68/-6)
• src/agents/stream-message-shared.test.ts (added, +44/-0)
• src/agents/stream-message-shared.ts (modified, +18/-1)

PR #71627: fix(agents): prevent Bedrock replay death loop on empty assistant content

• Repository: openclaw/openclaw
• Author: openperf
• State: closed | merged: True
• Link: https://github.com/openclaw/openclaw/pull/71627

Description (problem / solution / changelog)

Summary

• Problem: When AWS Bedrock Converse fires a stream/provider error before any assistant content block is produced, OpenClaw persists {"role":"assistant","content":[],"stopReason":"error",...} to the session JSONL via buildStreamErrorAssistantMessage (src/agents/stream-message-shared.ts:75-90). On the next turn Bedrock rejects the replay with "The content field in the Message object at messages.N is empty. Add a ContentBlock object to the content field and try again." Each subsequent user message replays the same poisoned turn, so the session enters a death loop and a gateway restart does not help (the JSONL is already corrupt). Affected sessions reported in #71572 cover both embedded agent runs and Feishu channel delivery.
• Root Cause: Three transcript-hygiene gaps line up to make the failure permanent:
  1. buildStreamErrorAssistantMessage creates the error turn with content: []. For Bedrock Converse, an assistant message must contain at least one ContentBlock, so this entry is invalid the moment it is persisted.
  2. normalizeAssistantReplayContent in src/agents/pi-embedded-runner/replay-history.ts only normalises legacy string content; it skips empty arrays entirely. It also forwards openclaw transcript-only mirror entries (provider:"openclaw", model:"delivery-mirror" from src/config/sessions/transcript.ts:128 and model:"gateway-injected" from src/gateway/server-methods/chat-transcript-inject.ts:89) to the actual provider, which duplicates content and, on strict providers, breaks turn-ordering rules.
  3. repairSessionFileIfNeeded in src/agents/session-file-repair.ts only drops malformed JSONL lines. The poisoned line is valid JSON, so repair leaves it untouched and the death loop survives every restart.
• Fix: Address each gap at its origin so a poisoned session cannot form, an in-flight session cannot replay poison even from older builds, and a session already poisoned on disk recovers without manual surgery. The fixes are scoped strictly to stopReason: "error" turns — empty content: [] is also legitimately produced by the silent-reply / NO_REPLY path (stopReason: "stop", output: 0) locked in by src/agents/pi-embedded-runner/run.empty-error-retry.test.ts. Substituting a failure sentinel into those turns would fabricate a failure statement in the next provider request and change model behavior even when no failure occurred, so they are left untouched.
  1. buildStreamErrorAssistantMessage writes a single canonical sentinel text block (STREAM_ERROR_FALLBACK_TEXT = "[assistant turn failed before producing content]") into content. The raw provider error string stays in the peer errorMessage field and is intentionally NOT placed in content because that array is replayed back to the model on the next turn — provider error strings can carry hostnames or upstream metadata, so replaying them as assistant content opens a prompt-injection / information-disclosure surface (CWE-200). Clients/UIs read errorMessage directly; providers do not include it in their wire payloads.
  2. normalizeAssistantReplayContent now (a) drops openclaw delivery-mirror / gateway-injected transcript-only assistant entries from the in-memory replay copy (the persisted JSONL is unchanged, so user-facing transcript surfaces stay intact) and (b) substitutes the sentinel text block when content is an empty array and stopReason === "error". Existing legacy string-content wrapping behavior is preserved.
  3. repairSessionFileIfNeeded rewrites persisted assistant entries that have content: [] and stopReason: "error" to the sentinel block, in addition to the existing malformed-JSONL drop. Idempotent: a session with no eligible entries returns repaired:false and bytes on disk are unchanged.
  • The sentinel string is exported from stream-message-shared.ts and imported by both replay-history and session-file-repair so a session repaired offline reads byte-identically to a live stream-error turn — that byte-identity is what keeps the repair pass idempotent.
• What changed:
  • src/agents/stream-message-shared.ts — export STREAM_ERROR_FALLBACK_TEXT; buildStreamErrorAssistantMessage returns a non-empty content block carrying only the sentinel (raw error stays in errorMessage).
  • src/agents/pi-embedded-runner/replay-history.ts — extend normalizeAssistantReplayContent to drop transcript-only openclaw assistant entries and to substitute the sentinel for empty content arrays gated on stopReason === "error"; preserve original-array-reference fast path when nothing changes.
  • src/agents/session-file-repair.ts — extend repairSessionFileIfNeeded to detect-and-rewrite assistant entries with empty content and stopReason: "error"; add rewrittenAssistantMessages to RepairReport (additive, optional); warn message reports only the non-zero counters via a small buildRepairSummaryParts helper; combine drop and rewrite signals into the repaired flag.
  • src/agents/stream-message-shared.test.ts (new, co-located) — covers: never returns empty content; content carries only the sentinel and never echoes raw error text (CWE-200 guard); same sentinel when errorMessage is blank.
  • src/agents/pi-embedded-runner/replay-history.test.ts (new, co-located) — covers: empty content array becomes sentinel only when stopReason=error; silent-reply turn (stopReason:"stop", content:[]) is preserved untouched; legacy string-content wrapping still works; delivery-mirror and gateway-injected are filtered from replay; original array reference returned when nothing changes.
  • src/agents/session-file-repair.test.ts — adds: persisted assistant content:[] with stopReason:"error" is rewritten with backup; warn message reports only non-zero counters; drop+rewrite warn reports both phrasings; silent-reply on disk (stopReason:"stop", content:[]) is NOT rewritten; idempotent on a healed session.
• What did NOT change (scope boundary):
  • No persisted JSONL entry is dropped from disk. Mirror filtering happens only in the in-memory replay copy in normalizeAssistantReplayContent; UI/transcript history surfaces remain unaffected.
  • Silent-reply turns (stopReason:"stop", content:[]) are never rewritten — neither in replay nor on disk. The fix is exclusively scoped to the death-loop-causing error turns.
  • errorMessage field on the assistant message is preserved verbatim, including blank/whitespace input. Aisle's CWE-200 finding on errorMessage field redaction is intentionally out of scope: that field is the primary diagnostic signal UIs and operator log paths read to surface failure causes to users; redaction is a cross-cutting concern across all error paths and belongs in a separate hardening PR (or storage/transport-layer access control), not in this single message constructor.
  • RepairReport only gains an optional field (rewrittenAssistantMessages?: number); existing callers that ignore the return value or only read repaired/droppedLines are unaffected.
  • Aisle's CWE-400 (unbounded read) finding on repairSessionFileIfNeeded is pre-existing behavior, not introduced by this PR — the entire fs.readFile/fs.writeFile path is older than this fix. Streaming reads are independent hardening worth their own PR.
  • No changes to provider plugins, transcript-policy, image sanitization, tool-call repair, compaction, or session-file format/version.
  • The Ollama plugin has its own local buildStreamErrorAssistantMessage (extensions/ollama/src/stream.ts:289) intentionally left untouched here; it is provider-scoped and not in the Bedrock replay path.

Reproduction

1. Run an OpenClaw 2026.4.23 / 2026.4.25 gateway against AWS Bedrock Converse (e.g. anthropic.claude-3-haiku or any other Bedrock-hosted model).
2. From an embedded agent or Feishu channel, send a prompt that triggers a stream/provider error before the model emits any content block (network drop, transient ValidationException, or any error path that flows through buildStreamErrorAssistantMessage).
3. Observe the session JSONL: the new assistant entry has "content": [], "stopReason": "error".
4. Send any next user message in the same session. The Bedrock SDK rejects with "The content field in the Message object at messages.N is empty. Add a ContentBlock object to the content field and try again."
5. Restart the gateway and retry — the same validation error fires because the JSONL is unchanged.

After this fix:

• Step 3 produces "content": [{"type":"text","text": "[assistant turn failed before producing content]"}]. The persisted turn is replay-valid; the original error string remains available in errorMessage for clients/UIs.
• For sessions already poisoned by older builds, the repair pass on next load rewrites the empty-content error entry to a sentinel text block (with backup) and the session resumes; rerunning repair is a no-op.
• Replay drops openclaw delivery-mirror / gateway-injected mirror entries from the provider request, removing a separate class of strict-provider rejections.
• Silent-reply turns (stopReason:"stop", content:[]) are preserved unchanged on every code path.

Risk / Mitigation

• Risk: Changing buildStreamErrorAssistantMessage shape could affect downstream consumers that special-case content.length === 0.
  • Mitigation: All call sites surveyed (src/agents/openai-ws-stream.ts:1284, src/agents/pi-embedded-runner/run/attempt.stop-reason-recovery.ts:49,78,119) pass the value through to the agent loop's error event, which already iterates content defensively. The errorMessage field is preserved verbatim for clients that surface it separately.
• Risk: Filtering delivery-mirror / gateway-injected from replay could remove user-visible turns from the model's context.
  • Mitigation: These entries are transcript-only by design (see transcript.ts:264 isRedundantDeliveryMirror). Persisted JSONL is unchanged, so transcript history surfaces (WebChat, TUI, REST, SSE) keep showing them. The strict-provider ordering fallback at replay-history.ts:564-574 is unchanged and remains the safety net.
• Risk: Repair rewriting on-disk content could damage a session if the rewrite path crashes or rewrites a non-error turn.
  • Mitigation: Repair already writes a .bak-${pid}-${ts} backup before any rewrite and uses an atomic tmp → rename swap. The new rewrite path reuses the same write/backup machinery; failure paths still clean up the tmp file and return repaired:false with a reason. The stopReason === "error" gate ensures only failed turns are eligible — silent-reply turns are preserved as-is. Idempotency test ensures a second call is a no-op on a healed session.
• Risk: Adding rewrittenAssistantMessages to RepairReport could be a typed-API regression for external callers.
  • Mitigation: Field is optional (?:) and additive; all in-tree callers either ignore the return value or only read repaired/droppedLines/reason. No external consumers found in the repo.

Change Type (select all)

• Bug fix

Scope (select all touched areas)

• Gateway
• Agents (embedded runner / replay)
• Session storage (file repair)
• Provider replay (Bedrock Converse)

Linked Issue/PR

Fixes #71572

Changed files

• CHANGELOG.md (modified, +5/-0)
• docs/reference/transcript-hygiene.md (modified, +16/-5)
• src/agents/pi-embedded-runner/replay-history.test.ts (added, +115/-0)
• src/agents/pi-embedded-runner/replay-history.ts (modified, +68/-10)
• src/agents/session-file-repair.test.ts (modified, +143/-0)
• src/agents/session-file-repair.ts (modified, +80/-6)
• src/agents/stream-message-shared.test.ts (added, +44/-0)
• src/agents/stream-message-shared.ts (modified, +18/-1)
• src/infra/diagnostic-events.ts (modified, +11/-13)