openclaw - ✅(Solved) Fix Session corruption: prefill error cascades into provider cooldown + repair makes it worse [3 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
openclaw/openclaw#77228Fetched 2026-05-05 05:50:58
View on GitHub
Comments
1
Participants
2
Timeline
13
Reactions
2
Author
Timeline (top)
referenced ×9cross-referenced ×3commented ×1

Error Message

A single 400 This model does not support assistant message prefill error from the LLM provider cascades into a full provider cooldown, making the agent completely unresponsive. The session file auto-repair mechanism then corrupts the transcript further, requiring a manual new session.

Phase 1: Prefill error (10:59:27 CEST)

[agent/embedded] embedded run agent end: isError=true error=LLM request failed: provider rejected the request schema or tool payload. The format error puts the entire provider into cooldown: After repair, the error changes to:

  1. A prefill/format error on one request should NOT put the entire provider into long-term cooldown
  • Single format error → 42+ minutes of complete agent unresponsiveness The core issue appears to be that [assistant turn failed before producing content] placeholder messages create invalid message sequences. When combined with blank/dropped user messages, the conversation violates the provider's constraint that it must end with a user message. The cooldown mechanism then amplifies a single-request format error into a prolonged outage.

Root Cause

The core issue appears to be that [assistant turn failed before producing content] placeholder messages create invalid message sequences. When combined with blank/dropped user messages, the conversation violates the provider's constraint that it must end with a user message. The cooldown mechanism then amplifies a single-request format error into a prolonged outage.

Fix Action

Fixed

PR fix notes

PR #77280: fix(auth-profiles): exclude format rejections from profile cooldown

Description (problem / solution / changelog)

Summary

  • Problem: A single session-specific request-shape rejection from a provider takes down every other healthy session sharing the same auth profile, and when all configured profiles for a provider share the same fault, locks out the entire provider for the configured backoff window. The reporter in #77228 saw "Provider github-copilot is in cooldown (all profiles unavailable)" persist for 42+ minutes after a single 400 — "This model does not support assistant message prefill. The conversation must end with a user message." — caused by one corrupted session whose transcript ended in a stream-error placeholder assistant turn. Healthy sessions on the same profile were unable to make any provider call during the entire window.

  • Root Cause: src/agents/pi-embedded-runner/run/auth-profile-failure-policy.ts:5-14 resolves which FailoverReasons should be persisted as auth-profile health signals. Today it excludes policy === "local" (helper-local runs) and failoverReason === "timeout" (transport timeouts) — both annotated as "should not poison shared provider auth health". A format-classified failure (src/agents/pi-embedded-helpers/errors.ts:710-727: a 400/422 whose payload couldn't be reclassified as auth/billing/rate-limit/etc.) is also a non-poisonous signal — it means the provider rejected the request payload shape, which is per-session and per-transcript, not a profile-wide reliability problem — but it is currently passed through as failoverReason and reaches markAuthProfileFailure at src/agents/auth-profiles/usage.ts:649. Inside computeNextProfileUsageStats (usage.ts:539-642), format runs through calculateAuthProfileCooldownMs (usage.ts:363-372) just like rate_limit / overloaded, producing 30s → 60s → 5min capped backoff, and crucially without the model scoping that rate_limit gets (usage.ts:637: cooldownModel is only set for rate_limit). So one bad transcript in one session repeatedly hits the same 400, the post-cooldown retry hits the same 400, the cooldown re-lengthens to its 5-min cap, and back-to-back cycles produce the 42-min provider-wide outage observed in the report. Other sessions on the same profile, with valid transcripts, are blocked the entire time.

  • Fix: Add failoverReason === "format" to the existing exclusion list in resolveAuthProfileFailureReason. This is the single chokepoint through which markAuthProfileFailure learns about run-time failovers in pi-embedded-runner/run.ts (call sites at :1858, :2005, :2506, :2615 all funnel through resolveRunAuthProfileFailureReason at :872). When a session's transcript shape is rejected, the rejection still surfaces to the user via the existing FailoverError, the run still logs the failure, but the auth-profile cooldown machinery is no longer triggered. The bad session continues to fail — that is a separate, per-session repair concern explicitly tracked as the other two open items in #77228 — but other sessions on the same profile keep working, and the provider is no longer killed for everyone for the cooldown window.

  • What changed:

    • src/agents/pi-embedded-runner/run/auth-profile-failure-policy.ts — extend the existing policy === "local" / timeout exclusion guard to also cover failoverReason === "format". Comment expanded to document why a request-shape rejection is per-session, not profile-wide.
    • src/agents/pi-embedded-runner/run/auth-profile-failure-policy.test.ts — add a format-rejection case (with and without policy: "shared"), asserting the resolver returns null so markAuthProfileFailure is never called.
    • CHANGELOG.md — single Fixes line under Unreleased referencing the issue with non-closing Refs syntax.
  • What did NOT change (scope boundary):

    • No changes to the failure-reason classification (src/agents/pi-embedded-helpers/errors.ts); 400/422 schema rejections still classify as format and still surface to the user as a FailoverError.
    • No changes to markAuthProfileFailure / computeNextProfileUsageStats / calculateAuthProfileCooldownMs. Profile-cooldown semantics for legitimately profile-poisoning reasons (auth, auth_permanent, billing, rate_limit, overloaded, model_not_found, unknown, …) are untouched, so the existing behavior verified by src/agents/auth-profiles.markauthprofilefailure.test.ts is preserved.
    • No changes to the streaming-error placeholder (src/agents/stream-message-shared.ts:STREAM_ERROR_FALLBACK_TEXT) that ends up in the transcript and is the upstream root of the prefill 400. Stripping or rewriting it for prefill-strict providers is a distinct and provider-specific concern; the existing sentinel design was deliberately chosen for Bedrock Converse compatibility (stream-message-shared.ts:75-90).
    • No changes to the auto-repair routine that (per the report) rewrites the transcript with 935+ null-role entries on the second turn. That is a separate failure mode in the session-file repair path; deserves its own narrowly-scoped PR.

Reproduction

  1. Start an agent on a prefill-strict provider/model (e.g. github-copilot/claude-opus-4.6 in the report; any provider that rejects "conversation must end with a user message" works) and configure two or more sessions on the same auth profile (e.g. session A and session B).
  2. In session A, drive the transcript into the failure shape: an assistant turn that errored out before producing content gets persisted as the final entry (STREAM_ERROR_FALLBACK_TEXT) followed by a blank/empty user turn that itself fails to produce a usable user message — the provider then sees a payload ending in an assistant message.
  3. Send a message in session A. Without this fix: provider returns 400 This model does not support assistant message prefill ... must end with a user message., the failure classifies as format, markAuthProfileFailure fires for the profile, the auth profile enters cooldown.
  4. Send a message in session B (different session, same auth profile, valid transcript). Without this fix: rejected with "Provider github-copilot is in cooldown (all profiles unavailable)" — even though session B's request was perfectly valid. With this fix: session B succeeds; only session A keeps failing on its own bad transcript.
  5. Cycle session A's retries past the cooldown expiry. Without this fix: the post-cooldown retry hits the same 400 and the cooldown re-extends to 5 minutes; back-to-back cycles reproduce the 42-minute outage in the report. With this fix: session A still fails per turn, but no profile cooldown is recorded, so no other session is affected and no extension cycle is created.

Risk / Mitigation

  • Risk 1 — losing telemetry on schema problems: Could excluding format from profile cooldown make schema problems invisible? No. The format failure still surfaces to the user via FailoverError, still logs through the existing run-warn path, and still appears in failureCounts if and when the resolver does return non-null on other code paths. We are removing only the profile-cooldown side effect, not the diagnostic surface. Mitigation: covered by the existing markAuthProfileFailure test suite continuing to pass — those tests do not assert anything about format, since format was not previously a documented profile-poisoning reason; the reporter's symptom shows it was de-facto poisoning behavior, not by design.
  • Risk 2 — masking a real provider-side schema fault: Could a provider's own bug (where everyone's request shape is rejected legitimately) hide behind this exclusion? Unlikely to be material: such an outage would manifest as format errors across many independent sessions and many profiles, and the operator would still see them as FailoverErrors in logs and per-turn UI. Profile-cooldown was never the right mitigation for a provider-side schema bug — failover/fallback chain configuration is. Mitigation: behavior matches the existing precedent for timeout exclusion (transport timeouts also surface to logs and UI but do not poison auth health).
  • Risk 3 — extending the exclusion list: Could other reasons need similar treatment? The two other "session-shape" reasons surfacing in the codebase are replay_invalid and schema (in the ProviderRuntimeFailureKind taxonomy at errors.ts:258-277). Both already collapse into format through classifyFailoverClassificationFromMessage. They are covered by this single change. Mitigation: the test asserts resolveAuthProfileFailureReason returns null for format regardless of policy, locking the contract.
  • Risk 4 — minimal blast radius: The change is one file in production code, four lines of effective logic edit. The chokepoint nature of resolveAuthProfileFailureReason means there is exactly one path to audit.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Agents (auth-profile failure policy)
  • Tests (resolver unit coverage)
  • Changelog (Unreleased Fixes entry)

Linked Issue/PR

Refs #77228 — addresses the cascading profile cooldown amplifier only. The two other items in the same issue (the stream-error placeholder leaving transcripts ending in assistant, and the auto-repair amplification that fills the JSONL with 935+ null-role entries) are independent failure modes; they remain open and out of scope here so they can be tracked and shipped separately.

Changed files

  • CHANGELOG.md (modified, +1/-0)
  • src/agents/pi-embedded-runner/run/auth-profile-failure-policy.test.ts (modified, +19/-0)
  • src/agents/pi-embedded-runner/run/auth-profile-failure-policy.ts (modified, +15/-2)

PR #77287: fix(replay-history): drop trailing stream-error placeholder before pr…

Description (problem / solution / changelog)

Summary

  • Problem: A session whose last assistant turn errored before producing content — persisted as { role: "assistant", stopReason: "error", content: [] } (or, after the offline session-file repair pass, with content: [{ type: "text", text: "[assistant turn failed before producing content]" }]) — gets sent to the next provider call as the trailing message. Prefill-strict providers (the report's repro case is github-copilot/claude-opus-4.6 in #77228, but the same pattern hits any model that requires the conversation to end with a user message) reject the call with 400 This model does not support assistant message prefill. The conversation must end with a user message.. The retry path keeps re-sending the same shape and the user sees a session that never recovers.

  • Root Cause: src/agents/pi-embedded-runner/replay-history.ts:327-399 (normalizeAssistantReplayContent) deliberately rewrites empty content: [] assistant turns whose stopReason === "error" (or isZeroUsageEmptyStopAssistantTurn(message) — the stopReason: "stop" zero-usage shape) into a non-empty single-text-block carrying STREAM_ERROR_FALLBACK_TEXT (src/agents/stream-message-shared.ts:90). The doc comment at replay-history.ts:370-396 explicitly states the goal: AWS Bedrock Converse rejects assistant messages with no ContentBlock, so the rewrite is needed to keep replay valid for Bedrock. That goal is correct for non-trailing error turns. The same rewrite is wrong when applied to the trailing error turn — Bedrock's contract is satisfied by the next user message that immediately follows in normal flow, but when the error turn is the last entry there is no following user message and the rewrite produces a request whose final message is assistant. src/agents/session-file-repair.ts:65 further persists the same sentinel content to disk, so on the next turn the loaded transcript already carries the sentinel as a non-empty trailing assistant block, which the existing Array.isArray(replayContent) && replayContent.length === 0 branch does not match (it now skips through unchanged) and the trailing-prefill 400 reproduces.

  • Fix: After the existing sentinel-rewrite loop in normalizeAssistantReplayContent finishes, walk back from the tail and drop trailing assistant turns that match the dropable-trailing predicate: an assistant turn whose content is either (a) empty and stopReason === "error" or isZeroUsageEmptyStopAssistantTurn(message), or (b) the single-text-block sentinel shape [{ type: "text", text: STREAM_ERROR_FALLBACK_TEXT }]. Both shapes carry zero usage and no model output; dropping is lossless. The next provider request now ends in the last user (or whatever non-droppable turn was before the synthetic tail), which Bedrock accepts (no empty assistant ContentBlock) and prefill-strict providers also accept (last message is user). The trim sits after the rewrite loop so it transparently catches the persisted-sentinel-on-disk shape as well as the in-memory rewrite shape, with one helper.

  • What changed:

    • src/agents/pi-embedded-runner/replay-history.ts — extend normalizeAssistantReplayContent with a post-loop tail trim plus two pure helpers (isReplayDroppableTrailingAssistant, isStreamErrorSentinelContent). Comment block records why dropping is lossless and which provider classes are protected.
    • src/agents/pi-embedded-runner/replay-history.test.ts — relocate two existing tests that locked in the buggy "rewrite trailing error to sentinel" behavior so they exercise the mid-turn sentinel rewrite (with a follow-up user message); add four new tests covering trailing drop for empty-error, trailing drop for zero-usage-empty-stop, trailing drop for persisted sentinel content, multi-turn trailing drop, and boundary preservation for real assistant content + non-error empty toolUse/length turns.
    • CHANGELOG.md — single Fixes line under Unreleased referencing the issue with non-closing Refs syntax.
  • What did NOT change (scope boundary):

    • Behavior is unchanged for non-trailing error/zero-usage-empty-stop turns: they still get rewritten with the sentinel, preserving Bedrock Converse compatibility verified by the relocated tests and by run.empty-error-retry.test.ts's "Clean stop with no output is a legitimate silent reply, not a crash" boundary.
    • No changes to STREAM_ERROR_FALLBACK_TEXT itself, to buildStreamErrorAssistantMessage, to session-file-repair.ts's on-disk write of the sentinel, or to any provider-specific replay path. The fix is provider-agnostic and lives at the single normalization chokepoint that all replay flows funnel through (attempt.ts:546, attempt.ts:3019, and replay-history.ts:628).
    • No changes to the cascading auth-profile cooldown that this trailing prefill 400 used to trigger; that amplifier is tracked separately under the same issue and is being addressed in a sibling PR.
    • No changes to the auto-repair routine that, on the second turn after the prefill 400, fills the JSONL with null-role entries; that is a third independent failure mode in the same issue and gets its own PR.

Reproduction

  1. Drive an agent on a prefill-strict provider/model (the report uses github-copilot/claude-opus-4.6) into the failure shape: a tool-call assistant turn aborts before producing content. The session manager persists the entry as { role: "assistant", stopReason: "error", content: [] } and (after the offline session-file repair pass that's documented at session-file-repair.ts:65) may rewrite it to the [{ type: "text", text: "[assistant turn failed before producing content]" }] sentinel shape on disk.
  2. Without sending another user turn (e.g. an internal retry, a heartbeat replay, or a continuation that loads the persisted transcript) trigger another provider call.
  3. Without this fix: provider returns 400 ... The conversation must end with a user message., the failover reason classifies as format, the session is stuck. The same shape continues to be re-sent on every retry.
  4. With this fix: normalizeAssistantReplayContent drops the trailing synthetic assistant turn before the request leaves OpenClaw. The provider sees a request ending in the last real user turn (or whatever non-droppable entry precedes the synthetic tail). Bedrock still accepts it (no empty ContentBlock), prefill-strict providers also accept it (last message is user), and the agent generates a fresh assistant reply.

Risk / Mitigation

  • Risk 1 — losing a legitimate trailing assistant turn: Could a turn that the user actually wants the model to "continue" be dropped? No. The drop predicate matches only:

    • empty content: [] plus stopReason: "error" (synthetic, zero-usage, the failed turn itself)
    • empty content: [] plus the isZeroUsageEmptyStopAssistantTurn shape (already documented in the existing rewrite branch as a synthesized failure surface — see replay-history.ts:387-388 and the boundary test run.empty-error-retry.test.ts referenced in the existing comment)
    • non-empty content that is exactly [{ type: "text", text: STREAM_ERROR_FALLBACK_TEXT }] (synthetic sentinel)

    Anything else — real text content, real usage, stopReason: "toolUse" or "length" — is preserved. Test cases lock both directions of this predicate. Mitigation: explicit "does not drop a trailing assistant turn that has real content" and "does not drop a trailing assistant turn with non-error empty content (toolUse / length)" tests; the run.empty-error-retry.test.ts silent-reply boundary continues to pass because nonzero-usage stop turns do not match isZeroUsageEmptyStopAssistantTurn (existing boundary preserved).

  • Risk 2 — breaking Bedrock Converse compatibility: Could Bedrock now reject calls that used to work because we removed the sentinel rewrite? No. The pre-existing rewrite branch is unchanged for non-trailing error turns — those still get the sentinel inserted. The trim only fires for trailing turns, and a trailing assistant message with empty content is exactly the case Bedrock would have rejected anyway; dropping it produces a request whose last message is user, which Bedrock accepts. Mitigation: relocated tests for "converts mid-turn assistant content: [] to a non-empty sentinel text block when stopReason is error" and "converts mid-turn zero-usage empty stop turns to a replay sentinel" continue to assert the rewrite path explicitly.

  • Risk 3 — provider-specific divergence: The fix is provider-agnostic. Could it diverge somewhere downstream? No. normalizeAssistantReplayContent is the single chokepoint upstream of all provider runtime calls (attempt.ts:546 normalizeMessagesForLlmBoundary, attempt.ts:3019 normalizedReplayMessages, replay-history.ts:628). After it returns, downstream provider plugins do their own provider-specific massaging on the message list, but they receive a list whose tail is already user-shaped, which is the universal contract. Mitigation: existing sanitizeProviderReplayHistoryWithPlugin and validateProviderReplayTurnsWithPlugin flows are downstream of this normalization; they continue to receive a structurally-valid list.

  • Risk 4 — repair amplification: This PR does not stop the auto-repair routine from filling the JSONL with null-role entries when triggered by a different invalid sequence. That is a separate root cause and gets its own PR. The trim here removes the trigger condition for the prefill 400 specifically. Mitigation: the changelog entry and the Refs reference are explicit about scope.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Agents (replay history normalization)
  • Tests (replay-history.test.ts)
  • Changelog (Unreleased Fixes entry)

Linked Issue/PR

Refs #77228 — addresses the trailing prefill-placeholder root cause only. The cascading auth-profile cooldown amplifier (a single format 400 currently poisons the whole auth profile across sessions) and the auto-repair amplifier (the routine writes 935+ null-role entries when reconciling an invalid sequence) are independent failure modes in the same issue and are out of scope here so each can ship as its own narrowly-scoped PR.

Changed files

  • CHANGELOG.md (modified, +1/-0)
  • src/agents/pi-embedded-runner/replay-history.test.ts (modified, +120/-4)
  • src/agents/pi-embedded-runner/replay-history.ts (modified, +67/-0)

PR #77288: fix(session-file-repair): drop null-role message entries instead of p…

Description (problem / solution / changelog)

Summary

  • Problem: After the on-disk session-file repair pass runs against a transcript that already accumulated null-role JSONL corruption, the post-repair file (and its .bak-* backup) still contains the corrupt entries — the report in #77228 saw "935+ entries with null roles (complete structural JSONL corruption)" surviving the repair, with the next provider call failing on a now-different shape error (400 messages: at least one message is required once enough valid entries had been displaced). The reporter's debug line [session-init] session file repair: rewrote 1 assistant message(s), dropped 1 blank user message(s) confirms the repair pass did run on the file but left the null-role entries in place.

  • Root Cause: src/agents/session-file-repair.ts:repairSessionFileIfNeeded (:160-295) iterates persisted JSONL lines, applies two narrow rewrites (isAssistantEntryWithEmptyContent rewrite at :36-58/60-68, repairUserEntryWithBlankTextContent at :75-136), and otherwise pushes the parsed entry to entries[] as-is (:219). The two rewrites only fire on a non-null assistant/user role; a type:"message" envelope whose message.role is null, undefined, or a blank string does not match either rewrite predicate and falls through to the as-is push. The repair then atomically rewrites the file to disk, persisting the null-role entries unchanged. There is no provider router downstream that can do anything useful with a role: null message — every classifier in src/agents/session-transcript-repair.ts, src/agents/cli-runner/session-history.ts, and the provider-runtime replay sanitizers branches on message.role ("user"/"assistant"/"toolResult"/"compactionSummary"); a null-role entry is unconditionally dead weight on the wire.

  • Fix: Add a structural-validity predicate isStructurallyInvalidMessageEntry and consult it inside the parse loop before the rewrite predicates run. When a type:"message" envelope has no message object, has a non-object message, or its message.role is missing, null, or a whitespace-only string, treat the line the same way the existing JSON.parse failure branch treats an unparseable line: increment droppedLines and skip the push to entries[]. The repair output is therefore guaranteed not to contain a null-role type:"message" entry, regardless of what the input file accumulated. Entries of other envelope types (type:"session", type:"summary", type:"custom", anything not "message") are not subject to this check and pass through unchanged — only the message envelope contract is enforced.

  • What changed:

    • src/agents/session-file-repair.ts — add the isStructurallyInvalidMessageEntry helper and wire it into the parse loop ahead of isAssistantEntryWithEmptyContent so the drop decision is made before either rewrite predicate runs. Doc comment records why preservation through repair is wrong and which downstream code branches on message.role.
    • src/agents/session-file-repair.test.ts — add three new tests: (1) drops role: null / missing-role / blank-role type:"message" entries and counts them toward droppedLines, asserting the post-repair file no longer contains "role":null; (2) drops type:"message" envelopes whose message field is missing or non-object; (3) preserves non-"message" envelope types (type:"summary", type:"custom") untouched so the structural check does not over-reach.
    • CHANGELOG.md — single Fixes line under Unreleased referencing the issue with non-closing Refs syntax.
  • What did NOT change (scope boundary):

    • No changes to the existing rewrite predicates (isAssistantEntryWithEmptyContent, repairUserEntryWithBlankTextContent) or to BLANK_USER_FALLBACK_TEXT / STREAM_ERROR_FALLBACK_TEXT. Existing test cases continue to assert the same rewrite behavior — the new structural check runs before those predicates and on different inputs (null-role), so it cannot change the verdict for any input the existing tests exercise.
    • No changes to file-write / atomic-rename / backup logic (:246-258). The repair still writes a .bak-* backup of the original and atomically renames the cleaned file in place.
    • No source-of-null-role investigation or fix. The producer of the null-role entries (suspected to be a write-side path or an external mutation; the reporter observed null roles in both the active .jsonl and the older .reset.<iso> archive, suggesting the corruption pre-dates the repair pass) is out of scope for this PR. Defending the repair output from preserving the corruption is the narrowly-scoped, surgically defensible fix; locating the producer can land in a separate follow-up.
    • No changes to the cascading auth-profile cooldown that this corrupted session can trigger upstream of the repair pass, and no changes to the trailing prefill-placeholder shape that originally tripped the provider 400. Each is a separate failure mode in the same issue and gets its own narrowly-scoped PR.

Reproduction

  1. Construct (or copy from a real-world reproducer) a session JSONL whose tail contains one or more type:"message" entries shaped like {"type":"message","id":"...","timestamp":"...","message":{"role":null,"content":"..."}}. The report's reproducer happens organically after a series of failures cascading off a prefill 400; for a deterministic reproducer in unit-test space, write the entries directly with JSON.stringify.
  2. Add a session header line as the first entry ({"type":"session","id":"...","version":7,"timestamp":"...","cwd":"..."}) and a normal role:"user" message somewhere in the body so the repair pass cannot bail on the entries.length === 0 / invalid session header early-returns.
  3. Call repairSessionFileIfNeeded({ sessionFile }).
  4. Without this fix: the post-repair file still contains every null-role entry, just re-serialized through JSON.stringify. result.droppedLines reflects only unparseable JSON lines; the null-role count is hidden, and the cleaned transcript is structurally identical (modulo the rewrite passes) to the original — including the corruption.
  5. With this fix: each null-role entry counts toward result.droppedLines and is absent from the post-repair file. The structural check fires before the rewrite predicates, so the existing rewrite paths are unaffected for any non-null-role input.

Risk / Mitigation

  • Risk 1 — over-broad role validation: Could the new check reject a legitimate envelope type whose top-level shape is also type:"message" but whose role lives somewhere else? No. The check only fires when entry.type === "message". Other envelope types ("session", "summary", "custom", "model-snapshot", etc.) skip the check entirely. The "preserves non-message envelope types" test locks this boundary, including a type:"custom" model-snapshot entry shape that is observed in production replay history. Mitigation: explicit positive test for type:"summary" and type:"custom"; explicit negative tests for role: null, missing role, and blank-string role; result.droppedLines is incremented in lock-step so any drop is observable through the existing repair report and debug log.
  • Risk 2 — losing recoverable data: Could a null-role entry have carried information worth keeping? No: every provider router branches on message.role and would silently drop the entry on the wire anyway. The only "value" of preserving null-role entries is keeping the original byte payload around for debugging — and the existing .bak-* backup written by the repair pass already preserves that byte payload verbatim before any drop decisions are made. Mitigation: the existing backup-then-rewrite atomic-rename flow at :246-258 is unchanged; original bytes remain available off-line under the .bak-* file for postmortem.
  • Risk 3 — interaction with the existing rewrite predicates: Could the new check race with the empty-assistant-error rewrite or the blank-user rewrite and produce a different verdict? No. The new check fires first in the parse loop. A null-role entry never reaches isAssistantEntryWithEmptyContent (which bails immediately because message.role !== "assistant") nor repairUserEntryWithBlankTextContent (which is gated on message.role === "user" at the call site :206). For any non-null-role entry, the new check is a no-op and the existing predicates run unchanged. Mitigation: existing rewrite tests at :101-281 continue to pass without modification.
  • Risk 4 — entries.length === 0 after the drop: If the repair drops all message entries (only the session header survives), the existing entries.length === 0 check at :225-227 already returns repaired: false with reason: "empty session file" — this is correct behavior and matches the existing contract: an empty transcript is reported, the original file is left untouched. The fix does not introduce a new "successfully wrote an empty transcript" failure mode. Mitigation: covered by existing assertions on the early-return branches.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Agents (session-file-repair on-disk pass)
  • Tests (session-file-repair.test.ts)
  • Changelog (Unreleased Fixes entry)

Linked Issue/PR

Refs #77228 — addresses the auto-repair amplifier only (the repair pass no longer preserves null-role JSONL corruption from the input). The cascading auth-profile cooldown amplifier (a single format 400 currently poisons the whole auth profile across sessions) and the trailing prefill-placeholder root cause (the stream-error sentinel ends up as the trailing message and triggers a prefill-strict 400) are independent failure modes in the same issue and are out of scope here so each can ship as its own narrowly-scoped PR.

Changed files

  • CHANGELOG.md (modified, +1/-0)
  • src/agents/session-file-repair.test.ts (modified, +119/-0)
  • src/agents/session-file-repair.ts (modified, +34/-0)

Code Example

[agent/embedded] embedded run agent end: isError=true error=LLM request failed: provider rejected the request schema or tool payload.
rawError=400 This model does not support assistant message prefill. The conversation must end with a user message.

---

[model-fallback/decision] decision=candidate_failed reason=format
[model-fallback/decision] decision=skip_candidate reason=format detail=Provider github-copilot is in cooldown (all profiles unavailable)
Embedded agent failed before reply: All models failed (1): github-copilot/claude-opus-4.6: Provider github-copilot is in cooldown

---

[session-init] session file repair: rewrote 1 assistant message(s), dropped 1 blank user message(s)

---

rawError=400 messages: at least one message is required
RAW_BUFFERClick to expand / collapse

Bug Summary

A single 400 This model does not support assistant message prefill error from the LLM provider cascades into a full provider cooldown, making the agent completely unresponsive. The session file auto-repair mechanism then corrupts the transcript further, requiring a manual new session.

Environment

  • OpenClaw 4.29, Linux 6.17.0-1011-azure (x64)
  • Provider: github-copilot / claude-opus-4.6
  • Channel: WhatsApp

Steps to Reproduce

  1. Agent is mid-conversation with active tool calls
  2. A tool call fails, producing an [assistant turn failed before producing content] placeholder
  3. A blank/empty user message follows (possibly from WhatsApp inbound during the failed turn)
  4. This creates an invalid message sequence where the conversation ends with an assistant message (the failed placeholder) rather than a user message

What Happens

Phase 1: Prefill error (10:59:27 CEST)

[agent/embedded] embedded run agent end: isError=true error=LLM request failed: provider rejected the request schema or tool payload.
rawError=400 This model does not support assistant message prefill. The conversation must end with a user message.

Phase 2: Provider cooldown cascade

The format error puts the entire provider into cooldown:

[model-fallback/decision] decision=candidate_failed reason=format
[model-fallback/decision] decision=skip_candidate reason=format detail=Provider github-copilot is in cooldown (all profiles unavailable)
Embedded agent failed before reply: All models failed (1): github-copilot/claude-opus-4.6: Provider github-copilot is in cooldown

Every subsequent user message for the next ~42 minutes hits the same cooldown wall. The agent cannot respond at all.

Phase 3: Session repair makes it worse (11:41:15)

[session-init] session file repair: rewrote 1 assistant message(s), dropped 1 blank user message(s)

After repair, the error changes to:

rawError=400 messages: at least one message is required

The transcript is now fully corrupted — both .reset and .bak files contain 935+ entries with null roles (complete structural JSONL corruption).

Expected Behavior

  1. A prefill/format error on one request should NOT put the entire provider into long-term cooldown
  2. The cooldown should either be very short or only apply to that specific session, not block all sessions
  3. Session file repair should not produce a worse state than what it started with
  4. If a transcript is irrecoverable, the system should auto-create a fresh session rather than repeatedly failing

Actual Behavior

  • Single format error → 42+ minutes of complete agent unresponsiveness
  • Auto-repair corrupts the transcript further
  • User had to manually start a new session

Root Cause Analysis

The core issue appears to be that [assistant turn failed before producing content] placeholder messages create invalid message sequences. When combined with blank/dropped user messages, the conversation violates the provider's constraint that it must end with a user message. The cooldown mechanism then amplifies a single-request format error into a prolonged outage.

Suggested Fixes

  1. Short cooldown for format errors: Format errors are session-specific, not provider-wide issues. Cooldown should be seconds, not minutes, and scoped to the session.
  2. Safer transcript repair: Validate the repaired transcript before committing. If repair produces an invalid state, fall back to creating a fresh session.
  3. Handle [assistant turn failed] placeholders: These should be cleaned from the transcript before sending to the provider, or replaced with a valid assistant message.
  4. Auto-recovery: If a session is stuck in repeated format errors, offer to reset it automatically rather than failing silently for 40+ minutes.

Log References

  • Session ID: 229feaa0-2692-401c-a828-66939bf80acc
  • Failed run IDs: 66a98a07, 3ef84b65 (and several in between)
  • Corrupted files: .jsonl.reset.2026-05-04T09-42-09.905Z, .jsonl.bak-59249-*

extent analysis

TL;DR

Implement a short cooldown for format errors, scoped to the session, and improve transcript repair to prevent corruption and auto-create a fresh session when necessary.

Guidance

  • Validate the cooldown mechanism to ensure it only applies to the specific session that encountered the format error, rather than the entire provider.
  • Modify the transcript repair process to validate the repaired transcript before committing changes and fall back to creating a fresh session if the repair produces an invalid state.
  • Consider handling [assistant turn failed] placeholders by cleaning them from the transcript or replacing them with a valid assistant message before sending to the provider.
  • Implement auto-recovery for sessions stuck in repeated format errors, offering to reset the session automatically after a certain threshold.

Example

No specific code example is provided due to the lack of explicit code in the issue, but the suggested fixes imply modifications to the cooldown mechanism, transcript repair logic, and error handling for [assistant turn failed] placeholders.

Notes

The provided analysis and suggested fixes assume that the cooldown mechanism and transcript repair process are modifiable. If these are part of a third-party library or service, alternative workarounds may be necessary.

Recommendation

Apply the suggested fixes, particularly implementing a short cooldown for format errors and improving transcript repair, as these address the root cause of the issue and can prevent similar outages in the future.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

openclaw - ✅(Solved) Fix Session corruption: prefill error cascades into provider cooldown + repair makes it worse [3 pull requests, 1 comments, 2 participants]