openclaw - 💡(How to fix) Fix [Bug]: memory-core and session-logs read tail-only JSONL as complete history when context engines rotate transcripts [2 comments, 3 participants]

TLDR: When a context engine such as lossless-claw rotates an OpenClaw session transcript, the JSONL file becomes a tail-preserving operational checkpoint. Several OpenClaw consumers still treat that JSONL as the complete raw session history, so memory-core session indexing, dreaming session-corpus ingestion, and the bundled session-logs skill can silently miss compacted-away messages even though the raw messages still exist in the context engine's durable store.

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

OpenClaw has multiple session-history consumers that still assume agents/<agentId>/sessions/<session-id>.jsonl is a complete transcript, but plugin context engines can rewrite that file into a compact tail while preserving full history elsewhere.

Steps to reproduce

1. Run OpenClaw with memory-core active and a plugin context engine that owns compaction/history storage.
2. Enable or keep enabled transcript rotation in that context engine. With lossless-claw 0.9.4, autoRotateSessionFiles defaults to enabled, with startup/runtime mode rotate and a 2 MiB threshold.
3. Grow a session transcript above the rotation threshold or invoke the context engine's rotate command.
4. Observe that the session JSONL now contains only recent tail messages plus session state, while older raw messages remain in the context engine store.
5. Run any OpenClaw consumer that reads session history from JSONL: memory-core session indexing, dreaming session transcript ingestion, session markdown export, or the bundled session-logs skill.

Expected behavior

OpenClaw consumers that need complete raw session history should either:

• read from the active context engine / durable transcript provider when one owns session history, or
• know that the JSONL is a tail-only checkpoint and mark the source as incomplete instead of indexing/reporting it as complete history.

Actual behavior

Several consumers read *.jsonl directly and treat the result as complete session history:

• skills/session-logs/SKILL.md:33 says complete conversation history is stored in JSONL files.
• skills/session-logs/SKILL.md:41-46 describes <session-id>.jsonl as the full conversation transcript.
• packages/memory-host-sdk/src/host/session-files.ts:297-305 enumerates session transcript files from the sessions directory.
• packages/memory-host-sdk/src/host/session-files.ts:545-642 reads the JSONL file and flattens only the messages present in that file.
• extensions/memory-core/src/memory/manager-sync-ops.ts:841 and :884 use that JSONL flattening path for session indexing.
• extensions/memory-core/src/memory/qmd-manager.ts:2234-2241 uses the same path for session markdown export.
• extensions/memory-core/src/dreaming-phases.ts:759, :834, and :879-905 ingest session transcript signals from the JSONL-derived entry into memory/.dreams/session-corpus/.

With a rotating context engine, those code paths see only the retained JSONL tail.

OpenClaw version

Observed by code inspection on 2026.5.9-beta.1, local checkout v2026.5.9-beta.1-8-gff8a61b0e4.

Operating system

Ubuntu/Pop!_OS 24.04-class Linux, kernel 6.18.7-76061807-generic, x86_64.

Install method

Local source checkout / gateway development install.

Model

Model-independent storage/indexing behavior. The active analysis session used openai-codex/gpt-5.5.

Provider / routing chain

Model-provider independent. The affected path is local session storage, memory-core indexing, and context-engine storage coordination.

Additional provider/model setup details

NOT_ENOUGH_INFO

Logs, screenshots, and evidence

Relevant OpenClaw code/docs evidence:

• skills/session-logs/SKILL.md:33 and :41-46 still document JSONL as complete/full history.
• packages/memory-host-sdk/src/host/session-files.ts:297-305 lists JSONL transcript files.
• packages/memory-host-sdk/src/host/session-files.ts:545-642 reads and flattens only the JSONL contents.
• extensions/memory-core/src/memory/manager-sync-ops.ts:841 and :884 feed JSONL-derived session entries into memory-core session indexing.
• extensions/memory-core/src/memory/qmd-manager.ts:2234-2241 exports session markdown from the same JSONL-derived entries.
• extensions/memory-core/src/dreaming-phases.ts:759, :834, and :879-905 build dreaming session-corpus ingestion from those JSONL-derived entries.

Relevant context-engine evidence from lossless-claw 0.9.4:

• src/db/config.ts:27 defines the default auto-rotate threshold as 2 * 1024 * 1024 bytes.
• src/db/config.ts:540-557 defaults autoRotateSessionFiles.enabled to true, backups to false, and startup/runtime modes to rotate.
• src/engine.ts:8101-8192 rewrites the active transcript so only the latest raw tail plus current session settings remain on disk, while preserving the current conversation id and context-engine state.

Impact and severity

Affected users/systems/channels:

• Users running memory-core session indexing or dreaming together with a context engine that rotates/truncates JSONL transcripts.
• Agents or operators using the bundled session-logs skill to answer questions about older raw conversation content.
• Any future context engine that stores full history outside the legacy JSONL file.

Severity: Medium to high data coverage bug. The data is not necessarily lost, but OpenClaw consumers can silently miss it or tell operators to inspect the wrong source.

Frequency: Always after a session JSONL has been rotated before the affected consumer indexed/ingested the older messages.

Consequence:

• memory_search corpus=sessions can have incomplete coverage.
• Dreaming can miss compacted-away conversation material if it did not scan before rotation.
• Session-log analysis instructions can lead agents to falsely conclude older raw messages do not exist.
• forceFlushTranscriptBytes-style checks that rely on JSONL byte size can be affected by an external rotate threshold.

Additional information

Related OpenClaw issues/PRs found during triage:

• #79902 - umbrella for SQLite companion seams.
• #79903 - durable session lineage for SQLite companion stores.
• #79904 - cursored session transcript API.
• #79905 - typed projections and rebuild hooks.
• #79970 - session identity mapping work.
• #79972 - SQLite transcript frontier and delta API.
• #76249 - transcript-byte flush behavior around large JSONL files.
• #69451 - memory/session files loaded directly from JSONL.
• #72611, #77831, #71656, #75842 - existing dreaming/session-corpus quality and contamination issues.

This issue is distinct from the existing dreaming quality/filtering issues: the core problem here is source-of-truth mismatch under context-engine-managed transcript rotation.

Potential fix direction:

• Add a session-history provider abstraction that can ask the active context engine for durable raw messages by session key/conversation id.
• Make memory-core session indexing, dreaming session-corpus ingestion, and session markdown export use that provider when available.
• Update the bundled session-logs skill and session docs so JSONL is described as the active transcript/tail checkpoint, not an unconditional complete history source.
• If a full-history provider is unavailable, label JSONL-derived session hits as possibly incomplete when a context engine owns compaction/rotation.