[Bug]: memory-core Dreaming normalized recall artifacts silently deletes daily memory files (memory/YYYY-MM-DD.md)

Bug type

Behavior bug (silent data loss)

Summary

The memory-core Dreaming pipeline step normalized recall artifacts before dreaming (rewrote recall store) silently deletes daily memory log files (memory/YYYY-MM-DD.md) without logging the deletion. 46 daily memory files dating from March through May 18, 2026 vanished between the evening of May 19 and the morning of May 20, 2026.

Reproduction

This was discovered post-hoc in a production OpenClaw instance. The exact reproduction steps are unclear because the deletion is silent, but the timeline and evidence are:

1. Run OpenClaw with memory-core plugin and Dreaming enabled (cron: 0 3 * * *)
2. Let daily memory files (memory/YYYY-MM-DD.md) accumulate over days/weeks via Heartbeat
3. Dreaming cron runs at 03:00 each night
4. Daily memory files disappear silently — no delete/unlink event logged anywhere

Evidence

Files existed and were read by Dreaming

events.jsonl proves the files existed and were successfully read by the Dreaming process on subsequent nights:

2026-05-16T01:00:00Z  memory.recall.recorded  query=__dreaming_daily__:2026-05-15  path=memory/2026-05-15.md  resultCount=3
2026-05-17T01:00:03Z  memory.recall.recorded  query=__dreaming_daily__:2026-05-16  path=memory/2026-05-16.md  resultCount=1
2026-05-17T01:00:03Z  memory.recall.recorded  query=__dreaming_daily__:2026-05-15  path=memory/2026-05-15.md  resultCount=3
2026-05-18T01:00:00Z  memory.recall.recorded  query=__dreaming_daily__:2026-05-17  path=memory/2026-05-17.md  resultCount=2
2026-05-18T01:00:00Z  memory.recall.recorded  query=__dreaming_daily__:2026-05-16  path=memory/2026-05-16.md  resultCount=1

Files no longer exist

Only memory/2026-05-19.md (created 19:36 on May 19) and memory/2026-05-20.md (created 17:30 on May 20) survive. All files from 2026-05-15.md through 2026-05-18.md (and older dating back to March) have vanished.

short-term-recall.json still references deleted files

The recall store contains 14 references to memory/2026-05-15.md — dangling pointers to a file that no longer exists.

Dreaming log at time of disappearance

03:00:01 [plugins] memory-core: managed dreaming cron could not be reconciled (cron service unavailable)
03:00:03 [plugins] active-memory: ... start (light phase)
03:00:06 [plugins] memory-core: normalized recall artifacts before dreaming (rewrote recall store)   ← SUSPECT
03:00:07 [plugins] memory-core: dreaming promotion complete (candidates=10, applied=0)
03:00:34 [plugins] memory-core: dreaming cleanup scrubbed 1 stale session entry and archived 3 orphan transcripts

Related known bug: #64068

The normalized recall artifacts step previously caused recallCount to reset to 0 for all entries (bug #64068, fixed in PR #71695). This proves the step performs destructive rewrites of the recall store. The current bug appears to be another destructive side effect of the same normalization process — this time deleting the source daily memory files themselves.

Related: #68408

PR #68408 documents that Dreaming writes "managed blocks" directly into daily memory files (memory/YYYY-MM-DD.md) when using inline storage mode. The normalization step that rewrites these files could corrupt or empty them.

Related: #82928

Issue #82928 documents ENOENT errors when reading memory/2026-05-15.md — the same missing-file symptom, but treated as a logging issue rather than a data loss issue.

Not caused by user/agent action

Full audit of 143 session transcripts from May 19-20 found:

• 0 write tool calls on memory/2026-05-* paths
• 0 edit tool calls on memory/2026-05-* paths
• 0 exec commands containing rm, mv, unlink, or delete targeting memory/ paths
• Heartbeat cleanup pattern YYYY-MM-DD-*.md (excluding YYYY-MM-DD.md) — correctly protects daily files
• No delete/remove/unlink/purge events in events.jsonl

Not caused by Dreaming cleanup

The dreaming cleanup step only archives orphan session transcripts (.jsonl files renamed to .deleted.TIMESTAMP). It does not touch memory/ daily files. Source code confirms: fs.rename(transcriptPath, archivedPath) only operates on agents/main/sessions/*.jsonl.

Root Cause Hypothesis

The normalized recall artifacts before dreaming (rewrote recall store) step at 03:00:06 rewrites the short-term recall store and, based on PR #68408, also rewrites daily memory files to strip/manage Dreaming blocks. During this rewrite, daily memory files are either:

1. Truncated to empty and not restored — a write-then-rename pattern that failed to complete
2. Replaced with an empty file — the normalization read the file, stripped blocks, and wrote back empty content
3. Deleted as part of a cleanup of "stale" recall sources — the normalization may have classified older daily files as stale and removed them

This is functionally identical to bug #64068 (where normalization reset recallCount to 0) — destructive side effects of the normalization rewrite step.

Impact

• Data loss: 46 daily memory files permanently lost (March through May 18, 2026)
• Dangling references: short-term-recall.json still points to deleted files
• No audit trail: No delete/unlink event logged anywhere — silent data loss
• Recall degradation: Dreaming promotion candidates reference files that no longer exist

Environment

• OpenClaw version: 2026.5.19 (build date May 19, 2026)
• OS: Linux 6.8.0-107-generic x64 (Ubuntu)
• memory-core: bundled with OpenClaw 2026.5.19
• Dreaming: enabled, managed cron 0 3 * * * Europe/Berlin
• Dreaming storage mode: separate (writing to memory/dreaming/light/, rem/, deep/)

Suggested Fix

1. The normalizeRecallArtifacts function should never delete daily memory files — it should only modify the recall store
2. Any rewrite of daily memory files (for block stripping) must use atomic write (write to temp file, then rename) to prevent data loss on failure
3. Deletion of daily memory files should be explicitly logged in events.jsonl with the reason
4. Add a guard: daily memory files (memory/YYYY-MM-DD.md) should be read-only during Dreaming — Dreaming writes to memory/dreaming/, not to daily files
5. Consider adding a daily file integrity check: after normalization, verify all daily files referenced in the recall store still exist