PR #74088: fix(memory-core): cap MEMORY.md size during dreaming promotions to pr…

• Repository: openclaw/openclaw
• Author: YB0y
• State: open | merged: False
• Link: https://github.com/openclaw/openclaw/pull/74088

Description (problem / solution / changelog)

fix(memory-core): cap MEMORY.md size during dreaming promotions to prevent unbounded growth (#73691)

Dreaming's deep-phase promotion path appends to ~/.openclaw/workspace-<agent>/MEMORY.md without a durable size budget. After weeks of use, the file grows past the bootstrap injection cap (~12KB/file), at which point bootstrap silently truncates promoted memory and, in the issue body's report, session writes can hit lock timeouts that wedge the gateway.

Adds a bounded compaction step in applyShortTermPromotions: before each write, drop the OLDEST auto-promoted sections (date-ordered) until existing + new section fits within memoryFileMaxChars (default 10,000 chars, safely below the 12KB bootstrap cap). User-authored content — anything that is not a ## Promoted From Short-Term Memory (DATE) section — is preserved unconditionally; only dreaming-owned sections are eligible for compaction.

Verified:

• pnpm install --frozen-lockfile
• pnpm test extensions/memory-core/src/memory-budget.test.ts extensions/memory-core/src/short-term-promotion.test.ts
• pnpm exec oxfmt --check --threads=1 extensions/memory-core/src/memory-budget.ts extensions/memory-core/src/memory-budget.test.ts extensions/memory-core/src/short-term-promotion.ts extensions/memory-core/src/short-term-promotion.test.ts CHANGELOG.md
• pnpm check:changed
• pnpm tsgo:core
• git diff --check

Closes #73691

Summary

• Problem: applyShortTermPromotions writes ${existingMemory}${section} with no size budget, compaction, or archive (extensions/memory-core/src/short-term-promotion.ts:1648-1652 on prior main). Bootstrap injection is already capped at 12KB/file, but the disk-side file keeps growing across daily sweeps; once oversized, bootstrap silently truncates promoted material and (per the issue body) session writes can hit lock timeouts that wedge the gateway.
• Why it matters: The reported failure mode is a hard gateway freeze for users with dreaming.enabled: true, requiring manual MEMORY.md truncation + hook disablement to recover. The bot review on #73691 explicitly cautioned against naive truncation: "A naive fix that truncates MEMORY.md could destroy useful durable memory. The fix should preserve older material through compaction, archive/search, or a structured replacement."
• What changed: New extensions/memory-core/src/memory-budget.ts with compactMemoryForBudget(...). Wired into applyShortTermPromotions so the budget check runs before each write. Compaction drops OLDEST auto-promotion sections by date until under budget; user-authored content is never touched. New optional memoryFileMaxChars parameter on ApplyShortTermPromotionsOptions (default DEFAULT_MEMORY_FILE_MAX_CHARS = 10_000, pass 0 to disable). Two new fields on the result: compactedSections: number, compactedDates: string[].
• What did NOT change (scope boundary): No public SDK contract change (MemoryHostEvent union is untouched — no new event variant). No format change to MEMORY.md sections. No archive layer (out of scope; this PR is the smallest fix that addresses the bot's anti-naive-truncation guidance). No doctor visibility (deferrable follow-up). The legacy ${existingMemory}${section} write path stays — only the existingMemory upstream of it is now compacted when over budget.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes #73691
• Related #
• This PR fixes a bug or regression

Root Cause (if applicable)

• Root cause: applyShortTermPromotions in extensions/memory-core/src/short-term-promotion.ts reads existing MEMORY.md and writes back ${existingMemory}${newSection} for every promotion run. There is no pre-write size check, compaction, or archive decision, so MEMORY.md is monotonically increasing.
• Missing detection / guardrail: No regression test asserts a durable MEMORY.md byte/character budget. extensions/memory-core/src/short-term-promotion.test.ts covers append behavior and duplicate-marker dedupe but not bounded growth.
• Contributing context (if known): Bootstrap injection at src/agents/pi-embedded-helpers/bootstrap.ts:87 already caps prompt-side reads at 12,000 chars/file and 60,000 total — that mitigates the immediate prompt-overflow path but not the disk-side growth, which is what the issue reports about session lock timeouts.

Regression Test Plan (if applicable)

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file:
  • extensions/memory-core/src/memory-budget.test.ts (new) — pure helper unit tests for the compaction algorithm.
  • extensions/memory-core/src/short-term-promotion.test.ts — new MEMORY.md budget compaction (#73691) describe block with two integration smokes proving the helper is wired through applyShortTermPromotions.
• Scenario the test should lock in: After repeated promotion writes, MEMORY.md does not grow past the budget. Older auto-promoted sections are dropped in date order; user-authored markdown survives.
• Why this is the smallest reliable guardrail: A pure-helper unit test covers every branch of the compaction algorithm without the apparatus of recordShortTermRecalls + ranking + rehydration. One integration smoke proves the wiring. This matches the repo's testing guidance: pure helper / contract unit tests at the boundary, plus one integration smoke per seam.
• Existing test that already covers this (if any): None. short-term-promotion.test.ts covers the existing append path; no test asserted a disk budget.
• If no new test is added, why not: N/A — 11 new helper unit tests + 2 new integration smokes added.

User-visible / Behavior Changes

For users with dreaming.enabled: true, MEMORY.md will now be capped at 10,000 chars after dreaming promotion writes. Older auto-promoted sections are dropped (oldest first by date) before the new section is appended. No user-authored content is touched — only sections matching the ## Promoted From Short-Term Memory (DATE) heading. Users who had already let MEMORY.md grow past the cap will see one-time compaction on the next dreaming promotion. The new optional memoryFileMaxChars knob lets operators raise/lower the budget or set 0 to disable. No config schema change; the option is a runtime parameter on applyShortTermPromotions.

Diagram (if applicable)

Before:
applyShortTermPromotions():
  existingMemory = read MEMORY.md      (e.g. 50KB after weeks of dreaming)
  section        = build new promotion (e.g. 1KB)
  write(existingMemory + section)      → 51KB  → bootstrap silently truncates → eventual gateway freeze

After:
applyShortTermPromotions(memoryFileMaxChars = 10_000):
  existingMemory = read MEMORY.md      (e.g. 50KB)
  section        = build new promotion (e.g. 1KB)
  if existingMemory.length + section.length > budget:
    drop oldest "## Promoted From Short-Term Memory (DATE)" sections by date
    until existingMemory.length + section.length <= budget
    (preserve any user-authored content unconditionally)
  write(compactedExisting + section)   → bounded under 10KB

Security Impact (required)

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? No
• Command/tool execution surface changed? No
• Data access scope changed? No
• If any Yes, explain risk + mitigation: N/A. The change only affects on-disk MEMORY.md content under the user's own workspace directory, drops only auto-emitted dreaming sections, and exposes no new I/O or trust surface.

Repro + Verification

Environment

• OS: Linux x86_64 (Ubuntu noble)
• Runtime/container: Node 22.22.2 via nvm; pnpm 10.33.0
• Model/provider: any (the bug is in dreaming's write path, not model-specific)
• Integration/channel (if any): any agent with dreaming enabled
• Relevant config (redacted): plugins.entries.memory-core.config.dreaming.enabled: true with regular dreaming sweeps

Steps

1. Configure an agent with dreaming.enabled: true and run for several weeks (or simulate by running the deep-phase sweep many times).
2. Inspect ~/.openclaw/workspace-<agent>/MEMORY.md.
3. Without this fix: MEMORY.md grows past 12KB. Bootstrap silently truncates; new sessions can stall on the session write lock.
4. With this fix: MEMORY.md never exceeds memoryFileMaxChars (default 10KB). Older auto-promoted sections are compacted out at write time. Compaction is reflected in applied.compactedSections / applied.compactedDates on the function return.

Expected

• File size on disk after promotion writes is bounded by memoryFileMaxChars.
• User-authored content (any ## heading not matching the dreaming promotion pattern, plus the file's preamble) is preserved.
• Compaction is reported on the function return; callers can log if they want.

Actual (with this PR)

• Matches expected. Two new integration tests in short-term-promotion.test.ts exercise both branches: (a) over budget → drops oldest section; (b) under budget → no-op.

Evidence

Attach at least one:

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Local verification (Node 22.22.2, pnpm 10.33.0):

$ pnpm test extensions/memory-core/src/memory-budget.test.ts \
            extensions/memory-core/src/short-term-promotion.test.ts

[test] starting test/vitest/vitest.extension-memory.config.ts
 Test Files  2 passed (2)
      Tests  56 passed (56)        ← 11 new helper + 45 existing + 2 new integration

[test] passed 1 Vitest shard in 14.99s

Human Verification (required)

What I personally verified beyond CI:

• Verified scenarios:

  • Helper unit tests cover: under-budget no-op, over-budget drop-oldest, drop-oldest in date order regardless of file order, preservation of user-authored content, all-promotion-sections-dropped fallback, no-promotion-sections fallback, budgetChars <= 0 disable, empty input, ## heading sandwiched between promotions, default-budget-below-bootstrap-cap invariant.
  • Integration smokes cover: over-budget drops oldest seeded section before write through applyShortTermPromotions; under-budget leaves seeded content intact.

• Edge cases checked:

  • User-authored markdown is preserved unconditionally.
  • Drop order is by date (asc), not file order — protects against operators editing the file by hand.
  • budgetChars: 0 opts out of compaction entirely.

• Boundary checks performed locally beyond CI: targeted pnpm test (56/56), pnpm exec oxfmt --check (clean), git diff --check (clean), pnpm check:changed (exit 0), direct pnpm tsgo:core / pnpm tsgo:extensions / pnpm tsgo:test:src (all exit 0).

• Live driver run (real exported applyShortTermPromotions against a real filesystem workspace, NOT vitest). Pre-seeded MEMORY.md at 11,714 chars (over the 10,000 budget), then ran 5 promotion sweeps. Result:

  seeded MEMORY.md: 11714 chars (over budget)
  sweep 1: applied=1 compacted=3 (2026-04-01, 02, 03) → 9478 chars  user notes intact
  sweep 2: applied=1 compacted=0                       → 9723 chars  user notes intact
  sweep 3: applied=1 compacted=0                       → 9968 chars  user notes intact
  sweep 4: applied=1 compacted=1 (2026-04-04)          → 9386 chars  user notes intact
  sweep 5: applied=1 compacted=0                       → 9631 chars  user notes intact
  
  bounded: PASS
  user content preserved: PASS
  total old sections compacted across 5 sweeps: 4

  File stayed under the 10 KB budget after every sweep (as the file approached the cap, the next sweep that would have overflowed dropped exactly one oldest section to make room). User-authored ## My Personal Notes and its bullets survived all 5 sweeps unchanged.

• What you did not verify: A multi-week live dreaming run end-to-end through the actual dreaming.ts cron schedule on a configured agent. The driver above exercises the exact same applyShortTermPromotions code the dreaming sweep invokes, so any wiring bug between the cron and this function is the only gap.

Review Conversations

• I will reply to or resolve every bot review conversation I address in this PR.
• I will leave unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? Yes
• Config/env changes? No
• Migration needed? No
• If yes, exact upgrade steps: N/A. Pre-existing oversized MEMORY.md files will see one-time compaction on the next dreaming promotion. Older promoted sections (oldest by date) are dropped to bring the file under budget. User-authored content is never touched. The memoryFileMaxChars parameter is optional with a sensible default; existing callers that omit it get the new default behavior. Public SDK types (MemoryHostEvent etc.) are unchanged.

Risks and Mitigations

• Risk: A user with a long-running dreaming history loses access to old promoted snippets after the one-time compaction.
  • Mitigation: The bot's review explicitly accepted compaction as the right approach (vs. naive truncation). Older promoted content is captured in the dreaming pipeline's deeper history (short-term store + memory events) — only the inlined MEMORY.md copy is dropped. A follow-up could add an archive layer (#73691 issue body's "Option B"); intentionally out of scope for this surgical fix.
• Risk: An operator with extreme MEMORY.md user-authored content (>10KB of personal notes) sees no compaction effect because no promotion sections exist to drop.
  • Mitigation: This is the documented "log and continue" failure mode. The function never refuses to write and never touches user content. Operators in this situation can raise memoryFileMaxChars or address the user content themselves; bootstrap-side truncation continues to apply on the prompt path.
• Risk: The 10KB default is too aggressive for some workloads.
  • Mitigation: memoryFileMaxChars is a runtime parameter; the budget can be raised per call or disabled by passing 0. No config schema migration is needed. Default chosen with margin below bootstrap's 12KB cap so promoted memory continues to reach new sessions.

Changed files

• CHANGELOG.md (modified, +1/-0)
• extensions/memory-core/src/memory-budget.test.ts (added, +178/-0)
• extensions/memory-core/src/memory-budget.ts (added, +164/-0)
• extensions/memory-core/src/short-term-promotion.test.ts (modified, +124/-0)
• extensions/memory-core/src/short-term-promotion.ts (modified, +33/-2)