PR #66546: feat(sessions): add transcriptRotateBytes and transcriptMaxLines to cap .jsonl growth

• Repository: openclaw/openclaw
• Author: konanok
• State: open | merged: False
• Link: https://github.com/openclaw/openclaw/pull/66546

Description (problem / solution / changelog)

Summary

• Problem: session.maintenance controls (rotateBytes, maxDiskBytes) only affect sessions.json, not individual transcript .jsonl files. These files grow unbounded in long-lived group chats, reaching 100MB+ and causing gateway CPU 100%.
• Why it matters: Users who configure rotateBytes reasonably expect transcript files to be capped, but .jsonl files are completely uncontrolled — a silent critical failure.
• What changed: Added transcriptRotateBytes and transcriptMaxLines config options. When enabled, transcript rotation archives oversized .jsonl files as .jsonl.bak.<timestamp> and writes a replacement with the session header + last N lines. Old backups are pruned to keep the 3 most recent per transcript.
• What did NOT change (scope boundary): No changes to existing rotateBytes, maxDiskBytes, pruneAfter, or enforceSessionDiskBudget logic. New fields default to null (disabled), so behavior is unchanged without explicit configuration.

Design Decisions

Why transcriptRotateBytes instead of reusing rotateBytes?

Three approaches were considered:

Chose B because:

1. Different thresholds: sessions.json is a structured index — 10MB is already large. .jsonl is a message stream — 10MB may be just a few thousand lines, and users may want 50MB before rotation. A shared threshold forces a compromise.
2. Different control dimensions: .jsonl rotation needs transcriptMaxLines (tail lines to keep for context window), which sessions.json does not need.
3. Backward compatibility: Reusing rotateBytes would make rotateBytes: "10mb" suddenly affect all .jsonl files — a behavioral change. Independent fields default to null (disabled).
4. Simplicity: Approach C adds mental overhead (priority rules between two fields) for little gain.

Why check only the active session's transcript on the hot path?

The original implementation called rotateTranscriptFiles() on every saveSessionStoreUnlocked(), which does a full directory walk (stat every .jsonl file). For a gateway with 100+ sessions, this means 100+ stat() calls per message.

Current design: the hot path (saveSessionStore with activeSessionKey) only stats the single active transcript. Bulk rotation (full directory walk) is gated behind bulkTranscriptRotation, which is only set by explicit maintenance entry points (sessions-cleanup). Runtime call sites without activeSessionKey (e.g. heartbeat-runner) skip transcript rotation entirely.

When both bulkTranscriptRotation and activeSessionKey are provided (e.g. sessions cleanup --active-key), the bulk path takes priority so all oversized transcripts are covered, not just the active one.

Why transcript rotation does not execute under mode: "warn"

By design, session maintenance uses a dual-mode system: "warn" (default) and "enforce".

• warn mode (safe-by-default): Logs warnings for prune/cap thresholds but does not execute any destructive operations. This prevents accidental data loss when users upgrade and gain new maintenance features.
• enforce mode: Executes all maintenance operations — pruning, capping, transcript rotation, and reset archiving.

This mirrors the existing pattern for maxSessions pruning and maxSessionEntries capping, which also require mode: "enforce" to take effect. Transcript rotation follows the same safe-by-default principle.

Known gap: The warn branch currently does not log a warning when transcript rotation thresholds are exceeded. This is a candidate for a follow-up enhancement.

Concurrency safety: rotation rollback

The rename → write replacement sequence has a TOCTOU window where a concurrent transcript append can recreate the file. The implementation handles this with layered defenses:

1. O_EXCL write: If a concurrent writer recreated the file, the replacement write throws EEXIST and is skipped. The archive is safe regardless.
2. weCreatedReplacement flag: Tracks whether we created the replacement file via O_EXCL. On rollback:
   • If true (open succeeded, writeFile failed): safe to unlink the partial file before restoring from archive.
   • If false (failure before open): the file at transcriptPath, if any, belongs to a concurrent writer and must not be deleted.

Self-Review Checklist

Issues found and fixed during development and bot review:

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes #66360
• This PR fixes a bug or regression

Root Cause (if applicable)

• Root cause: rotateBytes only targets sessions.json (the session index). No code path checks or limits the size of individual transcript .jsonl files. For long-running topic-bound sessions (Telegram forum topics, group chats), every message/tool call/tool result is appended in full with no cap.
• Missing detection / guardrail: No file-size check on transcript files during session store saves or maintenance sweeps.
• Contributing context (if known): The sessions.json rotation was added in an earlier release; transcript files were assumed to be bounded by pruneAfter session expiry, but topic-bound sessions are never pruned because they remain active.

Regression Test Plan (if applicable)

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file: src/config/sessions/store-maintenance.transcript-rotation.test.ts (50 tests)
• Scenario the test should lock in: Transcript file exceeding transcriptRotateBytes is archived and replaced with header + tail lines; rollback restores original on write failure (including partial write with weCreatedReplacement tracking); EEXIST path marks rotation as succeeded; bulk rotation covers all transcripts; bulkTranscriptRotation takes priority over activeSessionKey.
• Why this is the smallest reliable guardrail: Unit tests with fs mocks cover all rotation/rollback paths without requiring a running gateway.
• Existing test that already covers this (if any): None — this is a new feature.
• Known test gap: The EEXIST test verifies the code path does not error, but does not simulate a true concurrent writer recreating the file between rename and open. The O_EXCL defense is verified structurally, not via a race simulation.

User-visible / Behavior Changes

• Two new optional config fields under session.maintenance:
  • transcriptRotateBytes (string | number | null): Rotate transcript .jsonl files exceeding this size. Example: "10mb". Default: null (disabled).
  • transcriptMaxLines (number | null): Number of most recent lines to keep after rotation. Default: null (archive entire file, replacement contains only the session header).
• Rotation creates .jsonl.bak.<timestamp> archives, auto-pruned to 3 most recent per transcript.
• Requires session.maintenance.mode: "enforce" to take effect (consistent with existing maintenance behavior).

Diagram (if applicable)

Hot path (per message write):
  saveSessionStore(store, path, { activeSessionKey })
    → stat(active transcript)
    → if size > transcriptRotateBytes:
        rename(transcript → .bak)
        → open(O_EXCL) → writeFile(header + tail)
        → cleanup old .bak files (keep 3)
        on write failure:
          → if weCreatedReplacement: unlink partial file
          → rename(.bak → transcript)  // restore original

Bulk path (sessions-cleanup CLI only):
  saveSessionStore(store, path, { bulkTranscriptRotation: true })
    → readdir(sessions dir)
    → for each .jsonl: stat → rotate if oversized

Security Impact (required)

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? No
• Command/tool execution surface changed? No
• Data access scope changed? No

Repro + Verification

Environment

• OS: Darwin 25.4.0 (arm64, Apple Silicon), Docker (ubuntu:24.04)
• Runtime/container: Node v25.8.1
• Model/provider: N/A (infrastructure change)
• Integration/channel (if any): Telegram forum topics (long-lived sessions)
• Relevant config (redacted): transcriptRotateBytes: "10mb", transcriptMaxLines: 500

Steps

1. Create a session entry with a 15MB .jsonl transcript (42,714 lines)
2. Configure transcriptRotateBytes: "10mb" and transcriptMaxLines: 500
3. Run openclaw sessions cleanup --enforce

Expected

• topic-2.jsonl reduced to ~180KB (500 lines + session header)
• Old data archived as topic-2.jsonl.bak.<timestamp>

Actual

• Confirmed: 15MB → 180KB, 42,714 → 500 lines, archive created

Evidence

• Trace/log snippets
• Failing test/log before + passing after
• Screenshot/recording
• Perf numbers (if relevant)

Docker verification: 15MB topic-2.jsonl rotated to 180KB with .bak archive preserved.

Human Verification (required)

• Verified scenarios: pnpm build passes, pnpm check (lint + format + import cycles) passes, pnpm test -- src/config/sessions/store-maintenance.transcript-rotation.test.ts (50 tests) passes, pnpm test -- src/config/sessions/store.pruning.integration.test.ts (12 tests) passes, pnpm test -- src/config/schema.base.generated.test.ts (5 tests, schema drift) passes
• Edge cases checked: null/disabled defaults (no rotation), 0/negative values (treated as disabled), file under threshold (no-op), multiple .jsonl files in same directory, single-file hot path (rotateTranscriptFile) vs full walk (rotateTranscriptFiles), session header preservation after rotation, 0o600 file permissions, O_EXCL concurrent-write detection (structural, not race-simulated), partial-write rollback with weCreatedReplacement tracking, bulkTranscriptRotation priority over activeSessionKey
• What I did not verify: Full gateway integration test with live channels; extremely large files (>1GB); Windows file locking behavior; true concurrent-writer race simulation for EEXIST path
• Note: activeSessionKey was added to most runtime updateSessionStore call sites that have the key in scope. A few call sites (model-selection.ts, session-reset-model.ts, one path in commands-session-store.ts) do not pass it because the session key is not readily available or the update is fire-and-forget. These paths fall back to the pre-existing behavior (no transcript rotation on that write), which is safe.

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? Yes
• Config/env changes? Yes (two new optional fields, default null — no behavior change without explicit opt-in)
• Migration needed? No

Risks and Mitigations

• Risk: Rotation during active write could lose in-flight messages
  • Mitigation: O_EXCL write detects concurrent recreation and skips replacement. weCreatedReplacement flag ensures rollback only deletes files we created, not concurrent writers'. The session store lock serializes saveSessionStore calls.
• Risk: .bak files accumulate on disk
  • Mitigation: Automatic pruning keeps only the 3 most recent backups per transcript file (MAX_ROTATION_BACKUPS)
• Risk: Non-active sessions with large transcripts are not rotated on the hot path
  • Mitigation: bulkTranscriptRotation flag enables full-directory sweep from sessions-cleanup CLI. Takes priority over activeSessionKey so cleanup always covers all transcripts.
• Risk: transcriptRotateBytes: 0 or negative values could cause unexpected behavior
  • Mitigation: resolveTranscriptRotateBytes() returns null for <= 0, and the hot path guard checks != null && > 0
• Risk: A few updateSessionStore call sites do not pass activeSessionKey
  • Mitigation: These paths skip hot-path transcript rotation (equivalent to pre-existing behavior). Oversized transcripts from these paths are caught by bulk rotation via sessions-cleanup.

AI-Assisted

• This PR was developed with AI assistance (CodeBuddy)
• Testing level: Fully tested (50 unit tests + 12 integration tests + 5 schema drift tests)
• I understand what the code does

Changed files

• docs/.generated/config-baseline.sha256 (modified, +3/-3)
• src/auto-reply/reply/abort-cutoff.runtime.ts (modified, +13/-9)
• src/auto-reply/reply/abort.ts (modified, +18/-14)
• src/auto-reply/reply/agent-runner-execution.ts (modified, +32/-20)
• src/auto-reply/reply/agent-runner-session-reset.ts (modified, +7/-3)
• src/auto-reply/reply/body.ts (modified, +15/-11)
• src/auto-reply/reply/commands-acp/lifecycle.ts (modified, +15/-11)
• src/auto-reply/reply/commands-session-store.ts (modified, +21/-13)
• src/auto-reply/reply/directive-handling.impl.ts (modified, +7/-3)
• src/auto-reply/reply/directive-handling.persist.ts (modified, +7/-3)
• src/auto-reply/reply/get-reply-run.ts (modified, +7/-3)
• src/auto-reply/reply/session-updates.ts (modified, +17/-9)
• src/commands/sessions-cleanup.ts (modified, +1/-0)
• src/config/schema.base.generated.ts (modified, +31/-0)
• src/config/schema.help.ts (modified, +4/-0)
• src/config/schema.labels.ts (modified, +2/-0)
• src/config/sessions/runtime-types.ts (modified, +2/-0)
• src/config/sessions/store-maintenance.transcript-rotation.test.ts (added, +955/-0)
• src/config/sessions/store-maintenance.ts (modified, +348/-3)
• src/config/sessions/store.pruning.integration.test.ts (modified, +125/-0)
• src/config/sessions/store.ts (modified, +40/-1)
• src/config/types.base.ts (modified, +14/-0)
• src/config/zod-schema.session-maintenance-extensions.test.ts (modified, +85/-0)
• src/config/zod-schema.session.ts (modified, +15/-0)
• src/gateway/server-methods/agent.ts (modified, +14/-10)
• src/gateway/server-methods/sessions.ts (modified, +138/-98)
• src/gateway/server-node-events.ts (modified, +26/-22)
• src/gateway/session-compaction-checkpoints.ts (modified, +18/-14)
• src/gateway/session-reset-service.ts (modified, +107/-103)
• src/gateway/sessions-resolve.test.ts (modified, +5/-1)
• src/gateway/sessions-resolve.ts (modified, +10/-6)