hermes - 💡(How to fix) Fix feat(holographic): periodic fact checkpointing to survive abrupt session termination

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Fix Action

Fix / Workaround

The holographic memory plugin's auto_extract feature relies entirely on the on_session_end hook to persist facts from the conversation. This hook only fires at graceful session boundaries (CLI exit, /reset, gateway expiry with proper dispatch).

Code Example

# In HolographicMemoryProvider
def sync_turn(self, user_content: str, assistant_content: str, *, session_id: str = "") -> None:
    """Called after each turn — extract facts incrementally."""
    if not self._config.get("auto_extract", False):
        return
    if not self._store:
        return
    
    # Extract facts from user message only (assistant messages are responses, not facts)
    self._auto_extract_from_message(user_content)

---

# In gateway session loop
if time_since_last_extraction > checkpoint_interval:
    provider.checkpoint_facts(session_messages)

---

import signal
def _graceful_shutdown(signum, frame):
    provider.on_session_end(get_current_messages())
    sys.exit(0)

signal.signal(signal.SIGTERM, _graceful_shutdown)
signal.signal(signal.SIGINT, _graceful_shutdown)

---

plugins:
  hermes-memory-store:
    auto_extract: true
    extract_mode: "per_turn"      # "per_turn" | "session_end" | "both"
    extract_checkpoint_minutes: 5  # fallback timer if per_turn not available
RAW_BUFFERClick to expand / collapse

Feature Request: Resilient Holographic Memory — Surviving Abrupt Session Termination

Problem

The holographic memory plugin's auto_extract feature relies entirely on the on_session_end hook to persist facts from the conversation. This hook only fires at graceful session boundaries (CLI exit, /reset, gateway expiry with proper dispatch).

When the session ends abruptly — power loss, kill -9, OOM kill, Docker container stop, network disconnect — on_session_end is never called, and all conversational facts from that session are silently lost.

This is particularly impactful for:

  • Docker container users (container restarts lose session memory)
  • Long-running sessions that accumulate many facts before shutdown
  • Users who rely on auto_extract: true as their primary fact ingestion path

Current State (as of May 2026)

Related fixes have landed or are in progress:

  • #15165 / PR #16571 — Fixed on_session_end receiving empty message list on gateway restart
  • #11205 / PR #11304 — Fixed on_session_end not firing on gateway idle expiry
  • #22907 / PR #22959 — Fixed auto_extract storing raw messages instead of extracted facts

However, none of these address abrupt termination. The hook-based architecture fundamentally cannot handle SIGKILL, power loss, or forced container stop.

Proposed Solution: Periodic Fact Checkpointing

Add a periodic checkpoint mechanism that extracts and persists facts during the session, not just at the end.

Approach 1: Per-Turn Extraction (Recommended)

After each user-assistant turn, run a lightweight fact extraction pass on the last N messages and persist any new facts immediately.

# In HolographicMemoryProvider
def sync_turn(self, user_content: str, assistant_content: str, *, session_id: str = "") -> None:
    """Called after each turn — extract facts incrementally."""
    if not self._config.get("auto_extract", False):
        return
    if not self._store:
        return
    
    # Extract facts from user message only (assistant messages are responses, not facts)
    self._auto_extract_from_message(user_content)

Pros:

  • Facts are persisted immediately, no loss on crash
  • Minimal overhead (one regex pass per turn)
  • No architectural changes needed

Cons:

  • Slightly more DB writes (but SQLite WAL mode handles this well)
  • May extract duplicate facts (deduplication via UNIQUE constraint handles this)

Approach 2: Periodic Timer-Based Extraction

Run fact extraction every N minutes during an active session.

# In gateway session loop
if time_since_last_extraction > checkpoint_interval:
    provider.checkpoint_facts(session_messages)

Pros:

  • Batches extraction, fewer DB writes
  • Configurable interval

Cons:

  • More complex (requires timer/threading in gateway)
  • Still loses facts from the last interval before crash

Approach 3: Signal Handler for Graceful Shutdown

Install signal handlers for SIGTERM, SIGINT to call on_session_end before exit.

import signal
def _graceful_shutdown(signum, frame):
    provider.on_session_end(get_current_messages())
    sys.exit(0)

signal.signal(signal.SIGTERM, _graceful_shutdown)
signal.signal(signal.SIGINT, _graceful_shutdown)

Pros:

  • Handles Docker docker stop (sends SIGTERM)
  • Handles Ctrl+C in CLI

Cons:

  • Does NOT handle SIGKILL, power loss, OOM
  • Adds signal handling complexity

Recommendation

Combine Approach 1 + Approach 3:

  • Per-turn extraction for crash resilience (handles all cases)
  • Signal handlers for graceful shutdown optimization (reduces redundant extraction)

Configuration

plugins:
  hermes-memory-store:
    auto_extract: true
    extract_mode: "per_turn"      # "per_turn" | "session_end" | "both"
    extract_checkpoint_minutes: 5  # fallback timer if per_turn not available

Backward Compatibility

  • Default extract_mode: "session_end" preserves current behavior
  • Users opt in to "per_turn" or "both" for resilience
  • No breaking changes

Related Issues

  • #15165 — on_session_end receives empty messages on gateway restart
  • #11205 — on_session_end not called on gateway idle expiry
  • #22907 — auto_extract stores raw messages instead of facts
  • #14981 — on_session_finalize not fired on idle timeout
  • PR #16571 — Fix for #15165
  • PR #22959 — Fix for #22907

Motivation

I run Hermes in a Docker container on a personal computer. When the machine shuts down or the container is stopped, the holographic memory loses all facts from the current session. Periodic checkpointing would make the memory system resilient to real-world failure modes.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

hermes - 💡(How to fix) Fix feat(holographic): periodic fact checkpointing to survive abrupt session termination