hermes - 💡(How to fix) Fix [Bug]: Hindsight local_embedded causes infinite daemon crash-restart loop + CPU/RAM spike when running Hermes as root

Bug Description

When Hermes Agent runs as the root user on Linux with Hindsight configured in local_embedded mode, the Hindsight daemon enters an infinite crash-restart loop that:

1. Prevents Hindsight from ever working — PostgreSQL's initdb explicitly refuses to run as root by design
2. Consumes massive resources — each restart cycle loads embedding models (~958MB RAM, ~33% CPU)
3. Degrades the entire Hermes process — the daemon thread hammers the system indefinitely
4. Gives zero user-visible feedback — no error shown in the terminal; the user just sees Hermes become sluggish or hang during startup

This is a critical issue for the common use case of running Hermes on a VPS/cloud server as root.

Root Cause

Hindsight local_embedded uses PostgreSQL via pg0-embedded. PostgreSQL's initdb command has a hard check that refuses to run as root:

initdb: cannot be run as root

When this happens in the Hindsight daemon startup flow:

1. initialize() spawns a background thread (hindsight-daemon-start) that calls client._ensure_started()
2. The daemon manager starts, hits the initdb error, waits ~177 seconds, times out
3. It then retries — this repeats in an infinite loop
4. Each cycle loads embedding models (~958MB RAM, ~33% CPU)
5. The daemon never stabilizes; PostgreSQL data directory is never populated
6. No memories are actually stored or retrievable

Log evidence from ~/.hermes/logs/hindsight-embed.log:

Starting Daemon... ERROR: Application startup failed. Exiting... Daemon failed to start (timeout)
Starting Daemon... ERROR: Application startup failed. Exiting... Daemon failed to start (timeout)
... (repeats indefinitely)

Process evidence: ps aux | grep hindsight-api shows a process continuously consuming CPU and memory.

Contributing Factors (Compound Bugs)

This issue is made worse by several related bugs in the Hindsight plugin:

1. plugin.yaml missing hindsight-all dependency (already reported as #7718)

plugin.yaml declares only hindsight-client>=0.4.22, but local_embedded mode requires hindsight-all (which provides the hindsight top-level module needed by from hindsight import HindsightEmbedded). This was filed separately as #7718 but compounds the root-user issue — users may not even get far enough to hit the daemon crash loop because the import fails first.

2. post_setup silently swallows install errors

In __init__.py, the post_setup() method installs dependencies with capture_output=True and --quiet:

subprocess.run(
    [uv_path, "pip", "install", "--python", sys.executable, "--quiet", "--upgrade"] + deps_to_install,
    check=True, timeout=120, capture_output=True,
)

If the install fails (e.g., missing build tools, disk space, incompatible architecture), the error is silently swallowed. The setup wizard prints ✓ Dependencies up to date or shows a warning only in the except block — but the user may miss it in the interactive flow. There is no post-setup verification that the installed package actually imports successfully.

3. No runtime guard for root user + local_embedded incompatibility

initialize() detects local_embedded mode and unconditionally starts the daemon in a background thread. There is no check for:

• Whether the current user is root
• Whether PostgreSQL initdb can succeed
• Whether the daemon actually started successfully

The daemon startup runs in a fire-and-forget thread with no health check or timeout guard that would stop the retry loop.

Environment

• OS: Linux (VPS/cloud server)
• User: root
• Hermes: latest (via pip/uv)
• Hindsight mode: local_embedded
• Python: 3.12+

Suggested Fixes

Fix 1: Detect root user and refuse local_embedded mode (recommended)

In initialize() or _get_client(), add a check before daemon startup:

import os

if os.getuid() == 0 and self._mode == "local_embedded":
    logger.error(
        "Hindsight local_embedded cannot run as root. "
        "PostgreSQL initdb refuses root. "
        "Either run Hermes as a non-root user, or use cloud / local_external mode."
    )
    # Skip daemon startup entirely
    return

Fix 2: Add daemon startup health check with retry limit

The _start_daemon() thread should:

• Verify daemon started successfully (check a health endpoint or PID)
• Limit retry attempts (e.g., 3 retries then give up with a clear error)
• Log a prominent warning visible to the user

Fix 3: Fix plugin.yaml to declare mode-conditional dependencies

Track hindsight-all for local_embedded mode (see #7718).

Fix 4: Verify package imports in post_setup

After installing deps, verify the import actually works:

try:
    if mode == "local_embedded":
        import hindsight  # noqa: F401
    else:
        import hindsight_client  # noqa: F401
except ImportError:
    print("  ⚠ Installation verification failed!")

Related Issues

• #7718 — plugin.yaml missing hindsight-all dependency
• #8972 — macOS Apple Silicon daemon hangs (different platform, similar daemon startup failure)
• #7135 — macOS Apple Silicon daemon startup timeout (related daemon startup failure)
• #10590 — Hindsight daemon spams TUI with uv spinner (another manifestation of the daemon startup path being problematic)