hermes - 💡(How to fix) Fix [Feature]: Add 3-second degradation timeout for external memory providers (fallback to builtin when hindsight is down)

Problem

When the hindsight memory daemon is unresponsive (PostgreSQL crash, network issue, etc.), memory_manager.py's sync_all() and handle_tool_call() block indefinitely on the external provider call. There is no timeout or fallback to the builtin memory provider.

This causes the agent to freeze for 178-300s on every conversation turn, with no indication to the user that memory has degraded.

Proposed Solution

Wrap external (non-builtin) memory provider calls in a short timeout (3 seconds). If the provider times out, log a warning, increment a counter, and proceed without blocking. The builtin provider always executes directly (it's in-process and millisecond-fast).

Key Design Decisions

1. builtin provider is exempt — it's always fast and local, no timeout needed
2. Timeout, don't crash — TimeoutError is caught and logged, not re-raised
3. Stateless degradation — next turn retries the external provider; if it recovers, it works again immediately
4. Observability — _external_timeout_count and _external_timeout_name track degradation for monitoring

Affected Code

agent/memory_manager.py:

• sync_all() — the provider loop (around line 380)
• handle_tool_call() — the external provider dispatch (around line 470)

Reference Implementation

A working implementation is available with timeout counters, logging, and ThreadPoolExecutor-based timeout wrapper. Key pattern:

# In sync_all(): wrap non-builtin providers
if provider.name == 'builtin':
    provider.sync_turn(...)
else:
    with ThreadPoolExecutor(max_workers=1) as pool:
        f = pool.submit(provider.sync_turn, ...)
        f.result(timeout=3)

Full reference: see comment in #35195.

Impact

• Hindsight failure → 3-second delay instead of 178-second freeze
• User can keep using the agent with degraded memory (builtin still works)
• Zero behavior change when hindsight is healthy

Related

• #35195 — hindsight-embed pg0 hardcoded openssl path (root cause of daemon failure)
• This feature would make memory degradation a non-event for end users