Hindsight provider can submit retain work during interpreter shutdown

Bug description

The native Hermes Hindsight memory provider can still submit async retain work after provider shutdown has started. In the shutdown window this can call asyncio.run_coroutine_threadsafe(), which may fail during Python interpreter teardown with:

RuntimeError: cannot schedule new futures after interpreter shutdown

This also risks secondary teardown noise such as unclosed aiohttp client/session warnings if cleanup is interrupted.

Affected area

• Repository: NousResearch/hermes-agent
• Provider: plugins/memory/hindsight/__init__.py
• Tests: tests/plugins/memory/test_hindsight_provider.py

Observed behavior

During process/session shutdown, Hindsight retain sync can run late enough that Python is already tearing down executor/future machinery. The provider currently waits for existing background threads in shutdown(), but normal sync_turn() can still accept new work during or after shutdown begins.

Expected behavior

Once provider shutdown begins:

1. Normal sync_turn() calls should not submit new async work.
2. Existing in-flight background work should be joined/drained.
3. Any pending partial auto-retain batch should be flushed before client close, so recent conversation turns are not lost.
4. Client cleanup (aclose() / close path) should still be allowed through a shutdown-owned path.

Root cause hypothesis

The provider shutdown path joins existing _prefetch_thread / _sync_thread and then closes the Hindsight client, but there is no lifecycle gate that freezes normal retain submission once shutdown starts.

A late sync_turn() can therefore still reach the async scheduling path:

asyncio.run_coroutine_threadsafe(coro, loop)

while the Python interpreter is shutting down.

Proposed fix

Add provider lifecycle state and a shutdown-safe final flush path:

• Add guarded provider state such as _shutting_down, _closed, and _state_lock.
• Have sync_turn() return early once shutdown starts.
• Guard normal _run_sync() calls after shutdown begins.
• Allow only explicit shutdown-owned cleanup/final-flush paths via an allow_shutdown=True style escape hatch.
• Extract retain submission into a helper that can be used both by normal sync and shutdown final flush.
• Track the last retained turn counter to avoid duplicating an already retained full-session document during shutdown.
• Add regression tests for:
  • sync_turn() after shutdown does not call asyncio.run_coroutine_threadsafe().
  • shutdown flushes a pending partial batch before closing the client.

Local validation from a candidate fix

A local candidate fix was tested with:

python -m pytest tests/plugins/memory/test_hindsight_provider.py -q -o 'addopts='
python -m py_compile plugins/memory/hindsight/__init__.py tests/plugins/memory/test_hindsight_provider.py
git diff --check -- plugins/memory/hindsight/__init__.py tests/plugins/memory/test_hindsight_provider.py

Result:

77 passed in 0.37s

The local candidate commit only touches:

plugins/memory/hindsight/__init__.py
tests/plugins/memory/test_hindsight_provider.py

Local commit reference, if useful for comparison:

cdd9953c fix(hindsight): prevent late retain sync during shutdown

Additional test isolation note

Some Hindsight provider tests can be polluted by a real host-level ~/.hindsight/config.json. Tests that rely on env fallback or no legacy config should isolate HOME / Path.home() to a temporary directory.

Suggested labels

• bug
• memory
• hindsight