hermes - 💡(How to fix) Fix Python shutdown deadlock: worker thread blocks Py_FinalizeEx, process hangs forever after exit

Bug Description

After certain operations (observed after Ctrl+C or internal errors), the Hermes CLI process hangs indefinitely — terminal becomes unresponsive, no input accepted, but process stays alive consuming CPU.

Cause

A classic Python interpreter shutdown deadlock:

1. Exit is triggered (Ctrl+C / fatal error)
2. asyncio.run() cleanup calls _cancel_all_tasks()
3. A prompt_toolkit raw_mode exit handler triggers termios.tcsetattr(), which hits a signal handler (cli.py:12778) that raises KeyboardInterrupt
4. Python begins Py_FinalizeEx → wait_for_thread_shutdown
5. The main thread enters _pthread_cond_wait, waiting for all other threads to exit
6. A worker thread never exits — it continues executing Python bytecode (deep call chain: method_vectorcall → _PyFunction_Vectorcall → _PyEval_EvalFrameDefault). Thread does not check any exit flag or respond to signals.
7. Deadlock: main thread waits forever on condition variable, worker thread runs forever

Environment

• OS: macOS 26.5 (25F71), Apple Silicon arm64
• Python: cpython-3.11.15
• Hermes: installed from source (~/.hermes/hermes-agent/), run via hermes CLI
• Model provider: DeepSeek (deepseek-v4-pro)

Evidence

Process state (macOS sample)

• PID: 57674, state: S+ (sleeping foreground), launched 01:07, still alive at 02:55+
• Physical footprint: 174MB

Main thread stack:

Py_FinalizeEx → wait_for_thread_shutdown → lock_PyThread_acquire_lock
→ acquire_timed → PyThread_acquire_lock_timed → _pthread_cond_wait → __psynch_cvwait

Rogue worker thread (Thread_23283899):

thread_run → method_vectorcall → _PyFunction_Vectorcall → _PyEval_EvalFrameDefault
(looping through Python bytecode — NOT sleeping, actively executing)

Log excerpt

File ".../asyncio/runners.py", line 201, in _cancel_all_tasks
    loop.run_until_complete(tasks.gather(*to_cancel, return_exceptions=True))
  ...
File ".../prompt_toolkit/input/vt100.py", line 282, in __exit__
    termios.tcsetattr(self.fileno, termios.TCSANOW, self.attrs_before)
File ".../hermes-agent/cli.py", line 12778, in _signal_handler
    raise KeyboardInterrupt()
KeyboardInterrupt

lsof

• FD 14: state.db-wal (5.7MB) — write-ahead log still open
• FD 19: NPOLICY — network policy fd
• FD 24: TCP localhost:57252→localhost:amt-esd-prot (CLOSED)

Severity

Process requires kill -9 to terminate. SIGTERM has no effect (main thread is in kernel wait, worker thread ignores signals). Normal kill is silently ignored.

This is reproducible but interrmittent — likely race-condition dependent on whether a background thread happens to be in a Python execution context at the moment of shutdown.

Possible Mitigations

• Ensure all non-daemon threads have a stop event / check threading.main_thread().is_alive()
• Consider setting worker threads as daemon=True if they don't hold critical resources
• Or add a timeout to wait_for_thread_shutdown with forceful exit
• Investigate the cli.py:12778 signal handler raising KeyboardInterrupt during prompt_toolkit cleanup — this double-exception may exacerbate the issue