hermes - 💡(How to fix) Fix [Bug]: approval prompt get_input deadlocks against prompt_toolkit MainThread (regression of #13617) [4 comments, 2 participants]

Summary

The interactive CLI deadlocks when tools/approval.py's get_input is invoked from a background thread while prompt_toolkit's Application.run() is already running on the MainThread. Both compete for stdin; the user's keystrokes go to prompt_toolkit, never reach the approval thread, and the approval thread's read blocks indefinitely. The CLI displays the ❯ prompt and keeps the ⏲ timer ticking, but accepts no keyboard input. Ctrl+C is ignored (approval thread is blocked at a C-level stdin read, not interruptible by Python signal handlers). Only kill -TERM (and sometimes -KILL) on the agent process recovers.

This looks like a regression of closed issue #13617 ("Bug: Terminal approval prompt freezes input area, preventing user interaction") — same observable symptom, possibly a different code path (tools/approval.py:463 get_input + a concurrent terminal_tool._cleanup_thread_worker) not covered by that fix.

Filed in place of #15194, which I filed earlier based on log evidence alone and misdiagnosed as an auxiliary_client timeout bug. A py-spy dump (see below) made it clear the hang is in the approval path, not in an HTTP await. Will close #15194 as duplicate of this one.

Environment

• OS: macOS (Apple Silicon)
• Hermes: v0.11.0 (2026.4.23) + ~70 commits pulled today (most recent hermes update completed successfully)
• Python: 3.11.14
• Provider: Anthropic (claude-opus-4-6)
• MCP servers active at hang: github, atlassian, slack, plus one internal stdio server (~73 tools total after removing a 221-tool server that was unrelated).

Reproduction (observed, not yet minimized)

1. Start a session with several MCP servers registered and terminal_tool enabled.
2. Send a prompt that causes Hermes to run at least one shell-terminal tool call early in the turn.
3. After the terminal tool finishes and _cleanup_thread_worker is active, the next tool invocation (or the subsequent approval prompt) fires tools/approval.py:get_input on a background thread.
4. MainThread is in prompt_toolkit.application.Application.run → asyncio run_until_complete → selectors.select (normal idle state waiting for input).
5. Input area displays ❯ but accepts nothing. Ctrl+C is ignored. Timer continues counting. No new log lines appear in ~/.hermes/logs/agent.log.
6. Process is alive, ~0% CPU, status S — parked on get_input.

py-spy dump (sanitized, full process)

Process <PID>: <python> <home>/.local/bin/hermes
Python v3.11.14

Thread <MAIN> (idle): "MainThread"
    select (selectors.py:566)
    _run_once (asyncio/base_events.py:1898)
    run_forever (asyncio/base_events.py:608)
    run_until_complete (asyncio/base_events.py:641)
    run (asyncio/runners.py:118)
    run (asyncio/runners.py:190)
    run (prompt_toolkit/application/application.py:1002)
    run (cli.py:10718)
    main (cli.py:11074)
    cmd_chat (hermes_cli/main.py:1204)
    main (hermes_cli/main.py:9118)
    <module> (hermes:10)

Thread <BG> (idle): "mcp-event-loop"
    select (selectors.py:566)
    _run_once (asyncio/base_events.py:1898)
    run_forever (asyncio/base_events.py:608)
    run (threading.py:982)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)

Thread <BG> (active): "asyncio-waitpid-0"     # x4 of these
    _do_waitpid (asyncio/unix_events.py:1404)
    run (threading.py:982)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)

Thread <BG> (active): "Thread-2 (spinner_loop)"
    spinner_loop (cli.py:10498)
    ...

Thread <BG> (idle): "Thread-3 (process_loop)"
    wait (threading.py:331)
    get (queue.py:180)
    process_loop (cli.py:10509)
    ...

Thread <BG> (idle): "patch-stdout-flush-thread"
    wait (threading.py:327)
    get (queue.py:171)
    _write_thread (prompt_toolkit/patch_stdout.py:153)
    ...

Thread <BG> (active): "Thread-7 (_cleanup_thread_worker)"
    _cleanup_thread_worker (tools/terminal_tool.py:1115)
    ...

Thread <BG> (active): "Thread-31 (get_input)"        # <-- BLOCKED HERE
    get_input (tools/approval.py:463)
    run (threading.py:982)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)

Key frames:

• tools/approval.py:463 get_input — the blocking read.
• prompt_toolkit/application/application.py:1002 run on MainThread — already owns the terminal's input path via patch_stdout.
• tools/terminal_tool.py:1115 _cleanup_thread_worker — actively running when the deadlock occurred, so the approval likely fired right at / after terminal-tool cleanup.

Suspected root cause

tools/approval.py:get_input appears to call a blocking input() / sys.stdin.readline() / low-level read(0) from a non-main thread while prompt_toolkit.Application.run() is active on the main thread. prompt_toolkit's patch_stdout / input capture already owns the TTY, so the background read never sees a complete line. Additionally, the C-level blocking read is not interruptible by Ctrl+C in Python (signal only delivered to main thread), so the only recovery is SIGKILL.

Possible fixes:

1. Route all approval prompts through the same prompt_toolkit app (e.g. Application.create_background_task + a modal dialog / float) rather than a raw input() on a side thread.
2. If a side-thread prompt is unavoidable, temporarily suspend prompt_toolkit (app.output.flush() + app.exit(...) / run_in_terminal) before reading stdin, and restore afterward.
3. As a minimum, add a timeout + cancellation token to get_input so the CLI is recoverable without SIGKILL.

Expected behavior

1. Approval prompts must not deadlock the main input loop. Whatever thread owns stdin, keystrokes should reach the approval consumer.
2. Ctrl+C during an approval wait should cancel the approval (treated as "deny" or "abort"), return control to the prompt, not require SIGKILL.
3. A visible indicator (banner / toast) should tell the user an approval is pending — the current failure mode is invisible (just the normal ❯ with a ticking timer).

Workaround (for users hitting this)

• Run with --yolo to bypass approval prompts entirely:

  hermes --yolo

• Or disable the specific tool that's triggering approvals (e.g. hermes tools → disable terminal_tool).
• If already stuck: kill -TERM <pid>; if that fails (stdin read not interruptible), kill -KILL <pid>. The gateway (launchd-managed) will auto-respawn — safe to ignore.

Cross-references

• Closes / duplicates my own prior issue #15194 (logs-only diagnosis pointed at auxiliary_client, turned out to be this deadlock in disguise).
• Appears to be a regression of closed #13617 — same observable symptom (approval prompt freezes input), may be a code path that #13617's fix didn't cover.

Additional context I can provide on request

• Full un-truncated py-spy dump output
• agent.log tail around the freeze (trailing auxiliary_client lines that are NOT the cause — they're just the last logs written before the deadlock)
• Reproduce with verbose tracing if maintainers can share a debug flag