Symptom

After a user interrupts an in-flight gateway turn (Discord, in our case), the gateway enters a loop where the periodic "Still working..." notification fires every 3 to 6 minutes with the same payload:

Still working... (3 min elapsed — iteration 0/60, starting new turn (cached))
Still working... (6 min elapsed — iteration 0/60, starting new turn (cached))
Still working... (9 min elapsed — iteration 0/60, starting new turn (cached))
...

The iteration counter never advances past 0/60. The (cached) marker stays. The existing inactivity-based timeout (HERMES_AGENT_TIMEOUT, default 1800s) never fires. Subsequent user messages spawn new turns that inherit the same stuck state instead of killing the parent. In our Discord channel this manifested as a "ghost session" that kept posting heartbeats for over 30 minutes until the gateway was manually restarted.

Root cause

The agent-cache reuse path in gateway/run.py (around line 9843, in the per-message handler that resolves a turn against _agent_cache) does this when it reuses a cached AIAgent:

agent._last_activity_ts = time.time()
agent._last_activity_desc = "starting new turn (cached)"
agent._api_call_count = 0

These three resets create a blind spot for the existing inactivity watchdog:

1. _api_call_count = 0 makes the heartbeat read "iteration 0/60" on every cached reuse, which is why operators see no progress.
2. _last_activity_desc = "starting new turn (cached)" is the source of the (cached) flag.
3. _last_activity_ts = time.time() is the load-bearing one. The inactivity watchdog polls seconds_since_activity from get_activity_summary() to decide when to kill a turn. If _last_activity_ts is reset to "now" on every cached reuse, the watchdog never accumulates idle time and the 1800s deadline is perpetually pushed forward. The safety net is silently disabled.

This is the direct counterpart of #9051. The fix for #9051 added an early _touch_activity() reset at the start of run_conversation() to prevent a stale timestamp from a previous turn from triggering a false-positive immediate timeout. That fix solves one direction (stale timestamp causes premature kill) but creates the opposite failure mode in the cached-reuse path: a perpetually-fresh timestamp prevents any kill at all when the new turn never makes its first API call.

This issue documents the resulting loop and proposes a turn-level watchdog that preserves the cache-reuse behavior introduced for #9051 while restoring an independent safety net.

Why interrupts make it worse

The interrupt path in _send_busy_ack (around line 1592 of gateway/run.py) calls running_agent.interrupt(event.text) and queues the new message for a successor turn. The interrupted parent agent finishes its in-flight tool call (or hangs in pre-flight setup), then the successor turn dispatches, hits the _agent_cache with a matching signature, and the three resets above run. If the successor itself stalls before its first API call, it is now indistinguishable from a healthy paused turn from the watchdog's point of view, because the cached reuse just refreshed the timestamp.

In practice users who keep typing while the agent is busy keep retriggering this cycle. Each interrupt resets the safety net.

Repro

1. Open a Discord session bound to the gateway with a long-running task that takes more than HERMES_AGENT_NOTIFY_INTERVAL to complete (default 180s). A multi-tool research turn or a delegated subagent works.
2. While the turn is running, send a second user message in the same channel within ~10 seconds. Send a third one a few seconds after that.
3. Observe journalctl -fu hermes-gateway and the channel itself. Within one notify interval the heartbeat starts reading iteration 0/60, starting new turn (cached) and continues to do so every 3 minutes until the gateway is restarted.
4. Inspect the agent's activity summary via /status (or by attaching a debugger) to confirm _last_activity_ts is being refreshed on every heartbeat tick rather than reflecting actual API or tool activity.

The bug does not require Discord specifically. Any adapter that exercises the busy-ack and cached-reuse paths reproduces it.

Proposed fix

Implemented and tested locally on a private branch. Four parts, all in or next to gateway/:

• Turn-level watchdog (gateway/turn_watchdog.py). Two kill rules. cache_loop: iteration == 0 AND cached AND elapsed > 90s. stall: identical (iteration, cached, elapsed_bucket) tuple observed twice consecutively. The cache-loop rule owns the iter==0+cached case exclusively so the operator log attributes the kill to the right root cause instead of the generic stall reason. To avoid false positives on legitimate long Codex turns, the watchdog reads a new is_cached_turn field from get_activity_summary(). The flag is set in the cached-reuse path and cleared inside _touch_activity() the first time the turn makes any real progress, so a turn that genuinely advances past iteration 0 is never killed by this rule.
• Interrupt-orphan cleanup. When _send_busy_ack interrupts a running agent for a user message, also evict the cached agent for that session. The successor turn rebuilds a fresh AIAgent instead of inheriting the cached signature whose _api_call_count and _last_activity_ts were just reset. This costs a prompt-cache miss on the interrupt-then-reply path, which is acceptable for the safety guarantee.
• Heartbeat throttle. A small per-turn state object suppresses the "Still working..." Discord post when (iteration, elapsed_bucket) is unchanged from the previous heartbeat. Operators no longer see N copies of the same status when the watchdog is observing a stall window. The watchdog still evaluates every tick; only the user-facing send is suppressed.
• Audit log. Kills are appended as JSONL records to $HERMES_STATE_DIR/turn_watchdog.log (defaults to ~/.local/state/hermes/turn_watchdog.log). Each record carries ts, reason (cache_loop / stall / interrupt_orphan), turn_id, model, iteration, max_iterations, elapsed_s, and a truncated session_key. The Discord ephemeral message is for the user; the audit log is for post-mortem.

The local change is 5 files: one new module, one new test file, one new repro script, plus surgical edits in gateway/run.py and run_agent.py. Test coverage: 20 new unit cases covering both kill rules, the throttle, the audit log (including a swallowed-failure case so a broken disk cannot crash the turn loop), the cached-detection precedence, and an end-to-end stuck-turn integration test that drives a FakeAgent mirroring the pattern in tests/gateway/test_gateway_inactivity_timeout.py. The 74 existing gateway tests in tests/gateway/test_busy_session_ack.py, test_gateway_inactivity_timeout.py, test_agent_cache.py, test_session_reset_notify.py, and test_stuck_loop.py still pass on top of the change.

The repro is reproducible without a live gateway via a small script that simulates heartbeats against a FakeAgent whose get_activity_summary() returns the stuck-cached payload. Before the fix the simulation emits 60 heartbeats over 30 minutes with no kill. After the fix the watchdog kills at 120s with reason cache_loop and writes one audit-log record.

Happy to open a PR from a fork if you think this is interesting. Local branch is ready.

Environment

• Hermes Agent v0.11.0
• Python 3.11.15
• Linux x86_64, Ubuntu 24.04

Possibly adjacent to #10849 (which describes a different freeze mode where the agent stalls inside a terminal tool at iteration 24/90 and the existing inactivity timeout does fire after 1804s). The root cause and observable signature are different but both relate to gateway-loop stability.