hermes - 💡(How to fix) Fix [Bug]: Tool-progress consumer drops last event when __reset__ arrives during throttle window [1 pull requests]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Error Message

elif isinstance(raw, tuple) and len(raw) >= 1 and raw[0] == "reset": # Flush any pending unsent progress before clearing state, so the # last tool-progress event doesn't get silently dropped when it # arrived inside the throttle window. if can_edit and progress_lines and progress_msg_id: try: await _edit_progress_message(progress_msg_id, _progress_text(progress_lines)) except Exception: pass progress_msg_id = None progress_lines = [] last_progress_msg[0] = None repeat_count[0] = 0 continue

Root Cause

Root Cause Analysis (optional)

Fix Action

Fixed

Code Example

N/A - debug contains PII and I won't share it publicly like this

---



---

_remaining = _PROGRESS_EDIT_INTERVAL - (_now - _last_edit_ts)
  if _remaining > 0:
      await asyncio.sleep(_remaining)
      continue

---

elif isinstance(raw, tuple) and len(raw) >= 1 and raw[0] == "__reset__":
      progress_msg_id = None
      progress_lines = []
      last_progress_msg[0] = None
      repeat_count[0] = 0
      continue

---

elif isinstance(raw, tuple) and len(raw) >= 1 and raw[0] == "__reset__":
      # Flush any pending unsent progress before clearing state, so the
      # last tool-progress event doesn't get silently dropped when it
      # arrived inside the throttle window.
      if can_edit and progress_lines and progress_msg_id:
          try:
              await _edit_progress_message(progress_msg_id, _progress_text(progress_lines))
          except Exception:
              pass
      progress_msg_id = None
      progress_lines = []
      last_progress_msg[0] = None
      repeat_count[0] = 0
      continue
RAW_BUFFERClick to expand / collapse

Bug Description

The tool-progress accumulator in gateway/run.py silently drops the last tool event in a turn when that event arrives during the edit-throttle window (_PROGRESS_EDIT_INTERVAL = 1.5s), and no further tool events follow. The event is dequeued and appended to progress_lines, but it is never flushed to the platform: the consumer triggers an edit only when a new event is dequeued after the throttle expires. When the agent's final response bubble lands, the gateway queues reset, which clears progress_lines before any flush runs — so the pending edit is lost.

Net effect: a user firing N tools sees N-1 (or fewer) lines in the progress bubble, even though every tool ran successfully and is recorded in the audit log. The agent's own summary reports all N calls, making the discrepancy visible to the user.

Steps to Reproduce

  1. Configure hermes-agent with a chat platform that supports message edits (e.g. Telegram) and display.tool_progress: all.
  2. Start a session and write a prompt for the agent to fire multiple tool calls in a single turn where:
  • At least one identical tool name is called consecutively (e.g. skill_view: "obsidian" then skill_view: "memory"), and
  • The last call's start lands within _PROGRESS_EDIT_INTERVAL (1.5s) of the previous progress edit. [Example: ask the agent to call cronjob: list, two terminal echoes, read_file on any small file, search_files, then skill_view: "memory" followed by skill_view: "hermes-agent".]
  1. Send the prompt and watch the tool-progress bubble update as tools fire.

Expected Behavior

Every tool call that runs in the turn appears as its own line in the tool-progress bubble, in the order it ran. For the repro above, the bubble ends with seven lines — one per tool call — matching the agent's own end-of-turn summary.

Actual Behavior

The bubble shows N−1 lines (or fewer). The missing entries are the tool calls whose progress events arrive during the throttle window after the previous edit and are not followed by another tool event.

In the repro: the bubble ends with six lines, and the final skill_view is absent, even though the agent's end-of-turn summary confirms seven tool calls completed successfully. From the user's perspective, the bubble looks complete, and the missing tool is invisible — the only signal is the agent reporting N calls while the bubble shows N−1.

Affected Component

Gateway (Telegram/Discord/Slack/WhatsApp)

Messaging Platform (if gateway-related)

Telegram

Debug Report

N/A - debug contains PII and I won't share it publicly like this

Operating System

Ubuntu 26.04

Python Version

3.11.15

Hermes Version

Hermes Agent v0.14.0 (2026.5.16)

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

The progress consumer in gateway/run.py has two distinct flush paths and a gap between them.

Path 1 — Normal hot loop: Every iteration calls progress_queue.get_nowait(), classifies the message (dedup / reset /plain msg), appends to progress_lines, then runs the throttle check:

_remaining = _PROGRESS_EDIT_INTERVAL - (_now - _last_edit_ts)
if _remaining > 0:
    await asyncio.sleep(_remaining)
    continue

The continue returns to the loop top, which immediately calls get_nowait() again. An edit only fires below this throttle gate when a fresh event is dequeued AND the throttle window has expired. If the throttle was the reason we slept, the just-appended line waits for the next event to push it out.

Path 2 — Cancellation drain: The except asyncio.CancelledError handler at end-of-turn drains the queue, appends any leftovers, and does a final _edit_progress_message(...). This is the safety net for any unsent tail.

The race: When the last tool of the turn finishes, its progress event lands in the queue but is processed inside the throttle window. The consumer appends it to progress_lines, sleeps out the throttle, then loops back. No further tool events come, so the loop is now just polling an empty queue.

Meanwhile the agent finishes generating the final response. Before cancellation fires, gateway/run.py queues reset so the next tool-progress bubble doesn't keep editing above the final content message. The consumer dequeues reset and runs:

elif isinstance(raw, tuple) and len(raw) >= 1 and raw[0] == "__reset__":
      progress_msg_id = None
      progress_lines = []
      last_progress_msg[0] = None
      repeat_count[0] = 0
      continue

This clears progress_lines without flushing it first. The pending tail line is gone. By the time cancellation fires and the drain handler runs, there's nothing left to send.

Why the race is reliable, not rare: The throttle window is 1.5s, which is wider than the runtime of fast tools like skill_view, cronjob list, or trivial terminal echoes. Any turn whose last tool is fast and follows a recent edit is in the danger zone. Slow last-tool turns mask the bug because the throttle has time to expire and the user-visible content bubble landing happens later.

Why no error surfaces: Every path returns "success" — the consumer believes the line was queued and processed, the reset handler is intentional, and the drain handler has no leftover to log. The bubble shows N−1 lines and nothing in the logs flags an anomaly.

Proposed Fix (optional)

Suggested fix (one line in the reset handler):

  elif isinstance(raw, tuple) and len(raw) >= 1 and raw[0] == "__reset__":
      # Flush any pending unsent progress before clearing state, so the
      # last tool-progress event doesn't get silently dropped when it
      # arrived inside the throttle window.
      if can_edit and progress_lines and progress_msg_id:
          try:
              await _edit_progress_message(progress_msg_id, _progress_text(progress_lines))
          except Exception:
              pass
      progress_msg_id = None
      progress_lines = []
      last_progress_msg[0] = None
      repeat_count[0] = 0
      continue

This mirrors what the cancellation drain handler already does at end-of-turn, but applies it at the moment we know the current progress bubble is being closed off — which is precisely when an unflushed tail line would otherwise be lost.

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING