hermes - 💡(How to fix) Fix Telegram gateway: send_path_degraded stays true after successful reconnect, blocking all outbound messages

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

After a transient network interruption, the Telegram gateway successfully reconnects polling but _send_path_degraded remains True, causing all outbound send() calls to short-circuit as failures until a 60-second heartbeat probe clears the flag. If the probe also fails, the cycle repeats indefinitely — messages are received but never delivered.

Error Message

WARNING [Telegram] Telegram network error (attempt 1/10), reconnecting in 5s. Error: httpx.RemoteProtocolError: Server disconnected without sending a response. WARNING [Telegram] Send failed (attempt 1/2, retrying in 2.5s): send_path_degraded WARNING [Telegram] Send failed (attempt 2/2, retrying in 4.9s): send_path_degraded ERROR [Telegram] Failed to deliver response after 2 retries: send_path_degraded INFO [Telegram] Telegram polling resumed after network error (attempt 1) WARNING [Telegram] Send failed (attempt 1/2, retrying in 2.7s): send_path_degraded ... (repeats indefinitely)

Root Cause

In gateway/platforms/telegram.py:

  1. _handle_polling_network_error() sets _send_path_degraded = True on entry (line ~926)
  2. After successful start_polling(), the reconnect ladder resets _polling_network_error_count but does NOT clear _send_path_degraded
  3. _send_path_degraded is only cleared inside _verify_polling_after_reconnect() (line ~1024), which runs 60 seconds later via a deferred probe
  4. If the heartbeat probe fails (e.g., the send-side httpx pool is also stale), it re-enters the reconnect ladder, which sets _send_path_degraded = True again
  5. This creates a cycle where polling is healthy but sends are permanently blocked

The send() method at line ~1816 short-circuits immediately when the flag is True:

if getattr(self, "_send_path_degraded", False):
    return SendResult(success=False, error="send_path_degraded", retryable=True)

Additionally, _drain_polling_connections() only resets the polling request's connection pool (_request[0]). The general request pool (_request[1]) used for send_message/edit_message is never drained, so send-side connections may remain in a half-closed state.

Fix Action

Workaround

Restart the gateway process. No in-process recovery is possible once the cycle begins.

Code Example

if getattr(self, "_send_path_degraded", False):
    return SendResult(success=False, error="send_path_degraded", retryable=True)

---

WARNING [Telegram] Telegram network error (attempt 1/10), reconnecting in 5s. Error: httpx.RemoteProtocolError: Server disconnected without sending a response.
WARNING [Telegram] Send failed (attempt 1/2, retrying in 2.5s): send_path_degraded
WARNING [Telegram] Send failed (attempt 2/2, retrying in 4.9s): send_path_degraded
ERROR   [Telegram] Failed to deliver response after 2 retries: send_path_degraded
INFO    [Telegram] Telegram polling resumed after network error (attempt 1)
WARNING [Telegram] Send failed (attempt 1/2, retrying in 2.7s): send_path_degraded
... (repeats indefinitely)
RAW_BUFFERClick to expand / collapse

Summary

After a transient network interruption, the Telegram gateway successfully reconnects polling but _send_path_degraded remains True, causing all outbound send() calls to short-circuit as failures until a 60-second heartbeat probe clears the flag. If the probe also fails, the cycle repeats indefinitely — messages are received but never delivered.

Root Cause

In gateway/platforms/telegram.py:

  1. _handle_polling_network_error() sets _send_path_degraded = True on entry (line ~926)
  2. After successful start_polling(), the reconnect ladder resets _polling_network_error_count but does NOT clear _send_path_degraded
  3. _send_path_degraded is only cleared inside _verify_polling_after_reconnect() (line ~1024), which runs 60 seconds later via a deferred probe
  4. If the heartbeat probe fails (e.g., the send-side httpx pool is also stale), it re-enters the reconnect ladder, which sets _send_path_degraded = True again
  5. This creates a cycle where polling is healthy but sends are permanently blocked

The send() method at line ~1816 short-circuits immediately when the flag is True:

if getattr(self, "_send_path_degraded", False):
    return SendResult(success=False, error="send_path_degraded", retryable=True)

Additionally, _drain_polling_connections() only resets the polling request's connection pool (_request[0]). The general request pool (_request[1]) used for send_message/edit_message is never drained, so send-side connections may remain in a half-closed state.

Observed Behavior

From gateway.error.log during a real incident:

WARNING [Telegram] Telegram network error (attempt 1/10), reconnecting in 5s. Error: httpx.RemoteProtocolError: Server disconnected without sending a response.
WARNING [Telegram] Send failed (attempt 1/2, retrying in 2.5s): send_path_degraded
WARNING [Telegram] Send failed (attempt 2/2, retrying in 4.9s): send_path_degraded
ERROR   [Telegram] Failed to deliver response after 2 retries: send_path_degraded
INFO    [Telegram] Telegram polling resumed after network error (attempt 1)
WARNING [Telegram] Send failed (attempt 1/2, retrying in 2.7s): send_path_degraded
... (repeats indefinitely)

Polling is healthy (messages are received), but all responses fail to send.

Environment

  • Hermes Agent: current main
  • Platform: Telegram via long-polling mode
  • Proxy: TUN mode (Clash Verge / sing-box) with intermittent connectivity
  • httpx pool size: 512 (default)

Suggested Fix

Clear _send_path_degraded immediately after a successful start_polling(), or at minimum, reduce the heartbeat probe delay. The 60-second window is too long — users see the bot as unresponsive during this period.

Optionally, also drain the general request connection pool (not just the polling pool) during reconnect to prevent stale send-side connections.

Workaround

Restart the gateway process. No in-process recovery is possible once the cycle begins.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

hermes - 💡(How to fix) Fix Telegram gateway: send_path_degraded stays true after successful reconnect, blocking all outbound messages