Telegram Updater goes silent forever after a single network blip; reconnect ladder swallows the cleanup TimedOut and never re-fires

Summary

In gateway/platforms/telegram.py, _handle_polling_network_error exhausts a 10-attempt exponential ladder before declaring the adapter retryable-fatal. In practice, a single transient getUpdates 502 / Bad Gateway can trip the FIRST reconnect attempt into a state where:

1. await self._app.updater.stop() raises telegram.error.TimedOut from python-telegram-bot's _get_updates_cleanup (the "Suppressing error to ensure graceful shutdown" path).
2. The surrounding try / except Exception: pass in _handle_polling_network_error swallows the exception and proceeds to await self._app.updater.start_polling(...).
3. start_polling() returns without raising — but the underlying long-poll never actually resumes consuming updates.
4. No further error callback fires (polling is "alive" per Updater.running but quietly stuck), so the reconnect ladder never advances past attempt 1, the fatal-error path is never reached, and the gateway process keeps running indefinitely without any working Telegram polling.

The container looks healthy at every external surface — process is up, logs report only the initial "reconnecting in 5s, attempt 1/10" line, and Updater.running is True — but Telegram messages are silently dropped at the polling layer for hours/days until manually restarted.

Repro

Operator-context: running Hermes inside a Docker container with hermes gateway run as PID 1. No external supervisor; the container's restart policy is no. Pinned to tag v2026.4.16.

Observed in the wild after a transient Telegram API blip ~5 hours into a container's lifetime. Synthetic repro:

1. Boot the gateway with Telegram polling enabled.
2. Briefly fault Telegram egress: route the bot's outbound traffic through a transparent proxy that returns 502 Bad Gateway for getUpdates for ~30 seconds, then restore.
3. Observe the log sequence below.
4. Wait several minutes. Updater.running is True. Send a DM to the bot. The bot never replies, and no further log lines appear.

Observed log (sanitized)

Times are minutes in container-local epoch; <TS> placeholders.

<TS+0:00> WARNING gateway.platforms.telegram: [Telegram] Telegram network error, scheduling reconnect: Bad Gateway
<TS+0:00> WARNING gateway.platforms.telegram: [Telegram] Telegram network error (attempt 1/10), reconnecting in 5s. Error: Bad Gateway
<TS+0:25> ERROR telegram.ext.Updater: Error while calling `get_updates` one more time to mark all fetched updates. Suppressing error to ensure graceful shutdown. When polling for updates is restarted, updates may be fetched again. Please adjust timeouts via `ApplicationBuilder` or the parameter `get_updates_request` of `Bot`.
<TS+0:25> telegram.error.TimedOut: Timed out
[silence — no further log lines for 11 hours, until the container is manually restarted]

Notably absent (and expected if the ladder were progressing):

• No [Telegram] Telegram polling resumed after network error (attempt 1) (would indicate clean recovery).
• No [Telegram] Telegram polling reconnect failed: ... followed by reconnecting in 10s, attempt 2/10 (would indicate the ladder advancing).
• No Telegram polling could not reconnect after 10 network error retries. Restarting gateway. (the fatal-error path, which the ladder is supposed to land on if recovery genuinely fails).

This means start_polling() returned without raising on attempt 1, but the resumed polling task is stuck — likely waiting on a TCP read against an httpx connection pool that holds a stale connection from the failed long-poll, or PTB's Updater is in a state where running is True but the consumer task isn't actually fetching.

Root cause analysis

The reconnect path at gateway/platforms/telegram.py:295-310 (paraphrased):

try:
    if self._app and self._app.updater and self._app.updater.running:
        await self._app.updater.stop()
except Exception:
    pass

try:
    await self._app.updater.start_polling(
        allowed_updates=Update.ALL_TYPES,
        drop_pending_updates=False,
        error_callback=self._polling_error_callback_ref,
    )
    logger.info("[%s] Telegram polling resumed after network error ...")
    self._polling_network_error_count = 0
except Exception as retry_err:
    ...

Two issues compound:

1. Cleanup-failure invisibility. When stop() raises (e.g. _get_updates_cleanup hits a TimedOut), the bare except Exception: pass discards information that the cleanup didn't complete. PTB's Updater may be left in a state where running is internally true but its consumer task is wedged.
2. No probe that polling is actually working. start_polling() is treated as a confirmation of recovery. There is no subsequent verification that getUpdates calls are succeeding (e.g. an offset watermark advancing, a last_get_updates_at heartbeat, or a synthetic probe of the Updater's queue). When start_polling succeeds-on-paper but the polling task is wedged, no further error callback fires and the reconnect ladder never advances. The fatal-error path that an external supervisor relies on is never reached.

Independently — the same wedging is also a problem for operators running without an external supervisor (Docker without a restart policy, dev machines, etc.), since even the eventual fatal-error path depends on _polling_error_callback firing again, which it doesn't when polling is stuck rather than erroring.

Suggested fix shape

In _handle_polling_network_error, after start_polling() returns successfully, schedule a deferred liveness check that runs ~HEARTBEAT_SECONDS later (suggest 60s) and verifies that polling has made progress since the reconnect. Concretely:

• Capture progress_marker_before = self._polling_progress_marker (an offset / counter incremented by the gateway each time it hands a real update off to the dispatcher).
• After HEARTBEAT_SECONDS, compare to progress_marker_after.
• If unchanged AND no new error callback has fired in the interim, treat the reconnect as failed: increment _polling_network_error_count, do a hard rebuild of the Updater (re-instantiate the underlying Application / Bot / httpx client to drop stale connections), and reschedule _handle_polling_network_error with the next attempt.

This stays inside the existing reconnect abstraction (no PTB-internal coupling), turns silent wedging into a recovered-or-fatal outcome, and makes the existing MAX_NETWORK_RETRIES exhaustion path actually reachable when polling is stuck.

A test would simulate two failure modes in one repro:

• 502 then recovered: confirms recovery happens normally and the heartbeat doesn't false-positive on healthy polling.
• 502 then quietly-wedged-Updater: confirms the heartbeat detects the wedge and the ladder advances, eventually firing the fatal-error path.

Happy to send a PR with the heartbeat probe + tests if maintainers prefer that shape; want to validate the approach before investing in the implementation.

Environment

• Hermes tag: v2026.4.16
• Python: 3.11 (slim image)
• python-telegram-bot: bundled with Hermes installer
• Runtime: containerized, hermes gateway run as PID 1, no external supervisor.
• Triggered by: a transient Telegram API 502 / Bad Gateway against getUpdates.

What we've ruled out

• Token validity: getMe returns 200 throughout the silent window.
• Webhook conflict: getWebhookInfo shows no webhook and pending_update_count: 0 (i.e. no other consumer is stealing updates; the polling task is just not consuming).
• Container-level network: outbound curl https://api.telegram.org/bot.../getMe from inside the still-running container returns 200 in <1s during the silent window — egress is healthy, the wedge is at the Updater layer.
• Container OOM / restart: container shows running, 0 restarts, OOMKilled false.