Summary

BasePlatformAdapter._set_fatal_error() updates the runtime status and flips _running to False, but it does not call _notify_fatal_error(). The notify-the-gateway step is left up to each adapter author, who has to remember to do it from every mid-life failure path.

If an adapter sets a fatal error without notifying — or if its disconnect handler short-circuits on is_connected (== self._running) after _set_fatal_error already flipped _running to False — the gateway never sees the failure, _failed_platforms never gets an entry, _platform_reconnect_watcher does nothing, the python process stays alive (systemd Restart= doesn't fire), and you get a fully running gateway whose only messaging platform is dead.

How I hit it (slixmpp XMPP adapter)

1. slixmpp fires failed_auth per SASL mechanism the server rejects (see slixmpp/features/feature_mechanisms/mechanisms.py:_handle_fail — it fires the event and then calls _send_auth() to try the next mechanism).
2. ejabberd advertises SCRAM-SHA-512, SCRAM-SHA-256, PLAIN, ...; our adapter's _on_failed_auth runs on the rejected first mech, calls _set_fatal_error('auth_failed', retryable=False), which flips _running=False and writes `gateway_state.json` to `state=fatal, error_code=auth_failed`.
3. SCRAM-SHA-512 then succeeds, session_start fires, _mark_connected() flips _running=True and writes `state=connected` ... but the next per-mechanism rejection (or, in our case, a later failed_auth arriving after gateway logged ✓ xmpp connected) re-poisons _running=False.
4. Adapter keeps serving messages (sends/receives only check self._client, not _running) for ~17 hours.
5. ejabberd restarts → slixmpp disconnected → adapter's _on_disconnected checks if self.is_connected: → sees _running=False → skips _set_fatal_error('connection_lost', retryable=True) + _notify_fatal_error().
6. Gateway logs zero error lines. _failed_platforms empty. _platform_reconnect_watcher idle. systemd thinks everything is fine. Bridge silently dead.

Concrete timeline from our run (UTC):

The two adapter-side bugs are ours to fix

Yes — and we fixed them: dropped the failed_auth handler (the 45 s session_start timeout in connect() already covers real auth failures cleanly), and re-gated _on_disconnected on a _session_established flag instead of _running. (patch)

What I'd ask upstream to change

The class invariant should make this kind of bug impossible to silently cause. Two complementary asks, either of which would have surfaced the problem:

1. Auto-notify on _set_fatal_error

Schedule `_notify_fatal_error()` from inside `_set_fatal_error` whenever a handler is registered, so adapters can't forget:

```python def _set_fatal_error(self, code: str, message: str, *, retryable: bool) -> None: self._running = False self._fatal_error_code = code self._fatal_error_message = message self._fatal_error_retryable = retryable self._write_runtime_status_safe(...) if self._fatal_error_handler is not None: try: asyncio.get_running_loop().create_task(self._notify_fatal_error()) except RuntimeError: pass # called outside loop (e.g. lock_conflict during sync init) ```

This collapses a two-step API (`set` + `notify`) into one, and means a future plugin can't leave the gateway in the dark by forgetting one half.

2. Periodic reconciliation in the gateway

Today the gateway only consults _failed_platforms for retries. Add a slow loop (every N s) that also walks self.adapters and queues any adapter where has_fatal_error is True or is_connected is False — basically "trust the adapter's own state more than its notify discipline."

3. (Defense in depth, optional) systemd watchdog

`Type=notify` + `WatchdogSec=120` with periodic `sd_notify(WATCHDOG=1)` gated on "at least one required platform connected" would have killed and restarted the zombie regardless of the in-process bug. There's already a Stale systemd unit detected warning surfacing the gateway's awareness of its unit config, so this seems within scope.

(1) is by far the simplest and has the highest leverage.

Environment

• hermes-agent 0.13.0 (release v2026.5.7), Python 3.12.13
• slixmpp 1.13.2
• NixOS systemd unit (no Type=notify)
• Plugin vendored locally; no upstream xmpp-platform in this repo

Happy to send a PR for (1) if it sounds reasonable.