hermes - 💡(How to fix) Fix Unify MCP HTTP recovery around connection lifecycle and error classification [1 participants]

Summary

Hermes currently handles several MCP HTTP recovery cases as separate fixes:

• expired server-side MCP sessions
• terminated HTTP MCP streams
• stale idle HTTP connections
• timeout / empty-message failures after long-lived sessions
• protocol or endpoint mismatches that should not be retried as transport failures

These issues appear related at the user level: MCP tools stop working until Hermes is restarted or the MCP connection is manually refreshed.

I would like to propose an umbrella architecture issue before submitting a staged PR series, so the recovery model can be agreed on before implementation.

Related Issues / PRs

• #13383 — MCP server session expires during long-running gateway, no auto-reconnect
• #15125 — merged fix for Invalid or expired session
• #13795 — open PR for Session terminated transport recovery
• #17662 — open PR for TimeoutError after stale StreamableHTTP sessions
• #17003 — open issue for MCP HTTP connections going stale after idle periods
• #17244 — related but different: SSE discovery / endpoint mismatch, not a normal reconnect case

Problem

The current recovery behavior is spread across MCP operation handlers and specific error-message checks.

That makes each new failure mode require another targeted patch, for example:

• Invalid or expired session
• Session terminated
• TimeoutError
• empty error messages after idle disconnects
• stream closed
• connection reset
• server disconnected

This works for individual cases, but it does not provide a clear recovery model for long-running MCP HTTP connections.

It also increases the risk of two opposite mistakes:

1. Not reconnecting when the MCP transport/session is genuinely dead.
2. Reconnecting when the error is actually an auth failure, tool/business error, or protocol/configuration problem.

Proposed Direction

Introduce a unified MCP runtime recovery layer with clear separation of responsibilities.

Suggested boundaries:

All MCP operations should use the same executor:

• tool calls
• list resources
• read resource
• list prompts
• get prompt
• tool discovery / refresh where applicable

Recovery Model

Suggested behavior:

Important: classification should remain allow-list based. A broad rule like if "session" in message: reconnect would be too risky.

Examples of Recoverable Markers

Session identity errors:

Invalid or expired session
expired session
session expired
session not found
unknown session
invalid or missing session id

Transport-dead errors:

Session terminated
stream closed
EndOfStream
connection closed
connection reset
server disconnected

Timeout / stale cases:

TimeoutError
empty error after idle StreamableHTTP disconnect

These should remain separate from auth failures such as 401 Unauthorized.

Non-goals

This proposal should not:

• replace the existing OAuth recovery path
• trigger reauth for normal transport/session failures
• retry arbitrary tool/business errors
• hide permanent configuration problems
• introduce unbounded retries
• make every error mentioning “session” reconnect automatically

The retry policy should stay conservative: reconnect and retry once, then surface the error if recovery fails.

Suggested PR Series

I propose splitting this into small PRs:

1. Add centralized MCP error classification without changing behavior.
2. Add a unified MCP operation executor and route existing MCP operations through it.
3. Add a connection manager with single-flight reconnect and consistent ready waiting.
4. Extend recovery coverage for Session terminated, timeout/stale, and transport-dead cases.
5. Add idle health monitoring / pre-operation stale connection handling.
6. Add protocol/endpoint diagnostics so cases like SSE discovery mismatch are not treated as reconnectable transport failures.

Acceptance Criteria

• Invalid or expired session still reconnects and retries once.
• Session terminated reconnects and retries once.
• stale/idle HTTP transport failures reconnect and retry once when safely classified.
• successful retry resets the relevant server failure state.
• OAuth/auth failures remain separate from transport/session recovery.
• unrelated MCP tool errors do not trigger reconnect.
• protocol/endpoint mismatches produce clear diagnostics instead of retry loops.
• recovery behavior is shared across:
  • tool calls
  • list resources
  • read resource
  • list prompts
  • get prompt
• concurrent failures on the same server do not trigger multiple overlapping reconnects.