PR #14609: fix(gateway): recover stale pid files and release evicted clients

• Repository: NousResearch/hermes-agent
• Author: LeonSGP43
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/14609

Description (problem / solution / changelog)

Summary

• recover startup when a stale gateway.pid is still present after the runtime lock is acquired
• release provider clients when _evict_cached_agent() drops a cached agent
• add regression coverage for both recovery paths

Testing

• python3 -m pytest -o addopts='' -q tests/gateway/test_agent_cache.py -k 'evict' tests/gateway/test_runner_startup_failures.py -k 'stale_pid or start_gateway'
• python3 -m pytest -o addopts='' -q tests/tools/test_zombie_process_cleanup.py -k 'evict_does_not_call_close' tests/gateway/test_session_model_reset.py -k 'evict'

Closes #14598

Changed files

• gateway/run.py (modified, +34/-6)
• gateway/status.py (modified, +13/-0)
• tests/gateway/test_agent_cache.py (modified, +26/-0)
• tests/gateway/test_runner_startup_failures.py (modified, +38/-0)

PR #14710: fix(bedrock): evict cached boto3 client on stale-connection errors

• Repository: NousResearch/hermes-agent
• Author: AndreKurait
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/14710

Description (problem / solution / changelog)

What does this PR do?

Fixes a recurring class of failures on the Bedrock Converse code path where a pooled HTTPS connection to bedrock-runtime.<region>.amazonaws.com goes stale (NAT/VPN idle timeout, server-side TCP RST, proxy idle cull), the next Converse call fails with a library-internal exception, the agent's retry loop reuses the same cached boto3 client, and every retry hits the same dead connection pool — so the only recovery is a full process restart.

This is observable in production as repeating banners like:

⚠️  API call failed (attempt N/3): AssertionError
   🔌 Provider: bedrock  Model: <inference-profile>
   🌐 Endpoint: https://bedrock-runtime.us-east-1.amazonaws.com
   📝 Error:

(The empty Error: line is because str(AssertionError()) is an empty string — a separate minor issue I'll address in a follow-up.)

Root cause

agent/bedrock_adapter.py keeps a per-region module-level cache (_bedrock_runtime_client_cache) and there is no code path that invalidates a single entry on transient connection failure. The existing reset_client_cache() nukes the whole dict, which is global-blast-radius and never wired into the retry loop anyway.

When the underlying socket is dead, botocore / urllib3 surface it as one of:

• botocore.exceptions.ConnectionClosedError / ReadTimeoutError / EndpointConnectionError / ConnectTimeoutError (all inherit HTTPClientError → BotoCoreError)
• urllib3.exceptions.ProtocolError / NewConnectionError
• A bare AssertionError raised from inside urllib3 or botocore (internal connection-pool invariant check — no message attached)

In all three cases the cached boto3 client's connection pool is poisoned, and subsequent retries against the same cached client fail identically.

Fix

Add two helpers to agent/bedrock_adapter.py:

• is_stale_connection_error(exc) — classifies exceptions that indicate dead-client/dead-socket state. Matches the botocore ConnectionError + HTTPClientError subtrees, urllib3 ProtocolError / NewConnectionError, and AssertionError raised from a frame whose module name starts with urllib3., botocore., or boto3.. Application-level AssertionError is intentionally excluded (the predicate walks the traceback to the innermost frame and checks the module).

• invalidate_runtime_client(region) — per-region counterpart to the existing reset_client_cache(). Evicts a single cached client so the next call rebuilds it (and its connection pool). Returns True if a client was evicted, False otherwise.

Wire both into the Converse call sites:

• call_converse() / call_converse_stream() in agent/bedrock_adapter.py (defense-in-depth for any future caller).
• The two direct client.converse(**kwargs) / client.converse_stream(**kwargs) call sites in run_agent.py (the paths the agent loop actually uses today).

On a stale-connection exception the client is evicted and the exception is re-raised unchanged — the agent's existing retry loop then builds a fresh client on the next attempt and recovers without requiring a process restart. Non-stale exceptions (validation errors, throttling, etc.) are unaffected.

Related Issue

No existing tracking issue for this specific failure mode. Loosely related:

• #14598 — gateway-side httpx connection leak on cache eviction. Same family of "cached client outlives its pool" problem, different code path (gateway vs Bedrock adapter). This PR does not fix #14598.
• #14680 — rebuilds the Anthropic client after Ctrl-C interrupt. Same "rebuild the client to recover" pattern, different trigger (user interrupt vs stale socket). No file overlap.

Fixes #

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• agent/bedrock_adapter.py
  • Add is_stale_connection_error(exc) -> bool — classifies botocore/urllib3 transient-connection exceptions and library-internal bare AssertionError.
  • Add invalidate_runtime_client(region) -> bool — per-region client-cache eviction.
  • Wrap call_converse() and call_converse_stream() in a try/except that invalidates the cached client on stale-connection exceptions before re-raising.
• run_agent.py
  • Wrap the two direct client.converse(**kwargs) / client.converse_stream(**kwargs) invocations with the same invalidation-on-stale pattern. The region is looked up from the active agent's configuration so the correct cache entry is evicted.
• tests/agent/test_bedrock_adapter.py — three new test classes, 14 tests total (see below).

How to Test

Automated

pytest tests/agent/test_bedrock_adapter.py -v

All 116 tests pass. The new coverage:

• TestInvalidateRuntimeClient — per-region eviction correctness, non-cached region returns False, untouched regions are preserved.
• TestIsStaleConnectionError — positive cases (botocore ConnectionClosedError, EndpointConnectionError, ReadTimeoutError; urllib3 ProtocolError; library-internal AssertionError from urllib3.* and botocore.* frames) and negative cases (application-level AssertionError, unrelated ValueError/KeyError).
• TestCallConverseInvalidatesOnStaleError — end-to-end: stale error evicts the cached client, non-stale ValidationException leaves it alone, successful call leaves it cached.

Manual (on Bedrock)

1. Configure hermes with provider: bedrock and any Bedrock model.
2. Let the session idle for longer than the upstream idle timeout (on most networks 5–15 min is enough to trigger NAT TCP cull).
3. Send any message. The first request after the idle will hit the stale socket.
4. Expected: the retry completes successfully on attempt 2/3 with a freshly-built client, no restart required.
5. Prior behavior: all 3 retries fail with identical ConnectionClosedError / AssertionError, and the session is unusable until hermes is restarted.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(bedrock):)
• I searched for existing PRs to make sure this isn't a duplicate (checked all open bedrock PRs; none touch the client-cache-invalidation code path)
• My PR contains only changes related to this fix
• I've run pytest tests/agent/test_bedrock_adapter.py -q and all tests pass
• I've added tests for my changes
• I've tested on my platform: Linux (Amazon Linux, Python 3.11)

Documentation & Housekeeping

• I've updated relevant documentation — N/A (no user-visible API change)
• I've updated cli-config.yaml.example — N/A
• I've updated CONTRIBUTING.md or AGENTS.md — N/A
• I've considered cross-platform impact — the fix is pure Python touching only botocore/urllib3 exception types; no OS-specific code
• I've updated tool descriptions/schemas — N/A

Screenshots / Logs

Repro (pre-fix) from a real session where the NAT timeout dropped the pooled connection:

⚠️  API call failed (attempt 1/3): AssertionError
   🔌 Provider: bedrock
   🌐 Endpoint: https://bedrock-runtime.us-east-1.amazonaws.com
   📝 Error:

⚠️  API call failed (attempt 2/3): AssertionError
   (... same cached client, same dead pool ...)

⚠️  API call failed (attempt 3/3): AssertionError
   (... same cached client, same dead pool ...)

# Session dead — restart required.

Post-fix: attempt 1 still fails (the stale socket is unavoidable), the cache is evicted, and attempt 2 builds a fresh client and completes normally.

Changed files

• agent/bedrock_adapter.py (modified, +130/-2)
• run_agent.py (modified, +20/-2)
• tests/agent/test_bedrock_adapter.py (modified, +207/-0)