hermes - 💡(How to fix) Fix [Bug]: Gateway Memory Leak — aiohttp ClientSession Never Closed → 6.2 GB OOM

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Error Message

Before (leak):

session = aiohttp.ClientSession() try: async with session.post(...) as resp: ... except RemoteProtocolError: # retry — old session leaked!

After (no leak):

async with aiohttp.ClientSession() as session: async with session.post(...) as resp: ...

Root Cause

Root Cause Analysis (optional)

Code Example

# After fresh restart — 312 MB
root  1684233  6.8  0.6 2022948 312296 ?  Ssl  13:26  0:52 python -m hermes_cli.main gateway run

# Before restart — 6.2 GB peak
systemd: Consumed 54min 34.701s CPU time, 6.2G memory peak, 0B memory swap peak.

---

ERROR asyncio: Unclosed client session    — 39 occurrences
ERROR asyncio: Unclosed connector         — 36 occurrences
OpenRouter stream drop (RemoteProtocolError)20 occurrences

---

Report       https://paste.rs/iyFjm
agent.log    https://paste.rs/gtu7m
gateway.log  https://paste.rs/ZePso

---



---

# Before (leak):
session = aiohttp.ClientSession()
try:
    async with session.post(...) as resp:
        ...
except RemoteProtocolError:
    # retry — old session leaked!

# After (no leak):
async with aiohttp.ClientSession() as session:
    async with session.post(...) as resp:
        ...

---

session = aiohttp.ClientSession()
try:
    async with session.post(...) as resp:
        ...
finally:
    await session.close()
RAW_BUFFERClick to expand / collapse

Bug Description

The Hermes Gateway process leaks aiohttp.ClientSession objects. When OpenRouter drops a streaming connection (RemoteProtocolError: peer closed connection without sending complete message body), the ClientSession and its TCPConnector are never closed. Each leaked session holds stream buffers and connection state in m>

The gateway restarts fine (thanks to RestartForceExitStatus=75), but the cycle repeats every ~12 hours.

Steps to Reproduce

  1. Run hermes gateway run (or systemd service) with OpenRouter as provider
  2. Wait for OpenRouter stream drops (RemoteProtocolError: peer closed connection without sending complete message body). These happen naturally ~20 times/day on our instance.
  3. Observe ERROR asyncio: Unclosed client session in logs after each drop
  4. Watch RSS grow: starts at ~300 MB, climbs to 6+ GB over ~10 hours
  5. Eventually OOM or systemd kill

Expected Behavior

aiohttp.ClientSession should be properly closed after each stream attempt (success or failure). Memory should remain stable, not grow 20× over a day.

Actual Behavior

# After fresh restart — 312 MB
root  1684233  6.8  0.6 2022948 312296 ?  Ssl  13:26  0:52 python -m hermes_cli.main gateway run

# Before restart — 6.2 GB peak
systemd: Consumed 54min 34.701s CPU time, 6.2G memory peak, 0B memory swap peak.

Log evidence (single day):

ERROR asyncio: Unclosed client session    — 39 occurrences
ERROR asyncio: Unclosed connector         — 36 occurrences
OpenRouter stream drop (RemoteProtocolError) — 20 occurrences

Total Unclosed client session in syslog: 148 (across all time).

Affected Component

Gateway (Telegram/Discord/Slack/WhatsApp), Agent Core (conversation loop, context compression, memory)

Messaging Platform (if gateway-related)

Telegram, N/A (CLI only)

Debug Report

Report       https://paste.rs/iyFjm
agent.log    https://paste.rs/gtu7m
gateway.log  https://paste.rs/ZePso

Operating System

Ubuntu 24.04.4 LTS

Python Version

Python 3.11.15

Hermes Version

0.14.0

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

The retry logic in run_agent.py for stream drops creates new aiohttp.ClientSession on each attempt without closing the previous one. When OpenRouter drops the connection mid-stream:

  1. httpx or aiohttp raises RemoteProtocolError
  2. The retry loop creates a new ClientSession for the next attempt
  3. The old session is never await session.close()-ed
  4. Each leaked session retains its TCPConnector, stream buffers, and response handler objects
  5. At ~20 drops/day with stream buffers averaging hundreds of KB each, memory grows ~200-400 MB per drop

The problem compounds: retries 2/3 → 3/3 mean up to 3 sessions leaked per single stream failure.

Probable location: run_agent.py — stream handling in the conversation loop, or gateway/session.py — where agent sessions are created per incoming message.

Proposed Fix (optional)

All aiohttp.ClientSession usage should use async with context managers:

# Before (leak):
session = aiohttp.ClientSession()
try:
    async with session.post(...) as resp:
        ...
except RemoteProtocolError:
    # retry — old session leaked!

# After (no leak):
async with aiohttp.ClientSession() as session:
    async with session.post(...) as resp:
        ...

Or explicitly close in finally:

session = aiohttp.ClientSession()
try:
    async with session.post(...) as resp:
        ...
finally:
    await session.close()

Additionally: verify that stream-drop retries reuse the same session rather than creating new ones. If a new session per retry is intentional, all previous sessions must be closed before creating new ones.

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING