hermes - 💡(How to fix) Fix hermes update restarts gateway during update, causing OOM on low-memory servers

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Root Cause

The update flow runs pip install + git pull (memory-intensive) while the gateway is still running (~500 MB Python + ~1 GB Node/TUI), then restarts the gateway — which loads new modules while the old process is still draining. On a 1.6 GB server, the combined memory usage exceeds physical RAM and triggers OOM.

Fix Action

Fix / Workaround

A watchdog script had to be used as a workaround to ensure the gateway came back up.

Code Example

oom-kill: task_memcg=hermes-gateway.service, pid=74547 (node)
Out of memory: Killed process 74547 (node) total-vm:12.9GB, anon-rss:976MB
hermes-gateway.service: Failed with result 'oom-kill'

---

1. systemctl --user stop hermes-gateway
2. git pull + pip install
3. systemctl --user start hermes-gateway
RAW_BUFFERClick to expand / collapse

Bug Description

hermes update restarts the gateway service as part of its update flow. On a low-memory server (1.6 GB RAM), this causes an Out-of-Memory kill because both the update process (git pull + pip install) and the gateway process (Python + Node/TUI) compete for memory simultaneously.

Environment

ItemValue
ServerAlibaba Cloud lightweight instance
RAM1.6 GB + 2 GB swap
OSUbuntu 24.04 x86_64
CPU2-core Intel Xeon Platinum
Hermes versionv0.12.0 → v0.13.0

What Happened

  1. Ran hermes update --yes from a terminal session
  2. Update completed git pull + pip install successfully
  3. Update attempted to restart hermes-gateway.service
  4. Gateway restart triggered OOM — kernel killed the node process (TUI/Ink) inside the gateway cgroup:
oom-kill: task_memcg=hermes-gateway.service, pid=74547 (node)
Out of memory: Killed process 74547 (node) total-vm:12.9GB, anon-rss:976MB
hermes-gateway.service: Failed with result 'oom-kill'
  1. The update command was interrupted (exit code 130 / SIGINT)
  2. Gateway eventually restarted via systemd, but the update session was lost

A watchdog script had to be used as a workaround to ensure the gateway came back up.

Root Cause

The update flow runs pip install + git pull (memory-intensive) while the gateway is still running (~500 MB Python + ~1 GB Node/TUI), then restarts the gateway — which loads new modules while the old process is still draining. On a 1.6 GB server, the combined memory usage exceeds physical RAM and triggers OOM.

Proposed Fix

Option A (preferred): Stop the gateway before the heavy update steps (git pull, pip install), then start it after. This avoids memory overlap:

1. systemctl --user stop hermes-gateway
2. git pull + pip install
3. systemctl --user start hermes-gateway

Option B: Detect available memory before restarting. If memory is low, stop the gateway first, then restart after dependencies are updated.

Option C: At minimum, document this behavior and add a caveat for servers with < 2 GB RAM.

Impact

Users running Hermes on lightweight cloud instances (1-2 GB RAM) will experience gateway downtime after hermes update. Without a watchdog or manual intervention, the gateway may not recover.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

hermes - 💡(How to fix) Fix hermes update restarts gateway during update, causing OOM on low-memory servers