hermes - 💡(How to fix) Fix feat: limit CPU usage of Hermes-spawned scripts to prevent server hangs

Problem

Scripts and background tasks spawned by Hermes (e.g. eval benchmarks, long-running terminal() calls, execute_code blocks) can consume 100% CPU and hang the entire server — making the Hermes WebUI unresponsive and blocking all other Hermes operations.

Observed behaviour: A benchmark script looping on HTTP requests pegged the CPU, caused the server to become unresponsive, and ultimately required a full reboot.

Proposed Solutions

1. Auto-nice all spawned subprocesses (quick win)

Wrap all terminal() and execute_code subprocess invocations with nice -n 19 so the OS preempts them freely when Hermes or the WebUI needs CPU.

# In the subprocess launch code
import os
os.nice(19)

Or prefix shell commands automatically:

cmd = f"nice -n 19 {cmd}"

2. systemd CPU quota on the Hermes process group

Add a CPUQuota=50% constraint to the hermes-gateway.service unit so the entire Hermes process tree — including all spawned children — is capped at 50% CPU by the kernel.

[Service]
CPUQuota=50%

3. resource.setrlimit in execute_code

Before running user-submitted Python, set a CPU time limit:

import resource
# Limit CPU time to 60s per execute_code call
resource.setrlimit(resource.RLIMIT_CPU, (60, 60))

4. Configurable max_cpu_percent in config.yaml

Allow operators to configure CPU limits per deployment:

execution:
  max_cpu_percent: 50      # applies to all spawned processes
  nice_level: 19           # default niceness for background tasks
  cpu_time_limit: 120      # seconds, for execute_code

5. Separate cgroup for background tasks

Run all Hermes-initiated background processes in a dedicated cgroup with CPU limits, isolating them from the gateway and WebUI processes.

Priority

• Short-term: Apply nice -n 19 to all subprocess spawns — one-line fix, immediate protection
• Medium-term: systemd CPUQuota on the service unit — kernel-enforced, no code changes needed
• Long-term: Configurable limits via config.yaml for self-hosted deployments

Environment

• Ubuntu 22.04, systemd
• Hermes WebUI on port 8787 (Tailscale)
• 2-core VM — any runaway process immediately affects the UI