hermes - 💡(How to fix) Fix [Bug] `hermes gateway run --replace` / `gateway restart --profile X` from shell leaks orphan dispatcher → silent concurrent kanban.db writer → corruption

Summary

hermes gateway restart --profile <name> and hermes gateway run --replace invoked from a shell on a Linux systemd host spawn long-lived Python processes that escape the systemd service cgroup and survive subsequent systemctl restart hermes-gateway.service cycles. The orphans become silent concurrent writers on ~/.hermes/kanban.db, which—combined with the systemd-managed gateway also writing—produces classic multi-writer SQLite WAL corruption (disk I/O error, then database disk image is malformed, then dispatcher quarantine).

The orphan process is invisible to systemctl status (its PID isn't MainPID), is re-parented to PID 1 (PPID=1), and has no time-based supervisor — it can live for days. I caught two of them on a production Hermes droplet running stably-misbehaving for 1d4h and 2.5h respectively before the second concurrent dispatcher tick collided badly enough to corrupt the kanban DB.

Closely related to #27041 (hermes-web-ui spawning orphan gateways on macOS launchd) and #31485 (RFC for production-grade multi-profile supervisor with adopt-orphan), but distinct: the producer here is the CLI itself (hermes gateway restart|run --replace), not the web-UI Node process and not the /restart slash command (#12875).

Environment

• Linux 6.8.0-117-generic (Ubuntu)
• systemd 255
• Hermes Agent v0.15.1 (2026.5.29) — confirmed after a 312-commit update; the bug pre-existed in v0.14.0 and still reproduces
• hermes-gateway.service installed via hermes gateway install, Restart=always
• Single droplet, root-only install (HERMES_HOME=/root/.hermes, multiple --profile per-profile services for Aion + Lanista)

Reproduction

1. Install hermes-gateway.service under systemd (Restart=always).
2. From a shell (NOT via systemd), run any of:
   • hermes gateway restart --profile argus (the case I caught — meant to be a one-shot restart of the argus per-profile gateway)
   • hermes gateway run --replace (the case the older zombie originated from — looks like manual gateway run from a shell)
3. The command appears to complete; control returns to the shell.
4. ps -o pid,ppid,etime,cmd -p <new pid> shows the child Python process still running, PPID=1, not in any systemd unit's cgroup.
5. systemctl restart hermes-gateway.service later — the orphan is not affected (it isn't in the unit's cgroup).
6. lsof /root/.hermes/kanban.db — the orphan holds multiple read/write fds on the shared kanban DB.
7. strace -e openat,write -p <orphan> confirms the orphan is actively running its own kanban dispatcher tick and O_RDWR|O_CREAT opening kanban.db and kanban.db-wal.

Evidence from production

$ ps -o pid,ppid,etime,cmd -p 2361366 3982251
    PID    PPID     ELAPSED CMD
2361366       1  1-04:09:41 /usr/local/lib/.../python -m hermes_cli.main gateway run --replace
3982251       1     02:25:29 /usr/local/lib/.../python -m hermes_cli.main gateway restart --profile argus

$ systemctl show -p MainPID --value hermes-gateway.service
4154204    # ≠ either orphan

$ lsof /root/.hermes/kanban.db
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  3982251 root   20u   REG  253,1   864256 550038 /root/.hermes/kanban.db
python  3982251 root   32u   REG  253,1   864256 550038 /root/.hermes/kanban.db
python  3982251 root   35u   REG  253,1   864256 550038 /root/.hermes/kanban.db
# (plus the systemd-managed gateway's handles — TWO independent writers)

$ strace -f -e openat,write -p 2361366 -o /tmp/orphan.strace ; grep kanban /tmp/orphan.strace
... openat("/root/.hermes/kanban.db", O_RDONLY|O_CLOEXEC) = 30
... openat("/root/.hermes/kanban.db", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 30
... openat("/root/.hermes/kanban.db-wal", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 31

Impact

This is the silent root cause of two confirmed kanban DB corruption incidents I had on 2026-05-28 and 2026-05-30. Each time the diagnostic was "disk I/O error → database disk image is malformed → dispatcher quarantine on all boards". Investigation initially pointed at agents running raw sqlite3 REINDEX on the live DB during a transient cloud-volume hiccup — that was a real contributor and I closed that hole locally — but the underlying first contributor was the orphan gateway dispatcher writing in parallel with the systemd-managed one.

The May-30 incident lost the full task_runs table (header destroyed), required Python permissive-read row-by-row rebuild (the build's sqlite3 lacks sqlite_dbpage so .recover doesn't help), and dropped one task entirely.

The closed kanban-concurrency fixes (#32424, #30445, #31618, #32532) and the SQLite busy_timeout PRAGMA help against one extra writer hitting a momentarily locked DB, but don't help against two long-lived dispatchers that each genuinely believe they own the file — both pass busy_timeout happily and then race on WAL frames.

Workaround

After confirming an orphan is alive, kill -TERM <pid> (no shell loop needed). To prevent recurrence: never run hermes gateway run --replace or hermes gateway restart --profile X from a shell on a systemd-managed host. Always systemctl restart hermes-gateway.service (or the per-profile service).

This is brittle — there's no warning when a user does the wrong thing, and the failure mode is silent until the kanban DB corrupts hours/days later.

Suggested fix directions

In rough order of robustness:

1. CLI guard: Before hermes gateway run --replace or hermes gateway restart --profile X actually spawns, detect that the same (HERMES_HOME, profile) is already supervised by a systemd (or launchd) unit and refuse with a clear hint: use 'systemctl restart hermes-gateway.service' instead. Allow --force to override.

2. Adopt-orphan on startup: When gateway run --replace does proceed, before forking, scan for any other process matching the (HERMES_HOME, profile) signature (e.g., via a pid file at $HERMES_HOME/gateway.pid or matching argv + env) and either kill it or refuse to start. This is the spirit of #31485 narrowed to "kill the same-identity sibling, don't accumulate."

3. Pid-file + flock-based single-writer guarantee: Wrap the kanban dispatcher loop in a flock on $HERMES_HOME/kanban.dispatch.lock so even if a second process spawns, only one of them gets to write. This is defense in depth and would have caught the corruption regardless of how the orphan got there.

4. Periodic supervisor watchdog (already proposed in #32574): A platform-liveness loop that notices "I'm not MainPID and a same-identity process is" and self-exits.

Happy to provide more reproduction detail or test a candidate patch on the same droplet.

Related: #27041, #12875, #31485, #32574, #34966, and the closed kanban-corruption cluster #32424 / #30445 / #31618 / #32532.