PR #11720: fix(gateway): claim PID file immediately after old process exits on --replace

• Repository: NousResearch/hermes-agent
• Author: AJV20
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/11720

Description (problem / solution / changelog)

Fixes #11718

Problem

When two --replace invocations overlap (e.g. launchd auto-restart racing a manual restart), the second process reads an empty PID file after the first has called remove_pid_file(). It finds no existing PID, skips termination, and both instances end up running simultaneously — causing repeated Telegram polling conflicts:

Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

Fix

Write the new process PID to the file right after remove_pid_file(), before continuing initialization. Any concurrent --replace invocation will then see the current process PID and correctly terminate it instead of finding an empty file and proceeding unchecked.

remove_pid_file()
# Claim the PID file immediately so any concurrent --replace
# invocation sees this process rather than an empty file and
# skips the termination step, preventing duplicate instances.
from gateway.status import write_pid_file as _write_pid_now
_write_pid_now()

Changed files

• gateway/run.py (modified, +5/-0)

PR #11734: fix(gateway): prevent --replace race condition causing multiple instances

• Repository: NousResearch/hermes-agent
• Author: opriz
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/11734

Description (problem / solution / changelog)

Summary

Fixes #11718.

When starting the gateway with --replace, concurrent invocations could leave multiple instances running simultaneously. This triggered Telegram (and other platform) polling conflicts, causing the bot to become unresponsive.

Root Cause

write_pid_file() in gateway/status.py used a plain file overwrite. Two racing --replace processes could both proceed through the termination-wait logic, then the second process would silently overwrite the first process's PID record, leaving both instances alive.

Changes

• gateway/status.py: write_pid_file() now uses atomic O_CREAT | O_EXCL file creation. If the PID file already exists, it raises FileExistsError, ensuring exactly one process wins the race.
• gateway/run.py: Before writing the PID file, performs a defensive re-check via get_running_pid(). Also catches FileExistsError from write_pid_file(). In both cases, stops the runner and returns False so the process exits cleanly (non-zero exit code for systemd auto-restart compatibility).

Reproduction

Run two gateway --replace invocations concurrently (e.g. via systemd restart overlap or manual double-start):

# Terminal 1
hermes gateway run --replace &

# Terminal 2 (within ~1s)
hermes gateway run --replace &

Before the fix: both processes write to gateway.pid and start polling Telegram, causing:

Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

After the fix: the second process detects the race, logs an error, and exits before starting any platform adapters.

Test Plan

Existing tests

pytest tests/gateway/test_status.py -q                    # 14 passed
pytest tests/gateway/test_runner_startup_failures.py -q   # included in 75 passed
pytest tests/hermes_cli/test_update_gateway_restart.py -q # 34 passed
pytest tests/gateway/test_telegram_conflict.py -q         # 6 passed

Concurrent race simulation

A standalone script was used to verify the atomicity under real OS-level concurrency:

import multiprocessing
import os
import tempfile
from pathlib import Path
from gateway import status

def racer(pid_file, queue):
    status._get_pid_path = lambda: Path(pid_file)
    try:
        status.write_pid_file()
        queue.put(("ok", os.getpid()))
    except FileExistsError:
        queue.put(("race_lost", os.getpid()))

pid_file = Path(tempfile.mkdtemp()) / "gateway.pid"
q = multiprocessing.Queue()
p1 = multiprocessing.Process(target=racer, args=(str(pid_file), q))
p2 = multiprocessing.Process(target=racer, args=(str(pid_file), q))
p1.start(); p2.start()
p1.join(); p2.join()

results = [q.get() for _ in range(2)]
assert len([r for r in results if r[0] == "ok"]) == 1
assert len([r for r in results if r[0] == "race_lost"]) == 1

Result: exactly 1 process succeeds, the other receives FileExistsError.

Backwards Compatibility

• write_pid_file() signature unchanged; callers that don't handle FileExistsError will see the exception propagate (which is the desired behavior for race detection).
• remove_pid_file() already guards against deleting another process's PID file, so old-process atexit handlers won't clobber the new PID record.

Changed files

• gateway/run.py (modified, +19/-2)
• gateway/status.py (modified, +22/-2)
• scripts/release.py (modified, +1/-0)