PR #14332: fix(gateway): treat recycled PID with unreadable start_time as stale (#14176)

• Repository: NousResearch/hermes-agent
• Author: Sanjays2402
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/14332

Description (problem / solution / changelog)

What does this PR do?

gateway/status.py::find_gateway_pids() iterates over the PIDs recorded in ~/.hermes/gateway.lock to decide whether the gateway is "still running". For each candidate it:

1. Checks the PID is alive (os.kill(pid, 0)).
2. Compares the recorded start_time against the live process's start_time to detect PID recycling.
3. Falls back to _looks_like_gateway_process(pid) / _record_looks_like_gateway(record) heuristics.

When the recycled PID is owned by a different UID (typical on Linux when /proc/<pid>/stat is owned by another user, or under rootless container setups), _get_process_start_time returns None. The recorded-vs-live mismatch check then can't fire (current_start is None), and _looks_like_gateway_process can give a false positive on any long-lived python or hermes-related process the user happens to own. Result: the gateway thinks it's still running, refuses to start, and the user has to manually rm ~/.hermes/gateway.pid to recover.

Reporter (#14176) sees this in production with a systemd user service that restarts the gateway nightly — every few weeks the next PID up the queue lands on a recycled foreign PID, the lock file goes stale, and hermes gateway start fails with "Gateway already running".

Fix: be conservative. When the PID record carries a recorded_start but we can't read the candidate's current_start, skip the candidate (treat as stale) instead of falling through to the heuristic. Outside /proc-readable territory we don't have enough information to confirm this is the same gateway process, so prefer "no" over "maybe".

Related Issue

Fixes #14176

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• gateway/status.py (+10 / −0): in the find_gateway_pids() candidate loop, skip any PID whose recorded start_time exists but whose live start_time is unreadable. Same code path as the existing recorded-vs-live mismatch case, just covering the unreadable variant.
• tests/gateway/test_status.py (+40 / −0): one new regression case under TestGatewayPidState, test_get_running_pid_treats_recycled_pid_with_unreadable_start_time_as_stale. Monkeypatches _get_process_start_time to return None and _looks_like_gateway_process to return True (the strongest stress for the false-positive path) and asserts the PID file is cleaned and get_running_pid() returns None.

Core diff:

         recorded_start = record.get("start_time")
         current_start = _get_process_start_time(pid)
         if recorded_start is not None and current_start is not None and current_start != recorded_start:
             continue
+        # If the PID record carries a recorded start_time but we can't read
+        # the current process's start_time, the PID may have been recycled by
+        # the OS to a process the current user can't introspect (typical on
+        # Linux when /proc/<pid>/stat is owned by another UID). The downstream
+        # _looks_like_gateway_process heuristic can give a false positive in
+        # that situation — e.g. another long-lived python process — leaving
+        # a stale PID file that blocks future starts. Be conservative and
+        # skip this candidate. See #14176.
+        if recorded_start is not None and current_start is None:
+            continue

         if _looks_like_gateway_process(pid) or _record_looks_like_gateway(record):
             return pid

How to Test

Reporter-style repro on a Linux host:

1. Run the gateway, kill -9 the parent process to leave ~/.hermes/gateway.pid and ~/.hermes/gateway.lock populated with the dead PID.
2. Start a long-lived python process under a different UID (e.g. another hermes daemon under another account) that the test user can see via ps but NOT via /proc/<pid>/stat. Note its PID.
3. Edit the lock file to point at that recycled PID, keeping the original start_time field intact.
4. Run hermes gateway start.

Before: refuses to start with "Gateway already running". After: detects the start_time mismatch is unverifiable, treats the entry as stale, cleans the lock file, and starts a fresh gateway.

Automated regression suite:

pytest tests/gateway/test_status.py::TestGatewayPidState -q

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(gateway):)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix
• I've run pytest tests/gateway/test_status.py::TestGatewayPidState -q and all tests pass (11/11)
• I've added tests for my changes
• I've tested on my platform: macOS 26.5 (arm64), Python 3.11.14 via uv

Documentation & Housekeeping

• Documentation updates — N/A (internal helper, no user-visible API change beyond bug fix)
• cli-config.yaml.example — N/A (no new config)
• CONTRIBUTING.md / AGENTS.md — N/A
• Cross-platform impact considered — change is conservative on every platform; the false positive fix matters most on Linux but doesn't regress macOS/Windows behavior
• Tool descriptions/schemas — N/A

Not in scope

• The reporter's bash script idea (a dedicated hermes gateway clean-pid command) — that's nice-to-have but a separate UX surface; the in-band fix here is the higher-impact change since it stops the bad state from forming.
• Auditing gateway.target / Restart= semantics in the example systemd unit (the issue's secondary note) — that's a docs change for docs/deploy/ that deserves its own PR.
• Hardening atexit-vs-SIGKILL paths so a kill -9 of the gateway doesn't leave a PID file behind — a real concern but out of scope for the reported bug, which is about PID-file interpretation, not creation.

Screenshots / Logs

Verification

$ python3 -m py_compile gateway/status.py tests/gateway/test_status.py
OK

$ uv run --no-project --with pytest --with pytest-xdist --with pyyaml \
       --with python-dotenv --with prompt_toolkit --with rich --with httpx \
       --with fastapi --with pydantic python -m pytest \
       tests/gateway/test_status.py::TestGatewayPidState -q
...........                                                              [100%]
11 passed in 0.51s

(11 = 10 existing + 1 new regression case, all green.)

Changed files

• gateway/status.py (modified, +10/-0)
• tests/gateway/test_status.py (modified, +40/-0)