hermes - ✅(Solved) Fix [Bug]: process poll 无限轮询导致 Gateway 卡死 [2 pull requests, 1 participants]

hermes2026-04-29 07:29:03

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

NousResearch/hermes-agent#17327•Fetched 2026-04-30 06:48:22

View on GitHub

Comments

Participants

Timeline

Reactions

Author

sugershuo

Participants

sugershuo

Timeline (top)

labeled ×5cross-referenced ×2referenced ×2closed ×1

Error Message

def poll(self, session_id: str) -> dict: session = self.get(session_id) if session is None: return {"status": "not_found", ...}

# 补充：检查进程真实状态（如果 reader 线程还没设置 exited）
if not session.exited and session.process:
    real_exit = session.process.poll()
    if real_exit is not None:  # 进程已退出
        session.exited = True
        session.exit_code = real_exit
        # 尝试读取剩余输出
        try:
            remaining = session.process.stdout.read()
            if remaining:
                session.output_buffer += remaining
        except Exception:
            pass

# 原有逻辑...

Fix Action

Fixed

Fixed by PR: fix: prevent infinite process poll loop from reader-thread race (#17327) (https://github.com/NousResearch/hermes-agent/pull/17346)
Fixed by PR: fix(process): reconcile session.exited against real child exit in poll/wait (https://github.com/NousResearch/hermes-agent/pull/17430)

PR fix notes

PR #17346: fix: prevent infinite process poll loop from reader-thread race (#17327)

Repository: NousResearch/hermes-agent
Author: HiddenPuppy
State: open | merged: False
Link: https://github.com/NousResearch/hermes-agent/pull/17346

Description (problem / solution / changelog)

Fixes #17327

Root Cause

poll() only checked session.exited to determine process status, but _reader_loop only sets exited=True in its finally block — after stdout.read() returns. When a process exits quickly (e.g., hermes update when already up-to-date), stdout.read(4096) returns empty immediately, but the reader thread may not have reached its finally block yet. poll() then returns "status": "running" forever with no output — creating an infinite polling loop that blocks the gateway.

Reported scenario: 74 consecutive poll calls over 7 minutes, each returning output_preview: "" while the process had actually exited.

Fix (two layers)

Layer 1 — Race condition guard (root cause fix)

In poll(), when session.exited is False but session.process.poll() returns a non-None status (process already exited), read any remaining stdout, set exited=True, and move the session to finished. This closes the race window between the reader thread and the poll caller.

Layer 2 — Safety cap (defense in depth)

Added MAX_POLL_COUNT = 120 to ProcessRegistry. If a session is polled 120+ times without exited being set (e.g., the reader thread genuinely hangs), poll() auto-promotes to "completed" status and logs a warning, breaking the loop.

Also tracks _poll_count, _last_output_length, and _no_output_cycles per session for future diagnostics.

Verification

Python syntax verified via ast.parse
Only tools/process_registry.py is modified (+57 lines)
No API contract changes: poll() still returns the same dict shape, just with a new guard before the existing logic

Note: The first commit in this branch removes .github/workflows/ to work around a push-scope limitation on the fork token. Only the second commit is the actual fix.

Changed files

tools/process_registry.py (modified, +53/-0)

PR #17430: fix(process): reconcile session.exited against real child exit in poll/wait

Repository: NousResearch/hermes-agent
Author: teknium1
State: closed | merged: True
Link: https://github.com/NousResearch/hermes-agent/pull/17430

Description (problem / solution / changelog)

Closes #17327.

Summary

process(action="poll") no longer loops forever when a background terminal spawns a descendant that holds stdout open after the direct child exits.

Root cause: _reader_loop at tools/process_registry.py:653 only sets session.exited = True in its finally: block, which runs when stdout.read() returns EOF. When a command like hermes update spawns a gateway systemctl restart, the direct child exits quickly but a daemon descendant inherits the stdout pipe (via os.setsid) and keeps it open. Reader blocks indefinitely, session.exited stays False, poll() returns status: running forever. Feishu user @sugershuo saw 74 consecutive running polls over 7 minutes before manually killing the gateway.

Changes

tools/process_registry.py: add _reconcile_local_exit(). On poll() / wait(), if session.exited is False, call Popen.poll() directly. If the direct child exited, drain any immediately-readable bytes non-blocking (via fcntl O_NONBLOCK toggle) and flip session.exited. Safe no-op for env/PTY sessions, already-exited sessions, and live children.
tests/tools/test_process_registry.py: 5 regression tests — orphaned-pipe scenario (reproduces #17327 exactly), wait() parity, and three no-op guards (live child, already-exited, no-Popen).

Validation

	Before	After
Orphaned-pipe poll()	`running` forever	`exited` on first poll
Live-child poll()	`running`	`running` (unchanged)
wait() on orphaned pipe	timeout	returns immediately
test_process_registry.py	42 pass	47 pass

E2E-reproduced: spawned a shell that backgrounds a long sleep and exits, started the real _reader_loop, confirmed the reader thread stays alive after the direct child exits, confirmed poll() returns status: exited on the first call with captured output preserved.

Changed files

tests/tools/test_process_registry.py (modified, +128/-0)
tools/process_registry.py (modified, +80/-0)

Code Example

⚙️ process: "poll proc_ab6f5652717" (×1)
⚙️ process: "poll proc_ab6f5652717" (×2)
...
⚙️ process: "poll proc_ab6f5652717" (×74)

---

def _reader_loop(self, session: ProcessSession):
    """Background thread: read stdout from a local Popen process."""
    first_chunk = True
    try:
        while True:
            chunk = session.process.stdout.read(4096)  # ← 阻塞读取
            if not chunk:
                break
            # ... 更新 output_buffer ...
    finally:
        session.exited = True  # ← 只有在 reader 结束后才设置
        session.exit_code = session.process.returncode
        self._move_to_finished(session)

---

# process_registry.py 第 538-546 行
proc = subprocess.Popen(
    ...
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,  # ← stderr 已合并到 stdout，不会丢失
)

---

def poll(self, session_id: str) -> dict:
    session = self.get(session_id)
    if session is None:
        return {"status": "not_found", ...}
    
    result = {
        "status": "exited" if session.exited else "running",
        # ← 缺少以下防护：
        # 1. max_poll_count 上限
        # 2. 进程真实 poll() 状态检测
        # 3. 连续无输出异常检测
    }
    return result

---

MAX_POLL_COUNT = 30  # 最多轮询 30 次
if poll_count > MAX_POLL_COUNT:
    return {"status": "error", "message": "轮询超限，请检查进程状态"}

---

NO_OUTPUT_THRESHOLD = 5  # 连续 5 次无输出
if consecutive_no_output > NO_OUTPUT_THRESHOLD:
    # 检查进程真实状态
    if process.poll() is not None:  # 进程已退出
        return {"status": "completed", "output": accumulated_output}
    else:
        return {"status": "warning", "message": "进程运行但无输出，可能卡死"}

---

def poll(self, session_id: str) -> dict:
    session = self.get(session_id)
    if session is None:
        return {"status": "not_found", ...}
    
    # 补充：检查进程真实状态（如果 reader 线程还没设置 exited）
    if not session.exited and session.process:
        real_exit = session.process.poll()
        if real_exit is not None:  # 进程已退出
            session.exited = True
            session.exit_code = real_exit
            # 尝试读取剩余输出
            try:
                remaining = session.process.stdout.read()
                if remaining:
                    session.output_buffer += remaining
            except Exception:
                pass
    
    # 原有逻辑...

RAW_BUFFERClick to expand / collapse

Hermes "poll proc" 无限轮询 Bug 报告

报告时间: 2026-04-29 15:13 CST
影响版本: Hermes Agent v0.11.0 (2026.4.23)
严重程度: 🔴 High - 导致 Gateway 卡死，用户无法正常使用

问题现象

用户在 Feishu 发送 hermes update 后，Hermes 连续输出：

⚙️ process: "poll proc_ab6f5652717" (×1)
⚙️ process: "poll proc_ab6f5652717" (×2)
...
⚙️ process: "poll proc_ab6f5652717" (×74)

持续约 7 分钟，最终用户手动 kill 进程。

根因分析

1. 对比实验

对比项	2026-04-21 正常	2026-04-29 异常
会话消息数	40 条	164 条
使用后台模式	❌ 否	✅ 是 (2次)
poll 调用次数	0 次	74 次
执行方式	前台同步	后台异步 + 轮询

2. 触发条件

2026-04-21 正常:

hermes update 拉取 160 commits，前台 120s 内完成
直接返回结果，无轮询

2026-04-29 异常:

已是最新版本，git pull 无更新
命令快速退出（<1s），但 输出未及时捕获
Hermes 误判为"进程仍在运行"，触发后台模式
开始轮询，每次返回 status: running, output_preview: ""
无限循环

3. 代码缺陷定位

时序竞争问题（已验证）:

位置: tools/process_registry.py _reader_loop() 第 653-679 行

def _reader_loop(self, session: ProcessSession):
    """Background thread: read stdout from a local Popen process."""
    first_chunk = True
    try:
        while True:
            chunk = session.process.stdout.read(4096)  # ← 阻塞读取
            if not chunk:
                break
            # ... 更新 output_buffer ...
    finally:
        session.exited = True  # ← 只有在 reader 结束后才设置
        session.exit_code = session.process.returncode
        self._move_to_finished(session)

问题: 当进程快速退出时，stdout.read() 可能立即返回空字符串。但如果 poll() 在 reader 线程设置 exited=True 之前被调用 → 返回 status: "running"。

stderr 处理（✅ 已正确实现）:

# process_registry.py 第 538-546 行
proc = subprocess.Popen(
    ...
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,  # ← stderr 已合并到 stdout，不会丢失
)

注: Bug 报告初稿中关于 "stderr 被吞" 的分析 不准确。实际问题是时序竞争。

轮询逻辑问题:

位置: tools/process_registry.py poll() 第 803-828 行

def poll(self, session_id: str) -> dict:
    session = self.get(session_id)
    if session is None:
        return {"status": "not_found", ...}
    
    result = {
        "status": "exited" if session.exited else "running",
        # ← 缺少以下防护：
        # 1. max_poll_count 上限
        # 2. 进程真实 poll() 状态检测
        # 3. 连续无输出异常检测
    }
    return result

时间线重建

时间	事件
14:39:01	用户发送 `hermes update`
14:39:xx	前台执行超时 (120s)
14:41:xx	自动切换后台模式 (`background=true`)
14:41:xx	后台进程启动，开始轮询
14:41~14:46	74 次轮询，每次 `output_preview: ""`
14:46:49	用户手动 kill 进程
14:46:52	Gateway exit 75 (TEMPFAIL)
14:47:22	systemd 重启 Gateway，残留进程导致继续轮询

建议修复

短期修复（紧急）

添加轮询上限:

MAX_POLL_COUNT = 30  # 最多轮询 30 次
if poll_count > MAX_POLL_COUNT:
    return {"status": "error", "message": "轮询超限，请检查进程状态"}

检测无输出异常:

NO_OUTPUT_THRESHOLD = 5  # 连续 5 次无输出
if consecutive_no_output > NO_OUTPUT_THRESHOLD:
    # 检查进程真实状态
    if process.poll() is not None:  # 进程已退出
        return {"status": "completed", "output": accumulated_output}
    else:
        return {"status": "warning", "message": "进程运行但无输出，可能卡死"}

中期修复（根本）

在 poll() 中增加进程真实状态检测:

def poll(self, session_id: str) -> dict:
    session = self.get(session_id)
    if session is None:
        return {"status": "not_found", ...}
    
    # 补充：检查进程真实状态（如果 reader 线程还没设置 exited）
    if not session.exited and session.process:
        real_exit = session.process.poll()
        if real_exit is not None:  # 进程已退出
            session.exited = True
            session.exit_code = real_exit
            # 尝试读取剩余输出
            try:
                remaining = session.process.stdout.read()
                if remaining:
                    session.output_buffer += remaining
            except Exception:
                pass
    
    # 原有逻辑...

后台模式触发条件优化:

不要只依赖前台超时触发后台模式
检测命令类型，hermes update 等快速命令应禁用后台模式
或设置更短的前台超时（如 30s），快速失败让用户重试

临时规避方案

用户侧:

❌ 不要在 Feishu/消息平台发送 hermes update
✅ 使用 CLI 直接执行: hermes update

原因: 消息平台的后台模式更容易触发此 bug

总结

这是 设计缺陷，不是偶发 bug：

后台模式缺少防护机制（上限、异常检测）
存在时序竞争：reader 线程设置 exited 与 poll() 调用的竞争
进程状态判断不可靠（只看 session.exited，不看真实 process.poll()）

建议优先实施短期修复，避免用户再次遇到此问题。

附: stderr 处理验证 - stderr=subprocess.STDOUT 已正确实现，输出不会丢失。问题根因是时序竞争而非输出捕获。

系统信息

Operating System: Ubuntu 24.04 (WSL2)
Hermes Version: v0.11.0 (2026.4.23)
Python Version: 3.11.15

extent analysis

TL;DR

To fix the Hermes "poll proc" infinite polling bug, add a poll count limit and detect no-output exceptions in the poll() method.

Guidance

Add a poll count limit: Introduce a MAX_POLL_COUNT variable to prevent excessive polling, returning an error status when exceeded.
Detect no-output exceptions: Implement a NO_OUTPUT_THRESHOLD to check for consecutive no-output polls, handling potential process hangs or exits.
Improve process status detection: Modify the poll() method to check the real process status using session.process.poll() when the reader thread hasn't set exited.
Optimize background mode triggering: Rethink the conditions for triggering background mode, considering command types and shorter frontend timeouts.

Example

MAX_POLL_COUNT = 30
NO_OUTPUT_THRESHOLD = 5

def poll(self, session_id: str) -> dict:
    # ...
    if poll_count > MAX_POLL_COUNT:
        return {"status": "error", "message": "Polling limit exceeded"}
    # ...
    if consecutive_no_output > NO_OUTPUT_THRESHOLD:
        # Check process real status and handle potential exit or hang
        if session.process.poll() is not None:
            session.exited = True
            # ...

Notes

The provided code snippets and guidance focus on addressing the identified issues, but a thorough review of the Hermes codebase and testing are necessary to ensure a comprehensive fix.

Recommendation

Apply the suggested workaround by adding a poll count limit and detecting no-output exceptions to mitigate the issue until a more fundamental fix can be implemented.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #autograd error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

hermes - ✅(Solved) Fix [Bug]: process poll 无限轮询导致 Gateway 卡死 [2 pull requests, 1 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Fix Action

Fixed

PR fix notes

PR #17346: fix: prevent infinite process poll loop from reader-thread race (#17327)

Description (problem / solution / changelog)

Root Cause

Fix (two layers)

Layer 1 — Race condition guard (root cause fix)

Layer 2 — Safety cap (defense in depth)

Verification

Changed files

PR #17430: fix(process): reconcile session.exited against real child exit in poll/wait

Description (problem / solution / changelog)

Summary

Changes

Validation

Changed files

Code Example

Hermes "poll proc" 无限轮询 Bug 报告

问题现象

根因分析

1. 对比实验

2. 触发条件

3. 代码缺陷定位

时间线重建

建议修复

短期修复（紧急）

中期修复（根本）

临时规避方案

相关文件

总结

系统信息

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

RELATED_DISCOVERY

TRENDING