hermes - 💡(How to fix) Fix Bug: Concurrent ddgs web searches cause hard freeze (futex_do_wait deadlock, CTRL-C immune)

{ "title": "Bug: Concurrent ddgs web searches cause hard freeze (futex_do_wait deadlock, CTRL-C immune)", "labels": ["type/bug", "tool/web", "P2", "comp/tools"], "body": "## Summary\n\nWhen Hermes executes two web_search (ddgs backend) calls concurrently via execute_tool_calls_concurrent(), the process hard-freezes. All threads enter futex_do_wait / idle state, a TCP socket is stranded in CLOSE_WAIT, and SIGINT (CTRL-C) does not recover the process. Only SIGKILL from another terminal works.\n\nNot a resource exhaustion issue — the process consumes 0% CPU, 282MB RSS, no OOM.\n\n## Environment\n\n- Hermes Agent commit 64a9a199b (2026-05-19)\n- ddgs 9.13.0 (installed by uv in Hermes venv)\n- primp 1.2.2 (underlying HTTP client, compiled C/libcurl extension)\n- duckduckgo_search 8.1.1 (co-installed, same author)\n- Provider: deepseek-v4-flash via custom endpoint (api.deepseek.com)\n- Platform: Linux (Arch, kernel 7.0.3-arch1-2)\n- Python 3.11 (Hermes venv)\n- Web config: web.backend: \"\" (empty — all env keys empty, auto-detect falls through to ddgs as the only available backend)\n- web_search is listed in _PARALLEL_SAFE_TOOLS in agent/tool_dispatch_helpers.py\n\n## Reproduction\n\nThe agent must issue two web_search calls in a single LLM response (triggers concurrent dispatch). In the captured session, the LLM returned both simultaneously (2ms apart):\n\n11:06:40,939 - Web search via ddgs: \"PETA animal rights objections drug dog training welfare concerns\" (limit: 5)\n11:06:40,941 - Web search via ddgs: \"ethical concerns drug detection dogs welfare criticism activists\" (limit: 5)\n\n\n## Forensic Evidence\n\n### Process state (from /proc)\n\nProcess: S (sleeping)\nMain thread wchan: futex_do_wait\nAll 8 worker threads: I (idle)\nNo pending signals (SigPnd=0)\n\n\nThe SIGPnd=0 + futex_do_wait explains why CTRL-C fails — the main thread is blocked in a condition-variable wait that does not re-check the signal handler.\n\n### TCP socket state\n\nFD 24: socket:[551540993] -> CLOSE_WAIT\nRemote: 3.173.21.63:443 (AWS, likely DuckDuckGo HTML endpoint)\n\n\nThe socket was closed by the remote side, but Python never cleaned it up — a worker thread was blocked before it could process the socket closure.\n\n### Memory monitor (last log before freeze)\n\n11:07:06 - rss=282MB gc=(528, 9, 1) threads=11 uptime=63300s\n\n\nNo new log entries after this. The memory monitor (a background thread) fired once 26s after the searches started, then silence.\n\n### Child processes\n- deepwiki-mcp (PID 887880) was also dead/unreachable at the time of inspection\n\n## Root Cause Analysis\n\nWhen Hermes executes two web_search calls concurrently, each one:\n\n1. Creates a new DDGS() instance (ddgs library at v9.13.0)\n2. DDGS.__init__ sets timeout=5\n3. DDGS._search() spawns its own internal ThreadPoolExecutor (multiple engine-scraping workers)\n4. Each worker creates a primp.Client() (libcurl wrapper, compiled C extension)\n5. Both instances try to scrape the same DuckDuckGo HTML endpoints simultaneously\n\nThe result is a thread explosion:\n\nHermes Worker 1 -> DDGS A -> ThreadPoolExecutor A (2+ threads -> primp -> libcurl)\nHermes Worker 2 -> DDGS B -> ThreadPoolExecutor B (2+ threads -> primp -> libcurl)\n\n\nThis creates 4-8+ HTTP threads all hitting the same endpoints from the same IP. The interaction between DDGS' internal thread pools and primp's libcurl state management causes a deadlock at the Python threading level (futex_do_wait).\n\n## Impact\n\n- Complete hard freeze of the Hermes CLI session\n- CTRL-C does not work\n- Must kill -9 from another terminal\n- Any work-in-progress in that session is lost\n\n## Workaround\n\nDo not rely on ddgs as the auto-detected free backend. Configure a proper search backend:\n- Set a Firecrawl, Tavily, Exa, or Brave Free API key\n- Or run SearXNG locally and set SEARXNG_URL\n\nAlternatively, remove web_search from _PARALLEL_SAFE_TOOLS in agent/tool_dispatch_helpers.py to force sequential execution.\n\n## Suggested Fix\n\nEither:\n1. Remove web_search from _PARALLEL_SAFE_TOOLS — the ddgs backend is not safe to run concurrently\n2. Or add a global semaphore / rate limiter in the ddgs provider to prevent concurrent DDGS() instantiation\n3. Or document that ddgs should not be used when concurrent tool execution is enabled" }

Fix / Workaround

{ "title": "Bug: Concurrent ddgs web searches cause hard freeze (futex_do_wait deadlock, CTRL-C immune)", "labels": ["type/bug", "tool/web", "P2", "comp/tools"], "body": "## Summary\n\nWhen Hermes executes two web_search (ddgs backend) calls concurrently via execute_tool_calls_concurrent(), the process hard-freezes. All threads enter futex_do_wait / idle state, a TCP socket is stranded in CLOSE_WAIT, and SIGINT (CTRL-C) does not recover the process. Only SIGKILL from another terminal works.\n\nNot a resource exhaustion issue — the process consumes 0% CPU, 282MB RSS, no OOM.\n\n## Environment\n\n- Hermes Agent commit 64a9a199b (2026-05-19)\n- ddgs 9.13.0 (installed by uv in Hermes venv)\n- primp 1.2.2 (underlying HTTP client, compiled C/libcurl extension)\n- duckduckgo_search 8.1.1 (co-installed, same author)\n- Provider: deepseek-v4-flash via custom endpoint (api.deepseek.com)\n- Platform: Linux (Arch, kernel 7.0.3-arch1-2)\n- Python 3.11 (Hermes venv)\n- Web config: web.backend: \"\" (empty — all env keys empty, auto-detect falls through to ddgs as the only available backend)\n- web_search is listed in _PARALLEL_SAFE_TOOLS in agent/tool_dispatch_helpers.py\n\n## Reproduction\n\nThe agent must issue two web_search calls in a single LLM response (triggers concurrent dispatch). In the captured session, the LLM returned both simultaneously (2ms apart):\n\n11:06:40,939 - Web search via ddgs: \"PETA animal rights objections drug dog training welfare concerns\" (limit: 5)\n11:06:40,941 - Web search via ddgs: \"ethical concerns drug detection dogs welfare criticism activists\" (limit: 5)\n\n\n## Forensic Evidence\n\n### Process state (from /proc)\n\nProcess: S (sleeping)\nMain thread wchan: futex_do_wait\nAll 8 worker threads: I (idle)\nNo pending signals (SigPnd=0)\n\n\nThe SIGPnd=0 + futex_do_wait explains why CTRL-C fails — the main thread is blocked in a condition-variable wait that does not re-check the signal handler.\n\n### TCP socket state\n\nFD 24: socket:[551540993] -> CLOSE_WAIT\nRemote: 3.173.21.63:443 (AWS, likely DuckDuckGo HTML endpoint)\n\n\nThe socket was closed by the remote side, but Python never cleaned it up — a worker thread was blocked before it could process the socket closure.\n\n### Memory monitor (last log before freeze)\n\n11:07:06 - rss=282MB gc=(528, 9, 1) threads=11 uptime=63300s\n\n\nNo new log entries after this. The memory monitor (a background thread) fired once 26s after the searches started, then silence.\n\n### Child processes\n- deepwiki-mcp (PID 887880) was also dead/unreachable at the time of inspection\n\n## Root Cause Analysis\n\nWhen Hermes executes two web_search calls concurrently, each one:\n\n1. Creates a new DDGS() instance (ddgs library at v9.13.0)\n2. DDGS.__init__ sets timeout=5\n3. DDGS._search() spawns its own internal ThreadPoolExecutor (multiple engine-scraping workers)\n4. Each worker creates a primp.Client() (libcurl wrapper, compiled C extension)\n5. Both instances try to scrape the same DuckDuckGo HTML endpoints simultaneously\n\nThe result is a thread explosion:\n\nHermes Worker 1 -> DDGS A -> ThreadPoolExecutor A (2+ threads -> primp -> libcurl)\nHermes Worker 2 -> DDGS B -> ThreadPoolExecutor B (2+ threads -> primp -> libcurl)\n\n\nThis creates 4-8+ HTTP threads all hitting the same endpoints from the same IP. The interaction between DDGS' internal thread pools and primp's libcurl state management causes a deadlock at the Python threading level (futex_do_wait).\n\n## Impact\n\n- Complete hard freeze of the Hermes CLI session\n- CTRL-C does not work\n- Must kill -9 from another terminal\n- Any work-in-progress in that session is lost\n\n## Workaround\n\nDo not rely on ddgs as the auto-detected free backend. Configure a proper search backend:\n- Set a Firecrawl, Tavily, Exa, or Brave Free API key\n- Or run SearXNG locally and set SEARXNG_URL\n\nAlternatively, remove web_search from _PARALLEL_SAFE_TOOLS in agent/tool_dispatch_helpers.py to force sequential execution.\n\n## Suggested Fix\n\nEither:\n1. Remove web_search from _PARALLEL_SAFE_TOOLS — the ddgs backend is not safe to run concurrently\n2. Or add a global semaphore / rate limiter in the ddgs provider to prevent concurrent DDGS() instantiation\n3. Or document that ddgs should not be used when concurrent tool execution is enabled" }