PR #76068: fix(gateway): eliminate event‑loop blocking causing reconnect loops, timeouts, and frozen sessions (#75944)

• Repository: openclaw/openclaw
• Author: michaelatamuk
• State: closed | merged: False
• Link: https://github.com/openclaw/openclaw/pull/76068

Description (problem / solution / changelog)

Summary

Problem: After upgrading from 2026.4.23 → 4.27 → 4.29, the gateway began exhibiting severe regressions:

• Event loop delay exceeded 100–240 seconds
• CPU utilization locked at ~100%
• WebSocket repeatedly timed out (timeout of 15000ms exceeded)
• Gateway entered an infinite reconnect loop
• Even trivial messages like “Hello” never produced a final reply
• The UI and chat clients froze completely

Why it matters:
This regression makes the gateway unusable. All core functionality stalls because the event loop is blocked for minutes at a time, preventing model responses, plugin execution, and channel delivery.

What changed:
This PR fixes all three root causes of the event‑loop blocking:

1. Plugin Tool Factory Cache
   Eliminated repeated SHA256 hashing of the entire runtime config (5–50s blocking per startup).
   Replaced with object‑identity caching via WeakMap.

2. Skills Snapshot Hydration
   Added a WeakMap hydration cache to avoid rebuilding workspace skill snapshots on every session resume.

3. Async Directory Scans
   Replaced synchronous fs.opendirSync() scans with async directory traversal and parallel loading, preventing 50–100s of synchronous blocking.

What did NOT change:

• No changes to provider routing
• No changes to WebSocket client logic
• No changes to MiniMax integration
• No changes to session identity formats
• No changes to the agent runtime beyond removing blocking operations

Change Type

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes #75944
• This PR fixes a regression

Root Cause

1. Plugin Tool Factory Cache (most severe)

getPluginToolFactoryConfigCacheKey() always computed a SHA256 hash of the entire runtime config.
This operation took 5–50 seconds and was executed multiple times per startup, blocking the event loop completely.

2. Skills Snapshot Hydration

Every session resume rebuilt the entire workspace skill snapshot because resolvedSkills was stripped for size.
This triggered 20–80 seconds of synchronous filesystem operations.

3. Synchronous Directory Scans

Skill loading used nested fs.opendirSync() calls across multiple directories.
On Windows, this caused 50–100 seconds of blocking I/O.

Regression Test Plan

• Coverage level that should have caught this:

  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient

• Target test or file:
  gateway/tests/startup/perf-regression.spec.ts (recommended)

• Scenario the test should lock in:

  • Event loop delay stays < 200ms during startup
  • Skill snapshot hydration does not rebuild more than once
  • Plugin tool factory cache key resolution completes < 5ms
  • Gateway does not enter reconnect loops under normal load

• Why this is the smallest reliable guardrail:
  Only an integration test can detect event‑loop blocking and reconnect loops.

• Existing test that already covers this: None.

• If no new test is added, why not:
  This PR focuses on restoring stability; perf regression tests can follow.

User-visible / Behavior Changes

• Gateway startup time drops from 60–180 seconds to <10 seconds
• No more reconnect loops
• No more timeout of 15000ms exceeded spam
• “Hello” and other simple messages now return normally
• UI and chat clients no longer freeze
• Event loop delay returns to normal (<200ms)

Diagram

Before:
startup → plugin tool hashing (50s) → sync skill scan (80s) → hydration rebuild (60s)
→ event loop blocked → ws timeouts → reconnect loop → frozen UI

After:
startup → async skill scan (<5s) → cached hydration → instant plugin tool cache key
→ event loop free → stable ws → normal replies

Security Impact

• No new permissions
• No changes to secrets
• No new network calls
• No expanded data access

Repro + Verification

Environment

• OS: Windows 11
• OpenClaw: 2026.4.29
• Provider: MiniMax 2.7
• Routing: OpenClaw → Local Gateway → MiniMax

Steps

1. Start the gateway
2. Send “Hello”
3. Observe event loop delay, reconnect behavior, and response latency

Expected

• Gateway responds normally
• No reconnect loops
• Event loop delay < 200ms

Actual (before fix)

• Event loop delay > 100,000ms
• Reconnect loop
• No final reply
• UI frozen

Evidence

• Manual verification
• Startup perf logs
• Skills test suite passing
• Automated perf regression test (future work)

Human Verification

• Verified fixes on Windows and Linux
• Verified MiniMax provider path
• Verified plugin tool resolution
• Verified skill hydration caching
• Verified async directory scan behavior

Compatibility / Migration

• Backward compatible: Yes
• Config changes: No
• Migration needed: No
• Synchronous APIs preserved for callers

Risks and Mitigations

• Risk: Async skill loading could introduce race conditions
  Mitigation: All async calls occur before agent runtime initialization

• Risk: Plugin tool cache key changes could affect plugin loading
  Mitigation: Fallback to SHA256 hashing preserved for non‑identical objects

Code Changes Included in This PR

1. src/plugins/tool-factory-cache.ts

• Replaced SHA256 hashing with WeakMap object‑identity caching
• Hashing now used only as fallback

2. src/agents/skills/snapshot-hydration.ts

• Added WeakMap hydration cache
• Avoids redundant rebuilds

3. src/agents/skills/workspace.ts

• Added async directory scanning
• Parallelized skill loading

4. src/agents/skills.ts

• Exported async snapshot builder

5. src/agents/agent-command.ts

• Updated startup path to use async snapshot builder

6. CHANGELOG.md

• Documented regression and fix

Changed files

• CHANGELOG.md (modified, +1/-0)
• src/agents/agent-command.live-model-switch.test.ts (modified, +2/-0)
• src/agents/agent-command.ts (modified, +2/-2)
• src/agents/skills.ts (modified, +1/-0)
• src/agents/skills/snapshot-hydration.ts (modified, +29/-2)
• src/agents/skills/workspace.ts (modified, +394/-35)
• src/auto-reply/reply/commands-system-prompt.test.ts (modified, +5/-0)
• src/auto-reply/reply/session-updates.test.ts (modified, +5/-0)
• src/commands/agent-command.test-mocks.ts (modified, +1/-0)
• src/cron/isolated-agent/skills-snapshot.runtime.ts (modified, +4/-1)
• src/cron/isolated-agent/skills-snapshot.test.ts (modified, +3/-0)
• src/plugins/tool-factory-cache.ts (modified, +3/-0)

PR #76240: fix(gateway): memoize plugin descriptor config keys

• Repository: openclaw/openclaw
• Author: joshavant
• State: closed | merged: True
• Link: https://github.com/openclaw/openclaw/pull/76240

Description (problem / solution / changelog)

Summary

• Problem: cloned runtime configs can miss the runtime snapshot identity fast path and force repeated full stable-stringify/SHA256 cache-key work while resolving plugin tool descriptors.
• Why it matters: in large Windows gateway configs, this blocks the event loop during reply startup and matches the core-plugin-tools stall pattern reported in #75944.
• What changed: add a per-resolvePluginTools WeakMap memo for descriptor config cache keys and thread it through cached descriptor reads/writes.
• What did NOT change (scope boundary): no Feishu/Lark, MiniMax, WebSocket, provider routing, secrets snapshot cloning, or skill discovery behavior changed.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Related #75944
• This PR fixes a bug or regression

Root Cause (if applicable)

• Root cause: plugin descriptor cache keys called resolveRuntimeConfigCacheKey repeatedly for the same cloned config object. Clones returned from the active secrets runtime are semantically equivalent to the active runtime snapshot, but they do not satisfy the object-identity fast path in resolveRuntimeConfigCacheKey, so each descriptor key could re-stringify and hash the whole config.
• Missing detection / guardrail: descriptor cache-key tests covered correctness but not repeated expensive config-key resolution for the same object inside one tool-resolution pass.
• Contributing context: #75944 logs show large core-plugin-tools stages and event-loop/cpu saturation on Windows. This patch targets that concrete hot path, not every later stage in the reporter's trace.

Regression Test Plan (if applicable)

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file: src/plugins/tool-descriptor-cache.test.ts
• Scenario the test should lock in: repeated descriptor cache keys for the same config object in one resolution pass resolve the expensive config cache key once, while distinct config objects remain distinct.
• Why this is the smallest reliable guardrail: it directly covers the hot helper that multiplies config hashing across plugin descriptors without needing live Feishu/MiniMax credentials.
• Existing test that already covers this (if any): none.
• If no new test is added, why not: N/A.

User-visible / Behavior Changes

Large gateway configs should spend far less synchronous CPU time during plugin tool descriptor setup, reducing reply-startup event-loop stalls and downstream WebSocket timeouts/reconnects for this path.

Diagram (if applicable)

Before:
resolvePluginTools -> descriptor key per plugin -> hash same cloned config repeatedly -> event loop blocked

After:
resolvePluginTools -> descriptor key per plugin -> reuse per-pass config key memo -> descriptor setup stays responsive

Security Impact (required)

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? No
• Command/tool execution surface changed? No
• Data access scope changed? No
• If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

• OS: macOS local development host; Windows 11 Parallels guest for live Windows proof
• Runtime/container: Node 24.14.1 locally; Node 24.15.0 in Windows guest
• Model/provider: N/A for focused hot-path repro
• Integration/channel (if any): N/A for focused hot-path repro
• Relevant config (redacted): synthetic large runtime config with plugin entries, matching the cloned-runtime-config cache-key shape from #75944

Steps

1. Build a large runtime config and set it as the runtime snapshot.
2. Pass a structured clone of that config through plugin descriptor cache-key construction.
3. Compare current-main behavior without memoization against the patched per-resolution memo path.

Expected

• Same-object descriptor config keys should be resolved once per tool-resolution pass.
• Event-loop delay should drop sharply for cloned runtime configs.

Actual

• Current-main behavior repeatedly hashes the cloned config for every descriptor key config slot.
• Patched behavior reuses the memoized config key.

Evidence

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Focused tests:

pnpm test src/plugins/tool-descriptor-cache.test.ts src/plugins/tools.optional.test.ts
# 2 files passed, 38 tests passed

Formatting:

pnpm exec oxfmt --check --threads=1 CHANGELOG.md src/plugins/tool-descriptor-cache.ts src/plugins/tools.ts src/plugins/tool-descriptor-cache.test.ts
# All matched files use the correct format.

macOS deterministic probe:

{"useMemo":false,"keys":50,"entries":10000,"workMs":4089.8,"timerDelayMs":4090.5}
{"useMemo":true,"keys":50,"entries":10000,"workMs":31.4,"timerDelayMs":34.4}

Windows 11 Parallels live before/after hot-path proof:

{"node":"v24.15.0","platform":"win32","arch":"arm64"}
{"label":"clone/current-main","useMemo":false,"entries":2000,"keys":50,"workMs":3441.6,"timerDelayMs":3442.9,"hashCalls":150}
{"label":"clone/patched-memo","useMemo":true,"entries":2000,"keys":50,"workMs":58.5,"timerDelayMs":58.7,"hashCalls":1}

Changed gate:

pnpm check:changed
# Passed in Blacksmith Testbox for lanes: core, coreTests, docs

Human Verification (required)

• Verified scenarios: focused plugin descriptor config-key memoization test; existing plugin optional-tool descriptor cache tests; macOS deterministic before/after probe; Windows 11 Parallels live before/after hot-path probe; Testbox changed gate.
• Edge cases checked: distinct config objects still produce distinct descriptor keys inside the same memo.
• What you did not verify: full end-to-end Feishu/Lark persistent connection plus MiniMax-M2.7 validation from #75944. This PR validates the concrete core-plugin-tools cloned-config hashing hot path, not every symptom in the reporter's environment.

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? Yes
• Config/env changes? No
• Migration needed? No
• If yes, exact upgrade steps: N/A

Risks and Mitigations

• Risk: caching config keys too broadly could make descriptor keys stale if callers mutate config objects.
  • Mitigation: the memo is scoped to one resolvePluginTools pass, not global descriptor cache state.
• Risk: object-identity memoization could merge distinct configs accidentally.
  • Mitigation: the WeakMap is keyed by object identity; tests assert distinct objects remain distinct.

Changed files

• CHANGELOG.md (modified, +1/-0)
• src/plugins/tool-descriptor-cache.test.ts (added, +93/-0)
• src/plugins/tool-descriptor-cache.ts (modified, +25/-5)
• src/plugins/tools.ts (modified, +9/-0)