openclaw - 💡(How to fix) Fix [Bug]: Gateway CPU 60-80% on ARM64 Linux, event loop blocked 37s (v2026.4.29) [5 comments, 4 participants]

[Bug]: Gateway CPU 60-80% on ARM64 Linux, event loop blocked 37s (v2026.4.29)

Bug type

Behavior bug (incorrect output/state without crash)

Summary

Gateway process consumes 60-80% CPU on ARM64 Linux (aarch64) with v2026.4.29, even with minimal activity. Event loop is blocked up to 37-40 seconds per agent run, causing Web UI disconnections and 2-3 minute unavailability after each conversation.

Related to #75707 and #75688 — same version, similar symptoms, but on ARM64 architecture with different model provider (xiaomi-coding/mimo-v2.5-pro).

Environment

• OS: Ubuntu Linux 6.6.89-Gold_bug (aarch64 / ARM64)
• Node: v22.22.2
• OpenClaw: v2026.4.29 (a448042)
• Hardware: 8-core ARM64, 15GB RAM
• Agent model: xiaomi-coding/mimo-v2.5-pro
• Channel: openclaw-weixin
• Cron jobs: 6 total (1 running every 3 minutes)

Symptom

1. Gateway process (node) consistently at 60-80% CPU, load average 12-14 on 8 cores
2. Each agent run's prep phase takes 58-80 seconds, blocking the event loop
3. Web UI disconnects during agent runs and cannot reconnect for 2-3 minutes
4. Memory grows from ~650MB to ~1GB over time
5. Restarting (SIGUSR1 or kill) does not resolve the issue — CPU returns to high levels immediately

Diagnostic Data — Agent Run Prep Breakdown

Every embedded-run shows these consistently slow stages:

core-plugin-tools:  9-13 seconds
bundle-tools:       9-12 seconds
system-prompt:      20-29 seconds  ← largest bottleneck
stream-setup:       19-27 seconds
────────────────────────────────
total prep:         58-80 seconds (blocking event loop)

Diagnostic Data — Event Loop

eventLoopDelayP99Ms: 37,000-40,000ms
eventLoopUtilization: 1.0 (saturated)
cpuCoreRatio: 1.017-1.029

Diagnostic Data — I/O

rchar: 54GB (in ~2 hours of uptime)
syscr: 37 million read syscalls
plugin-runtime-deps: 98,619 files, 12,003 directories, 2.4GB

Root Causes Identified

1. Plugin runtime staging on every agent run — 98K files in ~/.openclaw/plugin-runtime-deps/ are scanned on every embedded-run. core-plugin-tools stage takes 9-13 seconds each time.

2. system-prompt construction is CPU-bound — Takes 20-29 seconds per run. Likely tokenizing/building tool definitions from 11 skills + 15 plugins on the main thread.

3. stream-setup blocks event loop — 19-27 seconds per run. Likely re-tokenizing context for the model API call on the main thread.

4. Cron job amplification — A cron job running every 3 minutes (stock alert monitor) triggered continuous agent runs, each blocking the event loop for 60-80 seconds. After disabling it, CPU dropped from 80% to 65%.

5. Compaction safeguard — contextTokens was unset (defaulting to 32000), potentially triggering compaction loops. Set to 128000.

Workarounds Applied

After all workarounds: still 62-65% CPU, event loop still blocked during agent runs.

Key Difference from #75707

The reporter in #75707 achieved ~15% idle CPU after workarounds. Our case still shows 62-65% CPU. Possible reasons:

• ARM64 architecture may have slower file I/O for the 98K plugin-runtime-deps
• The system-prompt and stream-setup stages may have ARM64-specific performance issues
• NODE_OPTIONS was not picked up (gateway restarted by OpenClaw process manager, not systemd)

Additional Observations

• plugins.allow whitelist is configured with 15 plugins — unused providers' dependencies (253MB @anthropic-ai, 193MB @zed-industries, 172MB @openai, 118MB @lancedb, etc.) are still installed and scanned
• plugins.entries.X.enabled: false does not prevent dependency loading (confirmed per #75707)
• Web UI disconnection is directly correlated with event loop blocking during agent prep

Suggestions

1. Cache plugin-runtime-deps — Don't re-scan 98K files on every agent run; cache by spec hash
2. Move heavy prep off main thread — system-prompt and stream-setup should not block the event loop
3. ARM64 optimization — File I/O for large node_modules may need ARM64-specific optimization
4. Respect plugins.allow for dependency loading — Only load dependencies for allowed plugins
5. NODE_OPTIONS should be configurable in gateway config, not just via systemd env

Logs

2026-05-02T11:07:12.781Z warn diagnostic liveness warning: reasons=event_loop_delay,event_loop_utilization,cpu interval=43s eventLoopDelayP99Ms=40835.7 eventLoopDelayMaxMs=40835.7 eventLoopUtilization=1 cpuCoreRatio=1.029

2026-05-02T11:00:20.687Z warn agent/embedded [trace:embedded-run] prep stages: phase=stream-ready totalMs=58745 stages=core-plugin-tools:9179ms, bundle-tools:8897ms, system-prompt:20228ms, stream-setup:19680ms