openclaw - 💡(How to fix) Fix [Bug]: Gateway event-loop blocked by provider auth prewarm + plugin metadata snapshot cache miss on v2026.5.22

Bug type

Regression (worked before, now fails)

Beta release blocker

No

Summary

After upgrading from v2026.5.20 to v2026.5.22, all WebSocket RPC calls (sessions.list, models.list, chat.history, agents.list) degrade from <300ms to 3–8 seconds. The Gateway event loop is saturated by two mechanisms introduced/changed in 5.22:

1. scheduleProviderAuthStatePrewarm — blocks the event loop for 23–71 seconds at startup
2. loadPluginMetadataSnapshot synchronous fallback — scans 91 bundled plugin manifests (3–5s) on every models.list/sessions.list call because the “immutable snapshot” cache misses after hot-reload

Rolling back to v2026.5.20 immediately restores all calls to <700ms.

Environment

• OpenClaw: 2026.5.22 (confirmed same behavior on 2026.5.24-beta.1 and beta.2)
• Working version: 2026.5.20
• Node: v22.19.0
• OS: macOS 15.5 (Apple Silicon M1 Pro)
• Agents: 7 configured (main + 6 specialized)
• Plugins: 4 enabled (browser, memory-core, searxng, skill-trigger-engine); 91 bundled total
• Custom providers: 2 (idealab via local proxy, dashscope)
• Service: launchd (ai.openclaw.gateway)

Steps to Reproduce

1. Install v2026.5.20 — verify sessions.list responds in <300ms
2. Upgrade to v2026.5.22
3. Restart gateway
4. Open Control UI webchat, switch between sessions
5. Observe 3–8 second delays on every session switch

Observed behavior

Startup phase (first 60s after boot)

[gateway] provider auth state pre-warmed in 70949ms eventLoopMax=12507.4ms

During this 71-second window, event loop utilization = 100%, ALL WebSocket requests queue:

sessions.list  18156ms
agents.list    11420ms
models.list     8889ms
chat.history   12462ms

Steady state (after prewarm completes)

Every models.list call takes 3–5 seconds because loadPluginMetadataSnapshot falls back to synchronous filesystem scanning:

models.list    avg 3144ms  (was <500ms on 5.20)
sessions.list  avg 4997ms  (was <300ms on 5.20)
chat.history   avg 4070ms  (was <700ms on 5.20)

v5.20 baseline (same machine, same config)

sessions.list  183–674ms
models.list    237–439ms
chat.history   85–698ms

Expected behavior

sessions.list, models.list, chat.history should complete in <500ms at all times (matching v5.20 behavior).

Root Cause Analysis

Layer 1: Provider Auth State Pre-warm (startup blocker)

scheduleProviderAuthStatePrewarm in server-startup-post-attach-*.js iterates ALL agents × ALL providers calling hasAuthForModelProvider (includes external CLI discovery). With 7 agents this takes 23–71 seconds blocking the event loop.

• sidecars.model-prewarm: 0.9–1.5s on v5.20 → 4.5–6.7s on v5.22
• sidecars.session-locks: 0.7–1.3s on v5.20 → 3.4–13.4s on v5.22

Partial workaround: Patching providerAuthPrewarm: { enabled: false } in server.impl eliminates the 70s startup block. But Layer 2 persists.

Layer 2: Plugin metadata snapshot cache invalidation

The 5.22 changelog claims “reuse immutable plugin metadata snapshots”. However, the cache (getCurrentPluginMetadataSnapshot) fails to return a hit when:

1. models.json does not exist → loadReadOnlyPersistedModelCatalog throws → falls back to synchronous loadReadOnlyStaticModelCatalog
2. Any config hot-reload changes configFingerprint → snapshot marked stale → every subsequent call re-scans

loadManifestMetadataSnapshot → loadPluginMetadataSnapshot performs a synchronous scan of all 91 openclaw.plugin.json files. Measured via find: 2.7 seconds of pure filesystem I/O blocking the main thread.

What I’ve tried

Related issues

• #75297 — Gateway event-loop saturation and very slow sessions.list/models.list on all versions after 2026.4.23
• #76382 — Gateway becoming very slow, CPU 100% (versions 4.24–5.2), closed by #76517
• #76166 — Control UI still repeatedly calls slow sessions.list on 2026.4.29
• #76421 — sessions.list/send gateway timeout after event loop stall

Suggested fix direction

1. Make loadPluginMetadataSnapshot async (or ensure the cached snapshot survives hot-reloads without full re-scan)
2. Ensure configFingerprint changes don’t invalidate the plugin manifest cache (plugin manifests don’t change on config edit)
3. Move scheduleProviderAuthStatePrewarm to a worker thread or yield every N plugins to prevent event-loop starvation
4. Add a user-facing config flag to disable provider auth prewarm (e.g., gateway.providerAuthPrewarm.enabled: false)