openclaw - 💡(How to fix) Fix [Bug] Plugin runtime cache races between version-keyed and "unknown"-keyed extractors → repeated ENOTEMPTY on cold start [3 comments, 3 participants]

Summary

The gateway creates two parallel plugin runtime cache directories on every cold start, with mismatched extraction code paths that race and produce repeated ENOTEMPTY errors during plugin extraction. This is reproducible with zero concurrent CLI activity and a wiped cache.

Side effects observed:

• 7+ ENOTEMPTY errors per cold start in gateway log
• Occasional PluginLoadFailureError: plugin load failed: discord on subsequent CLI invocations
• Bundled channel imessage consistently fails to load (failed to load bundled channel imessage)
• Boot latency increased while extraction retries

Environment

• OpenClaw: 2026.4.24 (cbcfdf6)
• macOS Darwin 25.3.0 (arm64), Mac mini
• Node v24.15.0
• Install: ~/.npm-global/lib/node_modules/openclaw
• Gateway runs as LaunchAgent (ai.openclaw.gateway), bind=loopback, port 18789

Reproduction

1. Stop gateway: launchctl unload ~/Library/LaunchAgents/ai.openclaw.gateway.plist
2. Kill stragglers: pkill -9 -f openclaw
3. Wipe cache completely: rm -rf ~/.openclaw/plugin-runtime-deps/*
4. Start gateway: launchctl load ~/Library/LaunchAgents/ai.openclaw.gateway.plist
5. Wait 60s, do not run any CLI commands during this window.
6. Inspect: ls ~/.openclaw/plugin-runtime-deps/

Observed

After the above (zero concurrent CLI calls), I get two runtime directories, not one:

openclaw-2026.4.24-619077f345ed   # proper version hash, dirs created with mode 755 (drwxr-xr-x)
openclaw-unknown-9b1225c0c592     # "unknown" key, dirs created with mode 700 (drwx------)

The differing umask suggests two distinct code paths are doing the extraction. Log shows repeated:

[channels] failed to load bundled channel imessage: ENOTEMPTY, Directory not empty:
  /Users/.../plugin-runtime-deps/openclaw-2026.4.24-619077f345ed/dist/extensions/node_modules/openclaw/plugin-sdk
  '/Users/.../plugin-runtime-deps/openclaw-2026.4.24-619077f345ed/dist/extensions/node_modules/openclaw/plugin-sdk'

grep -c ENOTEMPTY /tmp/openclaw/openclaw-YYYY-MM-DD.log → 7 occurrences in a single cold-start session.

On a second invocation (any openclaw <subcommand> call after the gateway is already up), I have intermittently seen:

[plugins] discord failed during register from .../dist/extensions/discord/index.js:
  Error: ENOTEMPTY, Directory not empty: .../openclaw-unknown-9b1225c0c592/dist/extensions/node_modules/openclaw/plugin-sdk

[openclaw] Failed to start CLI: PluginLoadFailureError: plugin load failed: discord:
  Error: ENOTEMPTY ...
    at maybeThrowOnPluginLoadError (.../loader-NucjcOgv.js:2712:8)
    at loadOpenClawPlugins (.../loader-NucjcOgv.js:3588:3)
    at resolveRuntimePluginRegistry (.../loader-NucjcOgv.js:2443:9)

Expected

Single runtime cache directory keyed off the resolved version hash, populated atomically (temp + rename) and protected by a per-process lock so a second extractor sees the existing cache instead of racing.

Hypothesis

There appear to be two extraction code paths active during early gateway boot:

1. A path that resolves the proper version (2026.4.24-619077f345ed) and unpacks with mode 755.
2. A path that fires before version resolution completes and falls back to an unknown key (9b1225c0c592), unpacking with mode 700.

When both paths target overlapping dist/extensions/node_modules/openclaw/plugin-sdk directories, the rename step in one races with files left over by the other → ENOTEMPTY.

Suggested fixes

• Resolve the runtime cache key (version hash) before any plugin extractor starts; never fall back to unknown while a real version is being computed in parallel.
• Use atomic temp-dir + rename() extraction with a lock file (or fs.mkdtemp + rename pattern) instead of in-place writes.
• Single-flight the extraction across subsystems via a shared promise/mutex within the gateway process.
• Surface a clearer error when the cache is poisoned (e.g., suggest cache-clear command) instead of ENOTEMPTY raw.

Workaround

Periodically rm -rf ~/.openclaw/plugin-runtime-deps/* and restart the gateway. Errors persist but the gateway eventually reaches running state and channels work in degraded mode.