openclaw - 💡(How to fix) Fix Cross-version image upgrade: bundled-runtime-mirror EACCES on stale symlink in plugin-runtime-deps/ [3 comments, 3 participants]

Summary

When an OpenClaw container is upgraded to a new image version, gateway crash-loops with EACCES: permission denied, copyfile … .openclaw-mirror-…tmp because the bundled-runtime-mirror copies through stale symlinks pointing at the read-only image layer.

Repro

Multi-tenant container deployment (badclaw fleet on Hetzner). Tenants on badclaw-openclaw:main-20260428-patched (OpenClaw v2026.4.26) recreated onto badclaw-openclaw:2026.4.27-patched-claude (OpenClaw v2026.4.27). The volume /home/agent/.openclaw is bind-mounted from the host, so it preserves plugin-runtime-deps/ across recreates.

Result: 2 of 3 tenants crash-looped (138 and 32 restarts before manual intervention). The 3rd tenant whose volume was effectively empty pre-upgrade booted cleanly.

Stack trace

{
  errno: -13,
  code: 'EACCES',
  syscall: 'copyfile',
  path: '/opt/openclaw/dist/agents/auth-profiles.runtime.js',
  dest: '/home/agent/.openclaw/plugin-runtime-deps/openclaw-2026.4.27-d2ac19f1349f/dist/agents/.openclaw-mirror-1-…-auth-profiles.runtime.js.tmp'
}
[openclaw] Failed to start CLI: Error: EACCES: permission denied, copyfile '…' -> '…'
    at Object.copyFileSync (node:fs:3105:11)
    at copyBundledRuntimeMirrorFileAtomic (file:///opt/openclaw/dist/bundled-runtime-root-Bvrq1R9V.js:142:6)
    at copyBundledPluginRuntimeRoot (file:///opt/openclaw/dist/bundled-runtime-root-Bvrq1R9V.js:65:3)
    at tracePluginLifecyclePhase.pluginId (file:///opt/openclaw/dist/bundled-runtime-root-Bvrq1R9V.js:36:3)
    …

Root cause

The destination directory at plugin-runtime-deps/openclaw-2026.4.27-…/dist/agents/ is a symlink pointing to /opt/openclaw/dist/agents/ — left over from the previous OpenClaw version's mirror initialization. When copyBundledRuntimeMirrorFileAtomic writes a .tmp file to that path, the kernel resolves the symlink and tries to write into the image's read-only layer. EACCES.

Confirmed: ``` $ ls -l /home/agent/.openclaw/plugin-runtime-deps/openclaw-2026.4.27-d2ac19f1349f/dist/agents lrwxrwxrwx 1 1001 1001 25 Apr 28 13:39 dist/agents -> /opt/openclaw/dist/agents ```

Many other entries under dist/ are also symlinks (auto-reply, media-understanding, plus dozens of *.js files), so removing just dist/agents produces the same crash on the next file.

Workaround

Wipe the entire plugin-runtime-deps/ directory before container recreate. OpenClaw rebuilds it cleanly from scratch on first boot. Costs ~30-60s extra boot per tenant.

Suggested fix

In bundled-runtime-root.js#copyBundledRuntimeMirrorFileAtomic (or equivalent in the newer source), detect when the destination's parent directory is a symlink and `unlink + mkdir` before writing. Equivalent: when initializing the mirror tree for a new OpenClaw version, never reuse symlink entries from a previous version's tree — always materialize as real dirs.

Environment

• Host: Hetzner CPX41, Ubuntu 24.04, Docker (overlay2 storage driver)
• Bind mount: ./tenants/<name>/openclaw:/home/agent/.openclaw
• Container UID: 1001:1001 (agent), running unprivileged
• OpenClaw versions involved: source v2026.4.26 → v2026.4.27, both running from a badclaw- prefixed branch with one out-of-tree patch (feat(bluebubbles): add reply-context API fallback)