openclaw - 💡(How to fix) Fix [Bug]: NemoClaw sandbox incompatible with OpenClaw 2026.4.20+ per-plugin runtime dep installer [3 comments, 3 participants]

Bug type

Crash (process/app exits or hangs)

Beta release blocker

No

Summary

Upgrading OpenClaw from 2026.4.15 to 2026.4.21 inside a NemoClaw sandbox breaks all channel workflows with PluginLoadFailureError: plugin load failed: acpx: failed to install bundled runtime deps: Error: npm error code EACCES. Root cause: OpenClaw 4.20+ installs per-plugin runtime deps on demand into /usr/local/lib/node_modules/openclaw/dist/extensions/<plugin>/node_modules/, but NemoClaw sandboxes mount /usr/local/ read-only for the sandbox user. No chown or permission adjustment from the host root context can make the sandbox's effective view of /usr/local/ writable.

Rollback to 2026.4.15 restores full functionality. The workaround documented in v7 of the community upgrade guide (chown dist/extensions/ from the host root context) is sufficient for 4.15 but does not resolve the 4.20+ incompatibility.

Steps to reproduce

Environment

• Host: DGX Spark, Ubuntu 24.04.4 LTS (aarch64), kernel 6.17.0-1014-nvidia
• NemoClaw: v0.1.0, gateway image ghcr.io/nvidia/openshell/cluster:0.0.15
• Docker container: openshell-cluster-nemoclaw
• Sandbox name: my-assistant
• Runtime: k3s + containerd overlay filesystem
• Sandbox user: uid=999(sandbox) gid=999(sandbox), CapEff=0
• Inference provider: nvidia/nemotron-3-super-120b-a12b via inference.local/v1 with allowPrivateNetwork: true
• Channels: Telegram bridge (host-side), bot @chief_nemoclaw_bot

OpenClaw upgrade path history on this sandbox: 3.11 → 3.28 → 4.9 → 4.11 → 4.14 → 4.15 → 4.21 (failed, rolled back).

Reproduction

Setup (working 4.15 baseline)

1. Sandbox running OpenClaw 2026.4.15 with user-prefix npm install and docker exec-based clean sync to /usr/local/lib/node_modules/openclaw/ — confirmed working for 7+ days prior to upgrade attempt.

2. Telegram bridge on host (~/.nemoclaw/source/scripts/telegram-bridge.js) spawning agents via:

   ssh -T -F $confPath openshell-my-assistant \
     "export NVIDIA_API_KEY=... && nemoclaw-start openclaw agent --agent main --local -m <msg> --session-id tg-<userId>"

Upgrade steps

Inside the sandbox (as uid 999):

npm i -g [email protected] --loglevel verbose
# 16 added, 419 removed, 438 changed in 8s — matches the 4.20 core/plugin split
~/.npm-global/bin/openclaw --version
# → OpenClaw 2026.4.21 (f788c88)

On the host (as root via docker exec):

docker exec -u root openshell-cluster-nemoclaw sh -c '
ROOTFS=$(find /run/k3s/containerd -path "*/rootfs/usr/local/lib/node_modules/openclaw/openclaw.mjs" 2>/dev/null | head -1 | sed "s|/usr/local/lib/node_modules/openclaw/openclaw.mjs||")
rm -rf ${ROOTFS}/usr/local/lib/node_modules/openclaw/*
cp -rf ${ROOTFS}/sandbox/.npm-global/lib/node_modules/openclaw/* ${ROOTFS}/usr/local/lib/node_modules/openclaw/
SANDBOX_OWNER=$(stat -c "%u:%g" ${ROOTFS}/sandbox)   # → 999:999
chown -R $SANDBOX_OWNER ${ROOTFS}/usr/local/lib/node_modules/openclaw/dist/extensions/
'

Post-upgrade state

Inside the sandbox:

$ ls -la /usr/local/lib/node_modules/openclaw/dist/extensions/acpx/
drwxr-xr-x   3 sandbox sandbox  4096 Apr 22 11:06 .
drwxr-xr-x 107 sandbox sandbox  4096 Apr 22 11:06 ..
-rw-r--r--   1 sandbox sandbox   133 Apr 22 11:06 .openclaw-runtime-deps-stamp.json
-rw-r--r--   1 sandbox sandbox   618 Apr 22 11:06 index.js
[...]

Ownership reports as sandbox:sandbox mode 755 — looks writable by POSIX.

openclaw doctor reports Plugins: Loaded 59, Errors: 0. A direct openclaw agent --local --agent main --message "hi" inside an interactive nemoclaw my-assistant connect session succeeds. Doctor also silently installed 2 plugins (amazon-bedrock, amazon-bedrock-mantle) on first run without surfacing an install prompt.

Failure surface — Telegram bridge path

When the bridge spawns the agent via SSH:

Telegram: Agent exited with code 1. [...]
openclaw Failed to start CLI: PluginLoadFailureError: plugin load failed:
  acpx: failed to install bundled runtime deps: Error: npm error code EACCES
  npm error syscall mkdir
  npm error path /usr/local/lib/node_modules/openclaw/dist/extensions/acpx/node_modules
  npm error errno -13
  npm error Error: EACCES: permission denied, mkdir
    '/usr/local/lib/node_modules/openclaw/dist/extensions/acpx/node_modules'
[... same error repeats for amazon-bedrock, amazon-bedrock-mantle, browser, microsoft ...]

Five plugins fail the same way. Reproduced deterministically via:

OPENSHELL=/home/marcopapa/.local/bin/openshell
confPath=$(mktemp)
$OPENSHELL sandbox ssh-config my-assistant > $confPath
ssh -T -F $confPath openshell-my-assistant \
  "export NVIDIA_API_KEY=$NVIDIA_API_KEY && nemoclaw-start openclaw agent --agent main --local -m 'hi' --session-id tg-debug"

Expected behavior

Expected behavior

After upgrading OpenClaw to 2026.4.21 inside a NemoClaw sandbox via the standard user-prefix npm i -g + host-side docker exec clean sync procedure, the Telegram bridge (and any other channel spawning nemoclaw-start openclaw agent) should successfully load plugins and respond to messages — just as it does on 2026.4.15. The per-plugin runtime dep installer introduced in 4.20 should either succeed against a writable location inside the sandbox, or the sandbox image should ship with those deps pre-installed, or NemoClaw should expose a redirect mechanism so OpenClaw can stage per-plugin deps outside the read-only /usr/local/ tree.

Actual behavior

See Steps to reproduce

OpenClaw version

2026.4.21

Operating system

Nvidia DGX Spark

Install method

npm global

Model

nemotron-3-super-120b-a12b

Provider / routing chain

openclaw -> nemotron

Additional provider/model setup details

No response

Logs, screenshots, and evidence

Impact and severity

Repository: NemoClaw Related: OpenClaw 2026.4.20 plugin dependency architecture change (per-plugin node_modules/ under dist/extensions/<plugin>/) Severity: Blocker for OpenClaw upgrades past 2026.4.19 on NemoClaw sandboxes Reporter: Marco Papa (Wizard), DGX Spark deployment, sandbox my-assistant Date: April 22, 2026

Root cause diagnostics

Sandbox cannot write anywhere under /usr/local/, even as owner

$ for path in /usr/local /usr/local/lib /usr/local/lib/node_modules \
              /usr/local/lib/node_modules/openclaw \
              /usr/local/lib/node_modules/openclaw/dist \
              /usr/local/lib/node_modules/openclaw/dist/extensions \
              /usr/local/lib/node_modules/openclaw/dist/extensions/acpx; do
    touch $path/writetest 2>&1 && echo "WRITE OK: $path" && rm $path/writetest
done
touch: cannot touch '/usr/local/writetest': Permission denied
touch: cannot touch '/usr/local/lib/writetest': Permission denied
touch: cannot touch '/usr/local/lib/node_modules/writetest': Permission denied
touch: cannot touch '/usr/local/lib/node_modules/openclaw/writetest': Permission denied
touch: cannot touch '/usr/local/lib/node_modules/openclaw/dist/writetest': Permission denied
touch: cannot touch '/usr/local/lib/node_modules/openclaw/dist/extensions/writetest': Permission denied
touch: cannot touch '/usr/local/lib/node_modules/openclaw/dist/extensions/acpx/writetest': Permission denied

$ touch /tmp/writetest && echo "TMP OK"       # → TMP OK
$ touch /sandbox/writetest && echo "HOME OK"  # → HOME OK

Pre-creating the target dir from root does not help

# On host, as root via docker exec:
docker exec -u root openshell-cluster-nemoclaw sh -c '
  mkdir -p $ROOTFS/usr/local/lib/node_modules/openclaw/dist/extensions/acpx/node_modules
  chown 999:999 $ROOTFS/usr/local/lib/node_modules/openclaw/dist/extensions/acpx/node_modules
'

# In sandbox, as uid 999 (owner, mode 755):
$ touch /usr/local/lib/node_modules/openclaw/dist/extensions/acpx/node_modules/test
touch: cannot touch '...': Permission denied
$ mkdir /usr/local/lib/node_modules/openclaw/dist/extensions/acpx/node_modules/testdir
mkdir: cannot create directory '...': Permission denied

Security modules and attributes — all cleared

• No AppArmor (/sys/module/apparmor/parameters/enabled empty)
• No SELinux (/sys/fs/selinux/enforce missing)
• No immutable/append attrs (lsattr -d shows only e for extent-format)
• No bind mounts overlaying the path (mount | grep /usr/local empty)
• Sandbox capabilities: CapEff: 0000000000000000 (none effective), CapBnd: 00000004a82c35fb
• Filesystem reports rw: findmnt -T ... shows / as overlay rw,relatime,...

Mount info

Root filesystem is a k3s overlay with 20 lowerdirs and 1 upperdir (snapshot 82). The /usr/local/ tree's read-only enforcement for the sandbox user is not visible in standard mount output. The kernel is denying writes via a mechanism that reports ownership correctly in stat() but rejects write()/mkdir()/creat() syscalls when invoked by the sandbox uid. Best guess: a containerd/k3s pod security policy, a non-POSIX ACL, or an overlay layer quirk where writes from the root context land in the upperdir but the sandbox's view of the merged mount is constrained read-only by the pod's mount namespace configuration.

Interesting: user-prefix install has the deps

$ find /sandbox -name "node_modules" -type d | head
/sandbox/.npm-global/lib/node_modules
/sandbox/.npm-global/lib/node_modules/openclaw/dist/extensions/amazon-bedrock-mantle/node_modules
/sandbox/.npm-global/lib/node_modules/openclaw/dist/extensions/amazon-bedrock/node_modules
/sandbox/.npm-global/lib/node_modules/openclaw/node_modules
[...]

The npm install that ran as the sandbox user under ~/.npm-global did create per-plugin node_modules/ dirs for 2 of the 5 affected plugins (amazon-bedrock, amazon-bedrock-mantle). These survived the cp -rf sync to /usr/local/ for the file contents, but the subsequent on-demand installs inside /usr/local/ still fail because that path is not writable by the sandbox.

Why 4.20+ broke, and 4.15 didn't

OpenClaw 2026.4.20 split plugin runtime deps out of core into per-plugin node_modules/ dirs under dist/extensions/<plugin>/. When a plugin loads and finds its deps missing, the runtime spawns npm install into the plugin's directory. This architectural choice conflicts with NemoClaw sandboxes in the following ways:

1. The install target is inside the read-only /usr/local/ tree.
2. The install happens at plugin load time, which is every agent spawn — not a one-time setup.
3. The writability assumption is invisible — stat() reports correct ownership, so neither the plugin loader nor OpenClaw's doctor detects the read-only constraint until the actual write attempt fails.

OpenClaw 2026.4.15 and earlier keep all runtime deps in /usr/local/lib/node_modules/openclaw/node_modules/ (populated at npm install -g time by the user-prefix mechanism and synced via docker exec cp by the host). No runtime writes to /usr/local/ are ever needed, so read-only enforcement is never hit.

Why this is a NemoClaw issue, not an OpenClaw issue

OpenClaw works correctly on every other deployment (Mac mini, standard Linux, containers without pod security enforcement). The 4.20+ model is reasonable for those environments. The incompatibility is specific to how NemoClaw constructs the sandbox mount namespace — /usr/local/ is not writable for the sandbox user despite stat() reporting otherwise.

Options NemoClaw could take:

1. Provide a writable overlay for /usr/local/lib/node_modules/openclaw/ so the 4.20+ per-plugin installer works. Cleanest fix from the OpenClaw side.
2. Expose an env var (e.g., OPENCLAW_PLUGIN_DEPS_ROOT=/sandbox/.openclaw/plugin-deps) that OpenClaw would honor as the per-plugin dep root instead of dist/extensions/<plugin>/node_modules/. Coordinated fix with OpenClaw.
3. Document the OpenClaw version ceiling for NemoClaw v0.1.0 as 2026.4.19 and surface a hard check in nemoclaw CLI that warns/blocks upgrades past that version until the sandbox image supports 4.20+.

Option 1 or 2 are preferable; option 3 is the minimum-viable safety net.

Additional information

Last known good version 2026.4.15

Rollback procedure (verified working)

1. Inside sandbox: npm i -g [email protected] --loglevel verbose
2. On host: clean sync via docker exec (standard v7/v8 procedure, no chown needed for 4.15)
3. nemoclaw stop && nemoclaw start (persisted TELEGRAM_BOT_TOKEN and NVIDIA_API_KEY in ~/.bashrc)
4. Verify on Telegram: /version returns OpenClaw 2026.4.15 (041266a)

Total rollback time: ~3 minutes.

Request

Please advise on:

1. Preferred fix direction (overlay, env-var redirect, version pinning, or other).
2. Whether NemoClaw's next release plans to bump the sandbox image to something that accommodates the 4.20+ model.
3. Whether there's an already-existing env var or config knob in the current gateway image (ghcr.io/nvidia/openshell/cluster:0.0.15) that would redirect OpenClaw's per-plugin install root out of /usr/local/. The bundle hint "bundle": { "stageRuntimeDependencies": true } in plugin manifests suggests staged installation was a design consideration — is there a way to point that staging dir at /sandbox/...?

Also flagging: openclaw doctor on 4.21 reports Errors: 0 despite 5 plugins having missing deps that will fail at runtime. This is an OpenClaw issue worth cross-filing — doctor should detect missing per-plugin node_modules/ and flag them as errors, not just as an advisory "Bundled plugins" section.

Happy to provide additional traces, run experiments, or test pre-release fixes against this sandbox.

Related context

• Community upgrade guide (NemoClaw sandbox config editing & OpenClaw upgrades) v7 documented the chown dist/extensions/ workaround that enables Mac mini + standard Linux 4.20+ upgrades. v8 attempted to extend the workaround to NemoClaw sandboxes and encountered this incompatibility.
• Previously filed: issue #1063 (npm timeout + Tavily DNS), responded to by Will Curran.
• PR #1062 (Tavily DNS fix) still pending release.