Summary

On openclaw 2026.4.26 running in a Kubernetes pod with a single custom channel plugin and a single proxy-style model provider configured (plugins.allow = ["my-channel", "my-proxy-provider"]), the gateway takes ~2m42s between [gateway] starting channels and sidecars... and the gateway calling the channel's gateway.startAccount callback.

During this window, /readyz and /healthz time out. The Node.js event loop is synchronously blocked for stretches of up to ~37 s at a time; an nginx sidecar in front of the gateway returns 504 Gateway Timeout to kubelet probes throughout.

Boot timing on 2026.4.21 for the same deployment was ~32 s (~80× faster). The regression appeared on first bump to 2026.4.26 (haven't bisected the intermediate releases yet).

Reproduction

K8s StatefulSet with:

• 1 custom HTTP-webhook channel plugin (no persistent connection)
• A single proxy-style model provider (the model is a <proxy-provider>/<vendor>/<model> triplet)
• nginx sidecar in front of the gateway, proxying /readyz and /healthz
• workspace PVC mounted at /home/node/.openclaw
• StartupProbe with 5s × 36 = 180 s ceiling (anything < 300 s misfires on the first cold boot)

Expected: ~30–60 s gateway warm-up.

Actual: ~2m42 s of silent gateway work between [gateway] starting channels and sidecars... and the first channel-ready log line.

Root cause (instrumented)

I patched the gateway dist with console.time-style markers to bisect the silent sidecars.channels step. With OPENCLAW_GATEWAY_STARTUP_TRACE=1 plus the patch:

sidecars.channels                   157,007 ms   eventLoopMax = 37,212 ms
└── prewarmConfiguredPrimaryModel   162,517 ms       ← THE BOTTLENECK
    ├── ensureOpenClawModelsJson     39,757 ms  (~40 s — matches eventLoopMax)
    └── resolveModel (sync)         122,399 ms  (~2 m to resolve ONE configured model, resolved=true)
└── params.startChannels()               85 ms  (channel start itself is healthy)

resolveModel is the synchronous path in model-CkUlgtmi.js:

const authStorage   = options?.authStorage   ?? discoverAuthStorage(resolvedAgentDir);
const modelRegistry = options?.modelRegistry ?? discoverModels(authStorage, resolvedAgentDir);

discoverModels instantiates the Pi model registry (createOpenClawModelRegistry → instantiatePiModelRegistry). Both discoverAuthStorage and discoverModels are fully synchronous, so they block the Node event loop for the duration. With an nginx sidecar in front of the gateway, every probe hit during this window returns 504.

Investigations ruled out

I have logs available for each of these.

• ❌ Codex live model discovery — disabled via plugins.entries.codex.config.discovery.enabled = false (saved ~30 s and eliminated codex_app_server stderr, but not the bulk).
• ❌ Node.js compile-cache — wiped the cache on the PVC; cold boot was only ~7 s slower than warm.
• ❌ Plugin runtime-deps materialization — log shows installed bundled runtime deps in 908 ms.
• ❌ EBUSY on plugin auto-enable — single warning per boot, no retry loop.
• ❌ Provider-discovery live catalog calls — only codex (disabled) and anthropic-vertex (returns null fast without GCP creds) actually run live catalogs.
• ❌ Channel start itself — the channel plugin's startAccount callback runs in 85 ms; the freeze is upstream of any plugin-side work.

Suggested fixes (rough order of cleanliness)

1. Cache the Pi registry instantiation. After first cold boot, subsequent in-process resolves should be sub-ms; even better, bake a serialized snapshot into the workspace and skip instantiatePiModelRegistry entirely on cache-hit.
2. Skip prewarmConfiguredPrimaryModel for proxy-style providers where Pi's registry isn't actually consulted at chat time. The configured primary model in the failing deployment is a proxy passthrough — Pi resolves it as resolved=true after 2 minutes of work that produces no useful state.
3. Yield the event loop inside instantiatePiModelRegistry. Chunk by setImmediate every N providers so /readyz can answer 503 quickly during cold boot instead of blocking long enough for upstream proxies (nginx, Envoy, etc.) to time out.
4. Add OPENCLAW_SKIP_MODEL_PREWARM=1 (or a config flag like agents.defaults.skipPrewarm = true). Currently the only way to skip prewarm is to set OPENCLAW_AGENT_RUNTIME to a non-auto/non-pi value, which has chat-time side effects.
5. Have plugins.allow also gate provider/channel discovery enumeration for bundled plugins — isProviderPluginEligibleForSetupDiscovery (in providers-CjE0WvyR.js) returns true unconditionally for non-workspace bundled plugins regardless of plugins.allow / plugins.deny / plugins.entries.<id>.enabled. A tightly scoped deployment shouldn't pay enumeration cost for ~120 unused bundled plugins.

Logs

Sanitized agent log excerpt from a typical boot (timestamps relative to gateway start):

+0s     [gateway] loading configuration…
+1s     [gateway] starting...
+1s     Config warnings:
        - plugins.entries.codex: plugin disabled (bundled (disabled by default)) but config is present
+1s     [gateway] failed to persist plugin auto-enable changes: Error: EBUSY
+2s     [gateway] starting HTTP server...
+2s     [gateway] http server listening (...; 2.4 s)
+3s     [gateway] starting channels and sidecars...
                                           ← 2m42 s of complete log silence
+2m45s  [<my-channel>] channel ready
+2m45s  [gateway] ready
+2m45s  [heartbeat] started

With OPENCLAW_GATEWAY_STARTUP_TRACE=1:

+760ms  startup trace: sidecars.session-locks       1.5 ms
+760ms  startup trace: sidecars.gmail-watch         0.0 ms
+760ms  startup trace: sidecars.gmail-model         0.0 ms
+760ms  startup trace: sidecars.internal-hooks      0.0 ms
                                          ← prewarmConfiguredPrimaryModel runs here, 2m42 s
+2m43s  startup trace: sidecars.channels           157,007 ms   eventLoopMax=37,212 ms
+2m43s  startup trace: sidecars.plugin-services       0.6 ms
+2m43s  startup trace: sidecars.memory                0.0 ms
+2m43s  startup trace: sidecars.restart-sentinel     78 ms
+2m43s  startup trace: sidecars.subagent-recovery     3.7 ms
+2m43s  startup trace: sidecars.main-session-recovery 2.7 ms
+2m43s  startup trace: sidecars.total              160,053 ms
+2m43s  startup trace: ready                          0.5 ms

Happy to share verbose / instrumented runs or the diagnostic patch on request.