openclaw - 💡(How to fix) Fix Per-request startup regression in 4.29: ~25–50s overhead before model output (core-plugin-tools 11s, auth 4–12s, model-resolution 8s) [2 comments, 3 participants]

Summary

v2026.4.29 introduced a per-request startup regression that adds 25–50 seconds of overhead before the model starts streaming. It's reproducible on every embedded-run, not a one-off cold-start cost. Rolling back to v2026.4.23 restores normal latency.

Symptoms after upgrading 4.23 → 4.29

User sends a message. Gateway receives it. ~25–50 seconds pass before the model produces the first byte. Then the model itself runs at normal speed. Subsequent messages do not "warm up" — every single request pays the same overhead.

Embedded-run trace stages on 4.29

Sample request — chat continuation, ~840 char query:

phase=attempt-dispatch totalMs=15973 stages=
  workspace:0ms@0ms,
  runtime-plugins:7ms@7ms,
  hooks:1ms@8ms,
  model-resolution:8503ms@8511ms,
  auth:4264ms@12775ms,
  context-engine:2ms@12777ms,
  attempt-dispatch:3196ms@15973ms

phase=stream-ready totalMs=24578 stages=
  workspace-sandbox:250ms@250ms,
  skills:63ms@313ms,
  core-plugin-tools:11004ms@11317ms,
  bootstrap-context:523ms@11840ms,
  bundle-tools:1002ms@12842ms,
  system-prompt:4226ms@17068ms,
  session-resource-loader:2193ms@19261ms,
  agent-session:1ms@19262ms,
  stream-setup:5316ms@24578ms

Second consecutive request (different runId/session, same conversation, model-resolution now warm):

phase=attempt-dispatch totalMs=26504 stages=
  ...
  model-resolution:640ms@642ms,
  auth:11687ms@12329ms,
  context-engine:0ms@12329ms,
  attempt-dispatch:14175ms@26504ms

The same workload on 4.23 completes end-to-end in well under 10 seconds. (4.23 doesn't emit these trace:embedded-run stage logs, so I can't directly diff numbers, but observed latency is dramatically lower.)

Hot path

core-plugin-tools (11s) is bracketed by prepStages.mark("core-plugin-tools") in dist/selection-*.js, right after createOpenClawCodingTools(...) and applyEmbeddedAttemptToolsAllow(...). Runs on every embedded-run.

auth (4–12s) and model-resolution (8.5s on first hit) also dominate the first phase.

I grepped the dist for EAGER, PRELOAD, cacheTools, etc. — there's no apparent way to amortize this work via config or env.

Reproduction

1. npm install -g [email protected], restart gateway, send a message — note end-to-end latency.
2. npm install -g [email protected], restart gateway, send the same message — note ~25–50s overhead before model output.
3. Check gateway logs for [trace:embedded-run] lines to see the stage breakdown.

Workaround

npm install -g [email protected] resolves the issue. Config is forward-compatible.

Hypothesis

createOpenClawCodingTools() and/or applyEmbeddedAttemptToolsAllow() likely got new sync I/O or schema-resolution work added in 4.29. Same for the auth and model-resolution phases. Either:

• the per-request work could be done once at gateway startup and cached, or
• the hot path could be made async/parallel with the actual model call.

Happy to provide more traces if useful.