openclaw - 💡(How to fix) Fix [Bug]: All cron isolated agent runs serialize on nested lane (maxConcurrent=1) — blocks parallel cron execution [1 comments, 2 participants]

Summary

All cron isolated agent runs are serialized through the nested command lane (maxConcurrent=1), regardless of cron.maxConcurrentRuns configuration. This means only one cron job can execute at a time across the entire gateway, even when jobs target different agents, different sessions, and have no shared state.

Root Cause (source code trace)

1. server-cron.ts:302 hardcodes lane: "cron" for all isolated agent jobs
2. isolated-agent/run.ts:645 calls resolveNestedAgentLane(params.lane) → returns CommandLane.Nested
3. pi-embedded-runner/run.ts:285-286 wraps the entire runEmbeddedPiAgent in enqueueGlobal():

   return enqueueSession(() =>
       enqueueGlobal(async () => {
           // entire agent run — LLM calls, tool use, everything
       })
   );

4. pi-embedded-runner/lanes.ts:11: resolveGlobalLane("cron") → CommandLane.Nested
5. gateway-cli-*.js: applyGatewayLaneConcurrency() sets concurrency for Main, Cron, and Subagent lanes but omits Nested, leaving it at default maxConcurrent=1

Steps to Reproduce

# Create 8 cron jobs firing simultaneously
for i in 1 2 3 4 5 6 7 8; do
  openclaw cron create \
    --name "test-lane-$i" \
    --cron "30 18 * * *" \
    --tz "Asia/Shanghai" \
    --agent main \
    --session isolated \
    --message "Write a 300-word article." \
    --no-deliver --exact --timeout-seconds 120
done

# Trigger all simultaneously
for id in $(openclaw cron list | grep "test-lane" | awk '{print $1}'); do
  openclaw cron run "$id" &
done
wait

Observed Behavior

Debug logs (/tmp/openclaw/openclaw-*.log) show all 8 jobs queue in nested lane and execute serially:

18:32:42 lane enqueue: lane=nested queueSize=1
18:32:42 lane enqueue: lane=nested queueSize=2
...
18:32:43 lane enqueue: lane=nested queueSize=8
18:32:57 lane task done: lane=nested durationMs=14868 active=0 queued=7
18:33:17 lane task done: lane=nested durationMs=19416 active=0 queued=6
...
18:34:31 lane task done: lane=nested durationMs=13634 active=0 queued=1

Each job takes ~15s. Total wall time: ~120 seconds (serial). Last job waits 108 seconds before starting.

In production with 42 cron jobs (4 platforms × ~10 jobs each), wait times reach 15-30 minutes.

Expected Behavior

Independent cron jobs targeting different agents/sessions should execute in parallel, up to cron.maxConcurrentRuns (which is already configurable and was fixed in #11595).

cron.maxConcurrentRuns: 32 currently allows 32 jobs to be dispatched simultaneously, but they all block on nested lane maxConcurrent=1 for actual execution.

Workaround

Patching applyGatewayLaneConcurrency() in gateway-cli-*.js:

function applyGatewayLaneConcurrency(cfg) {
    setCommandLaneConcurrency(CommandLane.Cron, cfg.cron?.maxConcurrentRuns ?? 1);
    setCommandLaneConcurrency(CommandLane.Main, resolveAgentMaxConcurrent(cfg));
    setCommandLaneConcurrency(CommandLane.Subagent, resolveSubagentMaxConcurrent(cfg));
    setCommandLaneConcurrency(CommandLane.Nested, 8); // ← added
}

After patch: all 8 jobs run in parallel, total wall time ~15 seconds, zero queue wait:

18:59:45 lane enqueue: lane=nested queueSize=8
18:59:45 lane dequeue: lane=nested waitMs=0 queueSize=0  (all 8 dequeued immediately)
18:59:59 lane task done: lane=nested durationMs=13984 active=7 queued=0
19:00:00 lane task done: lane=nested durationMs=15344 active=4 queued=0

Suggested Fix

Make nested lane concurrency configurable, similar to other lanes:

{
  "agents": {
    "defaults": {
      "nestedMaxConcurrent": 8
    }
  }
}

And in applyGatewayLaneConcurrency():

setCommandLaneConcurrency(
    CommandLane.Nested,
    cfg.agents?.defaults?.nestedMaxConcurrent ?? 4
);

Session-level serialization (enqueueSession) already prevents concurrent access to the same session, so increasing nested lane concurrency should be safe for independent jobs.

Difference from #14214

#14214 reported the same nested lane bottleneck but for sessions_send broadcasts. This issue is about cron isolated agent runs — the entire runEmbeddedPiAgent() call is wrapped in enqueueGlobal() routed to nested lane, serializing all cron execution.

Environment

• OpenClaw: 2026.3.13 (61d171a)
• OS: macOS (Apple Silicon)
• Node.js: v22.22.0