codex - 💡(How to fix) Fix Bug: Agent spawn slots leak across turns in persistent sessions (app-server / interactive CLI) [3 comments, 3 participants]

What version of Codex CLI is running?

codex-cli 0.121.0

What subscription do you have?

pro

Which model were you using?

No response

What platform is your computer?

No response

What terminal emulator and version are you using (if applicable)?

No response

What issue are you seeing?

Bug: Agent spawn slots leak across turns in persistent sessions (app-server / interactive CLI)

Codex version: b0324f9f0569ebfc5534fd6844971d9ae029c791

Summary

In persistent session modes (app-server transport, interactive CLI REPL), the AgentRegistry.total_count used to enforce agent_max_threads is only decremented when close_agent is explicitly called. If a spawned agent reaches a terminal state (Completed, Shutdown, etc.) but the model does not call close_agent before the turn ends, the spawn slot is permanently leaked. Over multiple turns, leaked slots accumulate until total_count >= max_threads, after which all spawn_agent calls fail with AgentLimitReached — even though zero agents are actually running.

This does not affect codex exec because each invocation starts a fresh process with a new AgentRegistry.

Reproduction

1. Start a persistent session (app-server or interactive CLI).
2. In Turn 1, have the model spawn several agents (e.g. 6+). Let some of them complete without the model calling close_agent on each one (this happens naturally when the model runs out of tool-call budget, the turn times out, or the context window fills up before cleanup).
3. In Turn 2, attempt to spawn_agent — it fails with AgentLimitReached { max_threads: N } even though no agents are running.

Observed in production: Over 9 rounds in an app-server session, Round 1 spawned 24 agents but only closed 14. By Round 4, spawn_agent was permanently blocked (spawned=0, close=0) for the remainder of the session.

Round 1: spawned=24, close=14, live=7  → 10 slots leaked
Round 2: spawned=6,  close=4,  live=4  →  2 more leaked
Round 3: spawned=2,  close=2,  live=2  →  0 new leaked (but cumulative damage done)
Round 4–9: spawned=0, close=0          →  permanently stuck

Root Cause Analysis

1. total_count increment path

AgentRegistry::reserve_spawn_slot() increments total_count via try_increment_spawned():

// codex-rs/core/src/agent/registry.rs:80-97
pub(crate) fn reserve_spawn_slot(
    self: &Arc<Self>,
    max_threads: Option<usize>,
) -> Result<SpawnReservation> {
    if let Some(max_threads) = max_threads {
        if !self.try_increment_spawned(max_threads) {
            return Err(CodexErr::AgentLimitReached { max_threads });
        }
    } else {
        self.total_count.fetch_add(1, Ordering::AcqRel);
    }
    Ok(SpawnReservation { state: Arc::clone(self), active: true, ... })
}

2. total_count decrement path — only via explicit shutdown

release_spawned_thread() is the only path that decrements total_count:

// codex-rs/core/src/agent/registry.rs:99-119
pub(crate) fn release_spawned_thread(&self, thread_id: ThreadId) {
    let removed_counted_agent = { /* remove from agent_tree */ };
    if removed_counted_agent {
        self.total_count.fetch_sub(1, Ordering::AcqRel);
    }
}

This is called from exactly two places:

• AgentControl::shutdown_live_agent() (control.rs:676) — called by close_agent tool handler
• AgentControl::handle_thread_request_result() (control.rs:655) — only on InternalAgentDied

3. wait_agent does NOT release slots

Both v1 (multi_agents/wait.rs) and v2 (multi_agents_v2/wait.rs) wait_agent handlers only observe status — they never call release_spawned_thread or shutdown_live_agent.

4. Turn completion does NOT release slots

There is no turn-boundary cleanup hook. When a turn completes, the AgentRegistry state (including total_count and agent_tree) is carried forward unchanged into the next turn.

5. maybe_start_completion_watcher does NOT release slots

The completion watcher (control.rs:898-971) watches for agents reaching final status and notifies the parent thread, but it does not call release_spawned_thread.

6. Why codex exec is immune

Each codex exec invocation starts a fresh OS process → fresh AgentRegistry → total_count starts at 0. Even with codex exec resume <thread_id>, the AgentRegistry is reconstructed from scratch (only agents with Open edge status are re-reserved via resume_agent_from_rollout).

Why the model doesn't always close_agent

The model is expected to call close_agent after wait_agent, but in practice this doesn't always happen:

• The model may exhaust its tool-call budget before closing all agents
• The turn may time out mid-execution
• Context window compaction may drop the close_agent intent
• With many concurrent agents (12+), the model may not track all IDs

This is especially problematic in automated/agentic workflows where turns are driven programmatically and agent counts are high.

Proposed Fix

Release spawn slots for agents in terminal state before enforcing the limit.

Modify AgentRegistry::reserve_spawn_slot() to reap finalized agents before checking the count:

pub(crate) fn reserve_spawn_slot(
    self: &Arc<Self>,
    max_threads: Option<usize>,
) -> Result<SpawnReservation> {
    if let Some(max_threads) = max_threads {
        if !self.try_increment_spawned(max_threads) {
            // Before failing, try to reclaim slots from finalized agents
            self.reap_finalized_agents();  // new method
            if !self.try_increment_spawned(max_threads) {
                return Err(CodexErr::AgentLimitReached { max_threads });
            }
        }
    } else {
        self.total_count.fetch_add(1, Ordering::AcqRel);
    }
    Ok(SpawnReservation { ... })
}

Alternative approaches (non-exclusive):

1. Reap in reserve_spawn_slot (above) — minimal change, lazy cleanup only when needed.
2. Auto-release on turn completion — add a turn-boundary hook that calls release_spawned_thread for all agents in agent_tree whose status is final (Completed, Shutdown, Errored, NotFound).
3. Auto-release in maybe_start_completion_watcher — when the watcher observes a final status, also call release_spawned_thread (not just notify the parent).

Option 2 or 3 would keep total_count accurate at all times rather than only at spawn time.

Impact

• Severity: High for persistent sessions (app-server, interactive REPL)
• Scope: Any multi-turn workflow that spawns agents across turns without perfectly closing each one
• Workaround: None available to API consumers — the AgentRegistry is internal and not exposed. The only current workaround is to destroy and recreate the entire session, losing all conversation context.

Files involved

What steps can reproduce the bug?

n

What is the expected behavior?

No response

Additional information

No response