codex - 💡(How to fix) Fix apply_patch / file_change tool deadlocks non-deterministically on file mutations (kind:add ~100%, kind:update ~50%) across codex CLI 0.125→0.133

Summary

The apply_patch / file_change tool in codex exec can deadlock non-deterministically on file mutations. Affected items emit item.started with status: "in_progress" but the matching item.completed event never arrives. turn.completed is never emitted. sandbox_errors stays empty. The spawn must be SIGKILL'd at the wall-clock timeout.

The behavior is probabilistic and variable. In one investigation session (2026-05-25 ~01:00-03:00 PDT), file_change kind:"add" wedged 5/5 times on a minimal repro. In a follow-up session ~18 hours later on the same machine / same codex version, 20/20 spawns of the same repro completed successfully. Both sessions used identical codex CLI versions, identical TOML config, identical briefs. So the bug is intermittent, but when it happens it is reliably a deadlock (not a slow path).

Observed wedge breakdown across the first session (n=11 file-mutation spawns):

• kind: "add" (new file): 5/5 wedges
• kind: "update" (in-place edit): 1/2 wedges (one success, one wedge on same file)
• command_execution shell-write (control): 0/4 wedges

Affected versions

Tested at both endpoints of the available range. Both wedge identically when the deadlock fires:

• @openai/[email protected] (pre-jump baseline)
• @openai/[email protected] (current latest on npm at investigation time)

The bug is not a regression in the 0.125→0.133 range — both endpoints exhibit the same behavior. Mid-range versions (0.126/0.127 are skipped on npm; 0.128/0.129/0.130/0.131/0.132 were not bisected).

Minimal reproduction

echo "Create a new TypeScript file at exactly /tmp/_codex_repro.ts containing:
const probe = \"hello\";

That's all. One file, one line. Use the apply_patch (file_change) tool to create the file." \
  | codex exec --model gpt-5.5 \
      -c model_reasoning_effort=high \
      -c approval_policy=never \
      -c sandbox_workspace_write.writable_roots='["/tmp"]' \
      --sandbox workspace-write \
      --skip-git-repo-check \
      --cd /tmp \
      --json -o /tmp/_codex_repro.final.txt -

When the wedge fires: the process hangs after the file_change item enters status: "in_progress". No subsequent item.completed event. No turn.completed. The file is not created. The hang is indefinite (we observed past 600s).

Evidence (wedged spawn)

JSONL event stream tail at the wedge point:

{"type":"thread.started","thread_id":"019e5cdf-69f0-7843-915d-e4be971089a5"}
{"type":"turn.started"}
{"type":"item.completed","item":{"id":"item_0","type":"agent_message","text":"I'll create the requested file ..."}}
{"type":"item.started","item":{"id":"item_1","type":"command_execution","command":"/bin/zsh -lc 'mkdir -p /tmp'","status":"in_progress"}}
{"type":"item.completed","item":{"id":"item_1","type":"command_execution","command":"/bin/zsh -lc 'mkdir -p /tmp'","exit_code":0,"status":"completed"}}
{"type":"item.started","item":{"id":"item_2","type":"file_change","changes":[{"path":"/tmp/_codex_repro.ts","kind":"add"}],"status":"in_progress"}}
[NO FURTHER EVENTS — process hangs until SIGKILL]

A small audit script (VerifyForgeExecution.ts) we use on every spawn returns:

{
  "codex_executed": false,
  "reason": "codex started but no turn completed — likely premature termination",
  "events_seen": 6,
  "turn_completed_count": 0,
  "item_completed_count": 2,
  "sandbox_errors": []
}

Note sandbox_errors: [] — this is not a sandbox or permissions issue. The writable_roots config is correctly applied; approval_policy=never is correctly applied; command_execution items in the same spawn complete cleanly. The bug is downstream of the sandbox layer.

Non-determinism evidence

The same file, same kind: "update" operation, same brief shape, can produce different outcomes across spawns within a single session:

• One spawn: file_change kind:"update" completes in 17s, file correctly modified, turn.completed present
• Next spawn (same target, same brief): file_change kind:"update" wedges at 90s, file unchanged, no turn.completed

And across sessions: a brief that wedged 5/5 in one session succeeded 20/20 in the next session 18 hours later.

This rules out a deterministic input-shape trigger. It looks like an interaction with backend state (server side?) or system load.

Hypotheses we ruled out

• Sandbox / writable_roots: command_execution items succeed on the same target paths with the same config; sandbox_errors stays empty during wedge. Bug is downstream of sandbox.
• Path-subtree specificity: wedges fire identically on /tmp/*.ts and ~/.claude/**/*.ts.
• Version regression in 0.125→0.133: both endpoints exhibit the bug identically.
• TOML override interaction: codex features list exposes no flag that disables the apply_patch tool in the affected range; tested with and without sandbox_workspace_write.writable_roots overrides.

Workaround (downstream)

Prepending a steering advisory to the brief — instructing the model to use command_execution shell-write (printf > path, cat <<EOF, sed -i, python heredoc) for all file operations instead of file_change — empirically eliminates most wedges. In our wrapper harness:

• Pre-advisory: ~80-100% wedge rate on kind:add (varies by session)
• Advisory only: ~82% success rate (historical average across sessions)
• Advisory + wedge-detect-and-retry harness (SIGTERM at 30s in_progress + one re-spawn with stronger preamble): ~97% combined success

This is a downstream workaround. The upstream apply_patch tool deadlock remains.

Environment

• macOS Darwin 25.3.0 (arm64)
• codex CLI 0.125.0, 0.128.0, 0.133.0 (all three exhibit the bug)
• Model: gpt-5.5, reasoning_effort=high
• Sandbox: workspace-write with TOML writable_roots override
• Approval policy: never (non-interactive codex exec)

Investigation

Investigation conducted with hypothesis-plural probes across four layers:

• H1 — apply_patch deadlock on kind:add → refined to "non-deterministic across kinds, not subtree-specific"
• H2 — workspace-write file lock → refuted via shell-write control
• H3 — version regression → refuted via 0.125.0 + 0.133.0 endpoint test
• H4 — TOML override interaction → refuted via codex features recon

19 production spawns + 7 unit-test assertions captured in our work tree. Full evidence chain (events files, hypothesis probes, control conditions) available on request.

Asks

1. Confirm whether this is a known issue in the apply_patch tool implementation.
2. If known: any expected fix timeline.
3. If new: happy to provide additional repro environments, full events files, or run targeted probes against pre-release builds. We have a small audit harness that can capture full JSONL event streams + timing per spawn.

Thanks for codex — the rest of the tool works very well; this single failure mode has been a recurring tax for non-interactive automation harnesses.