openclaw - 💡(How to fix) Fix Codex context-engine projection lacks exact pre-turn token accounting [2 pull requests]

Summary

Follow-up to #80760 and #80761.

#80761 makes the native Codex context-engine projection budget-aware and maps the rendered-context cap through the existing openclaw.json compaction reserve fields:

• agents.defaults.compaction.reserveTokens
• agents.defaults.compaction.reserveTokensFloor

That fixes the main send-side bug where LCM/context-engine could assemble a large frontier, but native Codex still received only a fixed 24k-character projected context block.

There is still a smaller accounting gap: before turn/start, the Codex app-server projection currently sizes the rendered context using contextTokenBudget, reserve config, and OpenClaw's normal approximate 4 chars/token text heuristic. Exact provider/runtime token usage is only available after the turn completes.

Why this matters

The new projection cap is correctly proportional, but it is still estimate-based. That can leave a slight mismatch between:

• what OpenClaw expects to send before the turn,
• what Codex actually tokenizes for the rendered turn/start input,
• what LCM/status can later report from observed post-turn usage.

This is usually small for ordinary Latin text, but it can drift for high-token-density content, CJK/multilingual content, unusual formatting, large quoted context blocks, or future Codex tokenizer/model changes.

Current behavior

The active Codex context-engine path now effectively does:

1. context engine assembles messages for the current runtime budget
2. OpenClaw projects those messages into a quoted Codex prompt string
3. projection uses contextTokenBudget - reserveTokens and a 4 chars/token estimate to choose a char cap
4. Codex receives turn/start
5. exact-ish usage/observed prompt tokens are only known after the turn

So OpenClaw can protect against gross overflow, but cannot pre-turn prove the exact rendered prompt token count.

Expected direction

Add a native Codex pre-turn accounting path or telemetry seam so projection/status can report exact or authoritative token accounting where available.

Possible implementation shapes:

• expose a Codex app-server/tokenizer API for the exact turn/start input token count before send
• add a provider/model tokenizer utility for the rendered Codex prompt string and developer instructions
• return projection stats from the Codex app-server route, including projectedPromptChars, estimated prompt tokens, reserve tokens, cap chars, and exact input tokens when available
• pass those stats into context-engine afterTurn/status telemetry so LCM can distinguish selected frontier, projected prompt, and provider-observed usage

Acceptance criteria

• Native Codex context-engine projection can report pre-turn exact token count when the runtime/tokenizer surface supports it, or explicitly mark it as estimated when not available.
• /status or equivalent diagnostics can distinguish:
  • LCM/frontier tokens selected by the context engine
  • rendered Codex projection chars/tokens before send
  • provider-observed prompt/input tokens after the turn
• Tests cover the estimate-vs-exact marker and ensure configured reserve fields remain honored.

Notes

This is intentionally separate from #80761. The PR fixes the observed fixed-cap truncation bug. This issue tracks the remaining observability/accounting gap so future debugging does not confuse LCM frontier size with Codex rendered prompt usage.