hermes - 💡(How to fix) Fix Gateway should not treat stale Codex app-server progress as final response after post-tool silence

Problem

During a Telegram /codex-pipeline-auto #1458 run on 2026-05-28, the Codex app-server runtime went silent for 90 seconds after a tool result. Hermes correctly retired the underlying app-server session, but the gateway treated stale non-empty progress/assistant text as the final response and did not retry or continue the workflow.

This is a durability failure for long-running gateway jobs: the gateway process stayed alive, but the user-visible task stopped without a recovery attempt.

Repro / Evidence

Observed on the Alfred Mac Hermes gateway:

1. Send Telegram command /codex-pipeline-auto #1458.
2. The run starts under session 20260528_095004_3b1543 with openai-codex / gpt-5.5 and api_mode=codex_app_server.
3. Codex completes at least one tool-shaped item, then emits no useful follow-up for 90s.
4. ~/.hermes/logs/agent.log shows at line ~7555: agent.codex_runtime: codex app-server session retired (turn error: codex went silent for 90s after a tool result; retiring app-server session.)
5. Immediately afterward the gateway logs: gateway.run: response ready: platform=telegram chat=7374830578 time=448.2s api_calls=1 response=220 chars and sends that 220-char response.
6. Gateway health was not the failure: memory heartbeat logs continued afterward at 10:11, 10:16, 10:21, 10:26, and 10:31.
7. User-level side effects had occurred: ~/.hermes/skills pipeline files were created around 09:55, so git status in the repo being clean did not mean the job made no changes.

A similar app-server retirement also appeared on 2026-05-27 23:05 and on 2026-05-28 09:48, but this issue tracks the gateway durability behavior observed during the Telegram pipeline run.

Observed Behavior

• CodexAppServerSession detects post-tool silence and sets TurnResult.should_retire=True, interrupted=True, and error="codex went silent for 90s after a tool result; retiring app-server session.".
• agent.codex_runtime.run_codex_app_server_turn() retires agent._codex_session but still returns final_response=turn.final_text with partial=True and error set.
• Gateway response normalization appears to return early when final_response is non-empty, so the partial/error state does not force a retry/continuation or a clear failure envelope.
• Telegram receives stale non-empty text as if the run completed, and the pipeline does not auto-retry or continue with a fresh app-server session.

Expected Behavior

When a Codex app-server turn ends with partial=True, error, or completed=False because the runtime was retired after post-tool silence, the gateway should treat that as an incomplete run even if final_response contains text.

The gateway should either:

1. automatically retry/continue once with a fresh app-server session while preserving projected tool results and transcript state, or
2. surface a clear durable failure that says the app-server was retired and the workflow was not completed, without treating stale progress text as the answer.

In either path, progress/heartbeat/interim assistant text must not satisfy the final-response contract for a partial app-server turn.

Acceptance Criteria

• Telegram/gateway does not mark a Codex app-server post-tool-silence retirement as a successful final response solely because final_response is non-empty.
• A partial app-server result with error or completed=False produces either a bounded automatic continuation/retry or an explicit incomplete-run message.
• Retry/continuation uses a fresh Codex app-server session after should_retire=True and preserves projected messages/tool results from the failed turn.
• Regression coverage covers final_response non-empty + partial=True + error in gateway response normalization/delivery.
• Regression coverage covers the Codex session post-tool watchdog path and verifies should_retire=True is propagated into gateway behavior.
• Logs distinguish retired and retrying/continuing from retired and surfaced incomplete failure.

Code Pointers

• agent/transports/codex_app_server_session.py around the post-tool quiet watchdog: sets interrupted, error, and should_retire when Codex is silent after a tool result.
• agent/codex_runtime.py around run_codex_app_server_turn(): retires agent._codex_session when turn.should_retire is set, but returns final_response=turn.final_text, partial=True, and error.
• gateway/run.py::_normalize_empty_agent_response(): currently returns early for any non-empty response, before checking failed, partial, error, or completed=False.
• gateway/run.py final-send suppression and progress cleanup paths around _run_agent() completion should preserve the distinction between final content and progress/interim content.

Suggested Labels

type/bug, comp/gateway, comp/agent, provider/openai, platform/telegram, P1