hermes - 💡(How to fix) Fix [Feature]: replace codex intermediate-ack heuristic with a cheap auxiliary completion classifier [1 comments, 2 participants]

Problem or Use Case

Hermes currently uses a heuristic _looks_like_codex_intermediate_ack() check to decide whether a Codex response is just an intermediate acknowledgement ("I'll inspect the repo", "let me check that", etc.) and whether the turn should continue.

This helps avoid premature turn termination, but the current keyword-based approach is brittle:

• it can false-positive on normal responses that happen to mention action/workspace terms
• it can false-negative on intermediate acknowledgements phrased differently
• the rule is difficult to keep complete across providers, models, and prompt styles

In practice, this means turn control is partially driven by a fragile text heuristic rather than a more explicit completion signal.

Proposed Solution

Add an optional lightweight auxiliary classification step for ambiguous Codex responses.

Suggested behavior:

• keep the current fast heuristic as the first-pass cheap path
• only for ambiguous cases, call a cheap auxiliary text model to classify the response as one of:
  • final answer
  • intermediate acknowledgement / planning utterance
  • incomplete generation
• continue the turn only when the classifier returns intermediate/incomplete
• keep the current heuristic as a fallback when no auxiliary provider is configured

This would preserve the low-cost path for obvious cases while reducing both false positives and false negatives.

Alternatives Considered

• Expand the heuristic keyword list further
  This improves coverage a bit but does not solve the core brittleness problem.
• Rely only on provider-native finish reasons
  This is cleaner, but some real-world outputs still appear ambiguous at the text layer.
• Always invoke a second model
  This is likely too expensive and adds latency; a hybrid heuristic + classifier approach seems more practical.

Feature Type

Performance / reliability

Scope

Medium (few files, < 300 lines)

Contribution

• I'd like to implement this myself and submit a PR

Debug Report (optional)