hermes - 💡(How to fix) Fix BUG: tool_use parser fails intermittently when closing tag appears inside JSON strings

Summary

When using provider=claude-subscription (or any OpenAI-compatible endpoint with Claude models), the model occasionally emits tool calls as <tool_use>{...}</tool_use> XML text in the content field instead of structured tool_calls. The existing fallback parser in _extract_tool_use_from_content() fails intermittently when the JSON arguments contain </tool_use> inside a string value (e.g. write_file content with HTML tags).

Root Cause

The current regex-based parser at run_agent.py:1556:

pattern = re.compile(
    r"<tool_use>\s*(.*?)\s*</tool_use>|<tool_use>\s*(.*)", re.DOTALL
)

The lazy quantifier (.*?) matches the first </tool_use> it encounters, even if that tag is inside a JSON string. This produces invalid JSON that fails json.loads(), causing the entire tool_use block to be treated as plain text and displayed to the user.

Example failure case

text = '<tool_use>{"name": "write_file", "arguments": {"content": "</tool_use>"}}</tool_use>'
# Regex captures: '{"name": "write_file", "arguments": {"content": "'
# json.loads() fails → tool_use treated as TEXT

Why intermittent

• Works when JSON does NOT contain </tool_use> in strings
• Fails when JSON contains </tool_use> in strings (write_file with HTML, web content, etc.)
• "Re-emit works" because the model generates different JSON on retry, or the provider correctly converts to structured tool_calls

Files affected

• run_agent.py:1536-1585 — _extract_tool_use_from_content() method
• run_agent.py:5137-5150 — fallback invocation in _build_assistant_message()

Proposed Fix

Replace the regex with an iterative scanner that finds the first </tool_use> producing valid JSON with a "name" field:

def _extract_tool_use_from_content(self, text: str):
    ...
    while True:
        start_idx = remaining.find("<tool_use>")
        ...
        # Scan for first </tool_use> that produces valid JSON with "name"
        while True:
            candidate_end = remaining.find("</tool_use>", search_pos)
            candidate_json = remaining[content_start:candidate_end].strip()
            try:
                tc_data = json.loads(candidate_json)
                if "name" in tc_data:
                    end_idx = candidate_end
                    break
            except (json.JSONDecodeError, KeyError):
                pass
            search_pos = candidate_end + 1
    ...

Test Plan

1. Run python test_tool_use_reproducer.py — all 9 edge cases should pass
2. Critical new edge cases:
   • </tool_use> inside JSON string value
   • Multiple tool_use blocks with mixed content
   • Explanatory text mentioning <tool_use> before actual tool call

Environment

• Provider: claude-subscription (OpenAI-compatible endpoint)
• Profile: victor
• Hermes version: latest main
• Recurrent: yes
• Non-deterministic: yes (depends on model output)

Deliverables

• Root cause identified
• Fix proposed with diff
• Test reproducer updated with edge cases
• PR opened (pending maintainer review)

/cc @sergionsantos