hermes - 💡(How to fix) Fix tool message content must be string: plugin tools returning dict cause upstream 400 (Z.ai error 1210, OpenAI/Manifest fallback_exhausted)

Summary

Plugin tool handlers that return Dict[str, Any] get persisted into the chat history as the literal Python dict for the role: "tool" message content field. OpenAI Chat Completions spec requires tool message content to be a string. Strict upstream validators reject this with HTTP 400. Permissive providers (e.g. OpenAI direct) silently coerce, masking the bug.

Reproduction

Any conversation that calls a plugin tool whose handler returns a dict will eventually fail when the next turn is sent to a strict upstream.

Concrete repro in this repo: all five plugins/workflow-engine/tools/*.py handlers are annotated -> Dict[str, Any] and return {...}:

• list_workflows.py → {"definitions": [...], "count": N}
• run_workflow.py → {"run_id": ..., "status": ..., "ok": True}
• workflow_status.py → {"run_id": ..., "status": ..., ...}
• approve_workflow.py → {"ok": True, ...}
• cancel_workflow.py → {"ok": True, ...}

After any of these is called in a conversation, the next chat/completions request contains:

{ "role": "tool", "tool_call_id": "...", "content": { "definitions": [...], "count": N } }

Observed upstream errors

• Z.ai (GLM-5.1): {"error":{"code":"1210","message":"API 调用参数有误，请检查文档。"}} ("API parameters incorrect")
• Manifest proxy aggregator: {"error":{"message":"Bad request to upstream provider","type":"upstream_error","status":400}}
• Manifest fallback chain: {"type":"fallback_exhausted","status":400,...} (because every provider in the chain rejects the same malformed payload)

Evidence

Captured 10/10 recent request_dump_*.json files for one session, all show msg with role:"tool" and dict-typed content, tool_call_id linking back to a workflow_list invocation. Pattern is 100% deterministic, not flaky.

Gateway log shows Repaired N message-alternation violations before request but the alternation repair does not normalize content type.

Proposed fix (defensive, one place)

Add a normalizer to agent/message_sanitization.py that runs before every outbound request:

import json

def _ensure_tool_content_is_string(messages: list) -> bool:
    \"\"\"Coerce non-string content on role=tool messages to JSON string.
    OpenAI spec mandates string content; some plugin handlers return dicts.\"\"\"
    found = False
    for msg in messages:
        if msg.get("role") == "tool" and not isinstance(msg.get("content"), (str, type(None))):
            msg["content"] = json.dumps(msg["content"], ensure_ascii=False, default=str)
            found = True
    return found

Wire into the existing sanitize pipeline (alongside _sanitize_structure_non_ascii etc.).

This protects every current and future plugin tool from the same mistake, even when authors return dicts.

Secondary fix (plugin layer)

Update all five plugins/workflow-engine/tools/*.py handlers to either:

• Return str via json.dumps(...), or
• Be wrapped by the dispatcher in tools/registry.py so any non-string return gets JSON-stringified before storage.

Same fix should be audited for other plugins (mcp_tool.py already does _extract_tool_result_text; non-MCP plugin handlers bypass that path).

Impact

• Web chat + Telegram bot intermittently fail with HTTP 400 every time a workflow tool is in conversation history
• Manifest's fallback_exhausted masks the actual upstream error string, making the bug very hard to diagnose
• Affects any strict OpenAI-compatible provider downstream (Z.ai confirmed; likely DeepSeek, Mistral, etc.)

Workaround

Patched all 5 workflow_engine tool handlers locally to return json.dumps(...). Rolling out via gateway restart.