hermes - 💡(How to fix) Fix Hindsight retain should strip inline base64 data URLs before extraction [1 pull requests]

Bug description

Hermes can pass inline base64 data URLs from multimodal/message payloads directly into Hindsight retain. When the retained text contains something like data:image/png;base64,<very large payload>, Hindsight treats the payload as normal text and sends it through memory extraction.

That makes one image-sized payload look like millions of meaningless text characters. In a local incident, this caused repeated extraction attempts, quota/rate-limit errors, and more than $100 of OpenAI API spend before the queue was stopped.

Impact

• Large inline data:*;base64,... payloads can be retained into long-term memory.
• Hindsight extraction can repeatedly process the same non-semantic payload.
• The retained content is not useful memory, but it is very expensive to tokenize/extract.
• This can cause runaway API usage, 429/quota failures, and degraded memory reliability.

Observed locally, using redacted/summarized logs only:

• Retain payload size: approximately 5.67 MB
• Repeated extraction attempts: approximately 197
• Error mode: OpenAI 429/quota/rate-limit failures
• Cost impact: more than $100 of OpenAI API usage

No API keys, billing screenshots, or private payloads are included here.

Reproduction sketch

A synthetic retain payload is enough to trigger the shape of the bug:

content = "please inspect this image: data:image/png;base64," + ("A" * 5_000_000)
provider.handle_tool_call("hindsight_retain", {"content": content})

Similar payloads can also arrive through auto-retained turn content when multimodal messages include image_url.url as a data URL.

Expected behavior

Hermes/Hindsight should not retain raw base64 data URL payloads as memory text. At minimum, the binary payload should be stripped or replaced with compact metadata before extraction.

Example safe retained text:

[omitted base64 image/png data URL from Hindsight retain, data_url_chars=5000022]

Actual behavior

The raw data:image/...;base64,... string can be retained and sent to Hindsight extraction as ordinary text.

Proposed fix

Sanitize base64 data URLs at Hermes' Hindsight retain boundary:

• sanitize direct hindsight_retain tool content
• sanitize direct hindsight_retain tool context
• sanitize _retain_memory content/context
• sanitize auto-retained user/assistant turn messages before queueing
• avoid logging raw retain context
• preserve surrounding semantic text
• replace the base64 payload with MIME/length metadata

A PR with tests is being opened against this issue.