PR #65783: fix(memory): preserve surrogate pairs in chunker; sanitize embed inputs

• Repository: openclaw/openclaw
• Author: jensenwang560-blip
• State: open | merged: False
• Link: https://github.com/openclaw/openclaw/pull/65783

Description (problem / solution / changelog)

Summary

• Fix the outer coarse-split loop in chunkMarkdown so it does not bisect a UTF-16 surrogate pair (root cause of indexing stalls).
• Add a defensive stripUnpairedSurrogates sanitizer at the embed boundary (the safety net #27753 originally requested).
• Tests for both, including a new regression test that triggers the exact code path the existing surrogate-pair test missed.

Closes #27753. Fixes #65782.

Why

Memory indexing stalls permanently for any agent whose content contains an emoji or supplementary-plane character that the chunker happens to split across a surrogate-pair boundary. openclaw memory status reports the index as clean / no issues while ~98% of sessions are unindexed.

End-to-end symptom (verbose):

[memory] embeddings: batch start  (×N)
Memory index failed (<agent>): openai embeddings failed: 500
{"error":{"message":"... 'utf-8' codec can't encode character '\\ud83c'
in position 3229: surrogates not allowed ..."}}

Two failure modes compound:

1. Chunker (packages/memory-host-sdk/src/host/internal.ts) — the inner fine-split loop already guards against splitting inside a surrogate pair, but the outer coarse loop advances by start += maxChars without checking the boundary. When the cut lands inside e.g. an emoji like 🌸 (U+1F338 = \uD83C\uDF38), the chunk ends with a lone high surrogate and the next begins with a lone low surrogate. The existing surrogate-pair test ("does not break surrogate pairs when splitting long CJK lines") does not catch this because it uses an all-surrogate-pair line — the high estimateStringChars weight forces the inner pass, which is correct.
2. Embedding HTTP path — Python JSON encoders (e.g. via LiteLLM proxies) reject lone surrogates with a 500 wrapping a UnicodeEncodeError. The retry policy only matches rate_limit | 429 | 5\d\d | cloudflare | tokens per day, but the body shape is a connect-error, not a transient one — and the indexer is transactional, so a single bad batch rolls back the entire run. Every subsequent openclaw memory index invocation reproduces the same failure.

What changed

packages/memory-host-sdk/src/host/internal.ts

- for (let start = 0; start < line.length; start += maxChars) {
-   const coarse = line.slice(start, start + maxChars);
+ for (let start = 0; start < line.length; ) {
+   let coarseEnd = Math.min(line.length, start + maxChars);
+   // Avoid splitting inside a UTF-16 surrogate pair (emoji, CJK Extension B+).
+   if (coarseEnd < line.length) {
+     const lastCode = line.charCodeAt(coarseEnd - 1);
+     if (lastCode >= 0xd800 && lastCode <= 0xdbff) {
+       coarseEnd += 1; // include the trailing low surrogate
+     }
+   }
+   const coarse = line.slice(start, coarseEnd);
+   start = coarseEnd;

Mirrors the existing inner-loop guard at the same file/line. New regression test ("does not break surrogate pairs at the coarse split boundary (issue #27753)") constructs a line of 39 ASCII + 🌸 + 39 ASCII so the outer cut at start + maxChars = 40 lands between the high and low surrogate. Without the fix, the test fails by producing chunks containing lone surrogates.

extensions/memory-core/src/memory/manager-embedding-ops.ts

Adds a tiny exported helper:

export function stripUnpairedSurrogates(text: string): string {
  return text.replace(
    /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g,
    "\uFFFD",
  );
}

…and applies it at the entry of embedBatchWithRetry and embedBatchInputsWithRetry. This is the change #27753 originally requested. It protects against any other code path that might produce a lone surrogate (LLM tool output, file ingestion, future chunkers) — chunker-only fix is necessary but not sufficient.

Adds manager-embedding-ops.test.ts with focused unit tests for the helper (preserves valid pairs, replaces lone high/low surrogates, handles reversed pairs, no-op on empty/non-string).

Test plan

• Verified live on installed runtime (openclaw 2026.4.5) by patching the equivalent bundled artifact under dist/manager-CKYnEo0k.js and running openclaw memory index --agent <id>:

  	before	after
  clawdea files	6	166
  clawdea chunks	93	4584
  dev files	2	17
  dev chunks	251	588

  openclaw memory search works end-to-end after the fix.

• CI runs pnpm test against the new unit tests in extensions/memory-core and packages/memory-host-sdk.

• Existing surrogate-pair test ("does not break surrogate pairs when splitting long CJK lines") still passes.

Notes for reviewers

• I did not run pnpm install locally to execute vitest — the runtime fix is verified live, and the new tests follow the same vitest patterns as adjacent files (manager-embedding-cache.test.ts, internal.test.ts).
• The retry-policy gap (fetch failed / UnicodeEncodeError 500 not retried) is a separate concern tracked in #56815 / #58255 — out of scope for this PR. With the chunker + sanitizer in place, the original failure no longer reaches the retry path.
• If the team prefers the helper to live in a shared utility module (e.g. under packages/memory-host-sdk) rather than manager-embedding-ops.ts, happy to relocate.

Changed files

• extensions/memory-core/src/memory/manager-embedding-ops.test.ts (added, +36/-0)
• extensions/memory-core/src/memory/manager-embedding-ops.ts (modified, +28/-4)
• packages/memory-host-sdk/src/host/internal.test.ts (modified, +27/-0)
• packages/memory-host-sdk/src/host/internal.ts (modified, +11/-2)