openclaw - 💡(How to fix) Fix BM25 full-text search returns 0 results for Chinese queries with multiple terms

Problem

When searching memory with Chinese queries containing multiple terms, memory_search always returns textScore: 0 for all results. Vector search works fine, but BM25/FTS5 keyword search never matches.

Root Cause

The buildFtsQuery function in extensions/memory-core/src/memory/hybrid.ts joins all query tokens with AND:

function buildFtsQuery(raw) {
  const tokens = raw.match(/[\p{L}\p{N}_]+/gu)?.map((t) => t.trim()).filter(Boolean) ?? [];
  if (tokens.length === 0) return null;
  return tokens.map((t) => `"${t.replaceAll("\"", "")}"`).join(" AND ");
}

For Chinese text, the regex /[\p{L}\p{N}_]+/gu treats consecutive CJK characters as a single token. So a query like 微信读书 书架 阅读偏好 becomes:

"微信读书" AND "书架" AND "阅读偏好"

This requires all three terms to appear in the same FTS5 chunk. In practice, Chinese text rarely has all keywords in one chunk, so the result is always 0 matches.

Evidence

Direct SQLite FTS5 queries confirm the index is healthy:

Same pattern for other queries:

• "基金" AND "023521" AND "博时" AND "持仓" → 0 results
• "基金" OR "023521" OR "博时" OR "持仓" → 48 results

Environment

• OpenClaw v2026.5.12
• FTS5 tokenizer: unicode61 (default)
• Embedding model: bge-m3 (1024 dim)
• Chunks indexed: 2692
• All chunks have text content (verified)

Suggested Fix

Change the FTS query builder to use OR instead of AND for joining tokens, or implement a hybrid approach:

// Option 1: Simple OR
return tokens.map((t) => `"${t.replaceAll("\"", "")}"`).join(" OR ");

// Option 2: Hybrid - first 2 terms AND, rest OR
// Or use BM25 ranking with OR logic (OR is standard for search engines)

Note: This is a regression for CJK users. The AND logic works for English where tokens are naturally word-separated, but fails for Chinese where the tokenizer groups consecutive CJK into single tokens.

Impact

Every Chinese memory search query with 2+ terms produces textScore: 0, effectively disabling BM25 keyword search for CJK users. This affects all Chinese, Japanese, and Korean users who search with multi-word queries.