openclaw - 💡(How to fix) Fix memory-core: gateway pins deleted SQLite inode after out-of-process reindex → memory_search stuck on "index metadata is missing"

Summary

A running gateway's cached MemoryIndexManager opens each per-agent memory-core SQLite once at construction and never reopens it after an out-of-process inode swap of the DB file. OpenClaw's own CLI (openclaw memory index, openclaw memory status --index) performs exactly such a swap (backup-rename → fresh-create → delete-backup, swapMemoryIndexFiles). After the swap the live gateway is pinned to the old, now-deleted inode whose meta table is empty, so:

• readMeta() returns null
• resolveCurrentIndexIdentityState returns { status: "missing", reason: "index metadata is missing" }
• the memory_search tool is paused for that agent until the gateway is restarted.

The kicker: the error text recommends running the very command that re-breaks it —

Tell the user to run: openclaw memory status --index or openclaw memory index --force.

Running that under a live gateway re-orphans the handle every time.

Version observed: OpenClaw 2026.6.1 (2e08f0f), Linux, builtin memory backend.

Reproduction

1. Gateway up; memory_search healthy for some agent (e.g. coli).
2. Run openclaw memory status --index (or memory index --force) for that agent while the gateway is live.
3. memory_search for that agent immediately returns disabled:true, error:"index metadata is missing".
4. openclaw memory status --deep (a separate CLI process) reports the store as perfectly healthy (indexIdentity: valid) because it opens the live inode. The disagreement between the in-gateway tool and the CLI is the diagnostic tell.

Evidence captured (reference host)

• /proc/<gateway-pid>/fd held deleted inodes: …/memory/coli.sqlite.backup-<uuid> (deleted) plus its -wal/-shm, and the same for a second agent. A third agent that had not been CLI-reindexed had no deleted handle and its search worked — that asymmetry pinpointed the cause.
• Copying the deleted inode read-only (cp /proc/<pid>/fd/<n> …) and inspecting it: chunks=1877, meta_rows=0. The live file: chunks=1961, meta_rows=1 with a valid memory_index_meta_v1 row. So the gateway is serving a pre-final-swap snapshot whose meta was never written.
• After a CLI memory index under a live gateway, the on-disk file mtime advanced but the gateway's deleted-inode fds were unchanged → confirms the in-process manager never reopened.

There is also a related meta-loss self-perpetuating variant: if an in-process sync runs while identity is already != "valid" and chunks are present, runSync takes the !needsFullReindex branch (needsExplicitIdentityReindex requires reason === "cli"), sets dirty = true and returns without writing meta — so the live store can be left at meta_rows = 0 and the gateway can never self-heal; only a CLI --force reindex (which writes meta) recovers it.

Root cause (code anchors, dist/manager-dZw31DAG2.js @ 2026.6.1)

• openMemoryDatabaseAtPath(dbPath, …) (~L493): new DatabaseSync(dbPath) binds a handle to the current inode; no dev/ino recorded.
• Manager ctor (~L3448): this.db = this.openDatabase(); — opened once.
• openDatabase() (~L1242): re-resolves the path but is only called from the ctor and the readonly-recovery path.
• Existing reopen precedent (~L3284): on a readonly-DB error the manager already does the right dance — closeDatabase(db); db = openDatabase(); resetVectorState(); ensureSchema(); meta = readMeta();. The fix generalizes this trigger from "readonly error" to "inode changed".
• resolveCurrentIndexIdentityState (~L1135) / refreshIndexIdentityDirty (~L3564) / status() are where a cheap stat-check hooks in before identity is judged.
• Swap writer: swapMemoryIndexFiles / moveMemoryIndexFiles / removeMemoryIndexFiles (manager-atomic-reindex region, ~L754).
• CLI reindex runs in its own process (reason:"cli" at ~L411/441/717 of dist/cli.runtime-*.js) and does no gateway-liveness check (grep isGatewayRunning|pidfile|lockfile|EBUSY → none).

Proposed fix — two parts, ship together

Part A — inode-aware reopen (the cure)

1. Record identity on open. In openDatabase() (or right after the ctor binds this.db), fs.statSync(resolveUserPath(store.path)) and stash this._dbInode = { dev, ino }.
2. Re-check before serving identity/search. At the top of refreshIndexIdentityDirty() (and/or status() / supplement search()), call maybeReopenIfInodeChanged():

maybeReopenIfInodeChanged() {
  let cur;
  try { cur = fs.statSync(resolveUserPath(this.settings.store.path)); }
  catch { return; } // file missing mid-swap: let the next call retry
  if (this._dbInode &&
      (cur.dev !== this._dbInode.dev || cur.ino !== this._dbInode.ino)) {
    log.warn("memory: db inode changed under open handle; reopening", {
      agentId: this.agentId, oldIno: this._dbInode.ino, newIno: cur.ino,
    });
    try { this.closeDatabase(this.db); } catch {}
    this.db = this.openDatabase();   // re-stats + re-stashes inode
    this.resetVectorState();
    this.ensureSchema();
    this._dbInode = { dev: cur.dev, ino: cur.ino };
  }
}

This reuses the exact sequence the readonly-recovery path already trusts, so the blast radius is small. Cost is one statSync per status/search call — single-digit microseconds against a millisecond recall budget. If even that is unwanted on the hottest path, gate it behind the existing "missing/mismatched" identity branch (only re-stat when identity looks wrong); that alone fixes the reported bug, at the cost of one stat on the unhappy path only.

Part B — CLI reindex guard (the prevention)

In the memory index / memory status --index CLI action, before swapping: detect a live gateway that owns the store (prefer the gateway's own pidfile/control socket over /proc scraping, for cross-platform). If live:

• Default: refuse with a clear message ("A gateway is running; an out-of-process reindex would desync its in-memory index. The live store is usually healthy and does not need a reindex. Re-run with --force and restart the gateway afterward, or run the reindex through the gateway."). Exit non-zero.
• Better follow-up: route the reindex through the gateway via RPC so the in-process manager performs the swap and reopens its own handle (no desync possible). Part A still protects against other out-of-process mutators (litestream restore, DR drills, manual file replace).

Either part alone leaves a gap; together they close the class.

Suggested tests

• Part A: open a manager over a temp DB, capture identity; rename() a different valid DB into place (simulate swap); assert the next status() reopens (new inode) and returns valid without a restart.
• Part A regression: swap in a DB whose meta is empty, then swap back to a valid one; assert no permanent "missing" latch.
• Part B: with a faked "gateway live" signal, assert memory index exits non-zero and does NOT swap; with --force, asserts it proceeds.

Workaround until fixed

• Don't run memory index / status --index against a store a live gateway owns. The stores are usually healthy and don't need a reindex.
• If a reindex is genuinely needed (e.g. the meta-loss variant above), run openclaw memory index --force --agent <id> then restart the gateway (reindex first so the restart reopens the freshly-written inode).

Happy to open a PR for Part A (the reopen) if that's welcome — it's a contained change reusing the existing recovery dance.