Summary

Gateway suffers severe memory leak (2.7GB in 4 minutes after restart) caused by the qmd embed boot process spawning parallel embed runs for all agents, each loading a full GGUF embedding model into memory.

Environment

• OpenClaw: 2026.4.11 → 2026.4.23
• OS: WSL2 Linux (x64), no GPU
• Agents: 6 (main-controller, code-specialist, data-processor, investment-analyst, kb-researcher, office-assistant)
• Embedding model: hf_ggml-org_embeddinggemma-300M-Q8_0.gguf (314MB per copy)
• Total memory: 16GB system

Problem

1. No skip when embeddings are already up-to-date

When all content hashes already have embeddings (confirmed by running qmd embed manually — completes in 0.66s), the Gateway boot embed still:

• Loads the 314MB GGUF model
• Initializes the embedding pipeline
• Only then discovers nothing needs to be done

This is a waste of both time and memory on every boot.

2. All agents embed in parallel

The Gateway starts embed for all 6 agents simultaneously at boot. Each one:

• Loads a separate 314MB GGUF model into memory (1.88GB total for models alone)
• Competes for CPU (no GPU acceleration)
• Often times out (120s default) because all 6 are fighting for resources

The timeouts trigger a retry loop: timeout → backoff 60s → retry → timeout → ... This loop leaks ~100-200MB per cycle in the Gateway process (RSS grows from 1.3GB to 2.7GB in ~15 minutes and keeps climbing).

3. Duplicate GGUF model files

Each agent stores its own copy of the GGUF model at ~/.openclaw/agents/<agent>/qmd/xdg-cache/qmd/models/hf_ggml-org_embeddinggemma-300M-Q8_0.gguf. For 6 agents, that is 6 × 314MB = 1.88GB of duplicate files.

Evidence

# Gateway process 4 minutes after boot
$ ps aux | grep openclaw-gateway
RSS: 2711 MB, CPU: 103%

# qmd embed log — repeats every ~3 minutes
[memory] qmd embed failed (boot): Error: qmd embed timed out after 120000ms; backing off for 60s

# Manual embed (standalone, no parallel) — instant
$ time qmd embed --workspace ~/.openclaw/agents/main-controller
✓ All content hashes already have embeddings.
real    0m0.664s

Suggested Fixes

1. Pre-check before loading model: Before spawning the embed process, check if any content hashes need embedding. If all are current, skip entirely (return immediately like the manual run does).

2. Serialize agent embeds: Process agents sequentially (or with limited concurrency, e.g., 1-2 at a time) instead of launching all 6 in parallel. This reduces peak memory from ~1.88GB to ~314MB.

3. Share the GGUF model file: Store one copy of the model in a shared location and symlink or reference it from each agent's qmd cache. (We worked around this manually with symlinks to ~/.openclaw/shared/qmd-models/.)

4. Fix memory leak on timeout: When embed times out and the child process is killed, ensure all allocated memory (buffers, model weights) is fully released back to the OS, not retained in the Gateway process.

Workaround (current)

// openclaw.json
{
  "memory": {
    "qmd": {
      "update": {
        "onBoot": false,
        "embedInterval": "4h"
      }
    }
  }
}

This disables boot-time embed and runs it every 4 hours instead. Memory stays stable at ~900MB instead of growing to 2.7GB+.

Related Files

• dist/qmd-manager-*.js — runUpdate(), shouldRunEmbed(), withQmdEmbedLock()
• dist/dreaming-B8_cmGiF.js — DEFAULT_MEMORY_DREAMING_FREQUENCY