openclaw - ✅(Solved) Fix [Bug]: chunkTextByBreakResolver final chunk has trailing whitespace [2 pull requests, 3 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
openclaw/openclaw#64036Fetched 2026-04-11 06:16:30
View on GitHub
Comments
3
Participants
2
Timeline
8
Reactions
0
Timeline (top)
commented ×3cross-referenced ×3labeled ×2

The chunkTextByBreakResolver function in src/shared/text-chunking.ts produces chunks with trailing whitespace in the final chunk (the one added after the loop exits). This is inconsistent with chunks produced inside the loop, which are properly trimmed.

Root Cause

Root Cause Analysis

Fix Action

Fixed

PR fix notes

PR #64054: Fix: trim trailing whitespace from final text-chunking chunk (Resolves #64036)

Description (problem / solution / changelog)

Fixes #64036.

Problem

chunkTextByBreakResolver in src/shared/text-chunking.ts was inconsistent: chunks emitted inside the loop were trimmed via trimEnd(), but the final chunk pushed after the loop preserved trailing whitespace.

chunkTextByBreakResolver("  ! ", 2, (w) => w.lastIndexOf(" ") || w.length);
// before: ["! "]   ← trailing space
// after:  ["!"]

Fix

Apply trimEnd() to the final remaining before pushing it, and skip the push entirely when the trimmed result is empty (matches the in-loop branch which already skips empty chunks).

Test plan

  • Added two regression tests covering the counterexamples from the issue (" ! " and "a b ").
  • pnpm test src/shared/text-chunking.test.ts — 6/6 pass.

Changed files

  • src/shared/text-chunking.test.ts (modified, +9/-0)
  • src/shared/text-chunking.ts (modified, +4/-1)

PR #64079: fix: trim trailing whitespace in final text chunk

Description (problem / solution / changelog)

Summary

Trim trailing whitespace from the final chunk emitted by chunkTextByBreakResolver, matching the existing behavior for chunks produced inside the main loop.

Changes

  • trim the final remaining chunk with trimEnd() before pushing it
  • skip pushing the final chunk when trimming leaves it empty
  • add regression coverage for final-chunk trailing whitespace cases

Testing

  • pnpm exec vitest run src/shared/text-chunking.test.ts
  • Note: repo-wide pre-commit/tsgo checks in this environment currently fail on unrelated missing optional dependencies/types outside this change

Fixes openclaw/openclaw#64036

Changed files

  • src/shared/text-chunking.test.ts (modified, +9/-0)
  • src/shared/text-chunking.ts (modified, +4/-1)

Code Example

// Counterexample from PBT
const text = "  ! ";
const limit = 2;
const result = chunkTextByBreakResolver(text, limit, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["! "]  <-- BUG: trailing space in final chunk
// Expected: ["!"]

---

while (remaining.length > limit) {
  // ... chunk extraction logic ...
  const rawChunk = remaining.slice(0, breakIdx);
  const chunk = rawChunk.trimEnd(); // Line 22: chunks ARE trimmed
  if (chunk.length > 0) {
    chunks.push(chunk);
  }
  // ... update remaining ...
}
if (remaining.length) {
  chunks.push(remaining); // Line 31: final chunk is NOT trimmed ← BUG
}

---

remaining = "  ! " (length 4 > 2, enter loop)
window = "  " (first 2 chars)
lastIndexOf(" ") in "  " = 1
breakIdx = 1 (valid)
rawChunk = " ".slice(0, 1) = " "
chunk = " ".trimEnd() = "" (empty, skipped!)
remaining = "  ! ".slice(2).trimStart() = "! "

---

remaining = "! " (length 2, NOT > 2, exit loop)

---

remaining.length > 0, so push remaining
chunks.push("! ")BUG: trailing space NOT trimmed!

---

// Case 1: Chunk inside loop - properly trimmed
chunkTextByBreakResolver("abc   def", 6, (w) => w.lastIndexOf(" "));
// Result: ["abc", "def"]  ✓ Both chunks trimmed

// Case 2: Final chunk after loop - NOT trimmed
chunkTextByBreakResolver("  ! ", 2, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["! "]  ✗ Trailing space preserved

// Case 3: Another example
chunkTextByBreakResolver("a b ", 2, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["a", "b "]  ✗ Trailing space preserved in final chunk

---

if (remaining.length) {
  const finalChunk = remaining.trimEnd();
  if (finalChunk.length > 0) {
    chunks.push(finalChunk);
  }
}

---

// Case 1: Final chunk after loop - NOT trimmed
chunkTextByBreakResolver("  ! ", 2, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["! "]  ✗ Trailing space preserved

// Case 2: Another example
chunkTextByBreakResolver("a b ", 2, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["a", "b "]  ✗ Trailing space preserved in final chunk

---
RAW_BUFFERClick to expand / collapse

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

Summary

The chunkTextByBreakResolver function in src/shared/text-chunking.ts produces chunks with trailing whitespace in the final chunk (the one added after the loop exits). This is inconsistent with chunks produced inside the loop, which are properly trimmed.

Discovery

This bug was discovered through property-based testing (PBT) using fast-check. The failing test case was:

// Counterexample from PBT
const text = "  ! ";
const limit = 2;
const result = chunkTextByBreakResolver(text, limit, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["! "]  <-- BUG: trailing space in final chunk
// Expected: ["!"]

Root Cause Analysis

The Bug Location

Looking at the implementation in src/shared/text-chunking.ts:

while (remaining.length > limit) {
  // ... chunk extraction logic ...
  const rawChunk = remaining.slice(0, breakIdx);
  const chunk = rawChunk.trimEnd(); // Line 22: chunks ARE trimmed
  if (chunk.length > 0) {
    chunks.push(chunk);
  }
  // ... update remaining ...
}
if (remaining.length) {
  chunks.push(remaining); // Line 31: final chunk is NOT trimmed ← BUG
}

The inconsistency:

  • Line 22: Chunks produced inside the loop are trimmed with trimEnd()
  • Line 31: The final remaining chunk is pushed without trimming

Step-by-Step Execution Trace

For input " ! " with limit 2:

Iteration 1:

remaining = "  ! " (length 4 > 2, enter loop)
window = "  " (first 2 chars)
lastIndexOf(" ") in "  " = 1
breakIdx = 1 (valid)
rawChunk = " ".slice(0, 1) = " "
chunk = " ".trimEnd() = "" (empty, skipped!)
remaining = "  ! ".slice(2).trimStart() = "! "

Iteration 2:

remaining = "! " (length 2, NOT > 2, exit loop)

After loop (line 30-32):

remaining.length > 0, so push remaining
chunks.push("! ")  ← BUG: trailing space NOT trimmed!

Result: ["! "]

Evidence of Inconsistency

// Case 1: Chunk inside loop - properly trimmed
chunkTextByBreakResolver("abc   def", 6, (w) => w.lastIndexOf(" "));
// Result: ["abc", "def"]  ✓ Both chunks trimmed

// Case 2: Final chunk after loop - NOT trimmed
chunkTextByBreakResolver("  ! ", 2, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["! "]  ✗ Trailing space preserved

// Case 3: Another example
chunkTextByBreakResolver("a b ", 2, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["a", "b "]  ✗ Trailing space preserved in final chunk

Suggested Fix

Apply trimEnd() to the final chunk before pushing:

if (remaining.length) {
  const finalChunk = remaining.trimEnd();
  if (finalChunk.length > 0) {
    chunks.push(finalChunk);
  }
}

Note: We also need to check finalChunk.length > 0 to avoid pushing empty strings (e.g., if remaining is all whitespace).

Steps to reproduce

// Case 1: Final chunk after loop - NOT trimmed
chunkTextByBreakResolver("  ! ", 2, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["! "]  ✗ Trailing space preserved

// Case 2: Another example
chunkTextByBreakResolver("a b ", 2, (w) => w.lastIndexOf(" ") || w.length);
// Result: ["a", "b "]  ✗ Trailing space preserved in final chunk

Expected behavior

All chunks should have no trailing whitespace, consistent with the behavior for chunks produced inside the loop.

Actual behavior

Trailing space preserved in final chunk

OpenClaw version

commit 2645ed154b425ead760484a97bf7520f3e16e457

Operating system

N.E.

Install method

No response

Model

N.E.

Provider / routing chain

N.E.

Additional provider/model setup details

No response

Logs, screenshots, and evidence

Impact and severity

No response

Additional information

No response

extent analysis

TL;DR

Apply trimEnd() to the final chunk before pushing it to the chunks array to ensure consistency in trimming whitespace.

Guidance

  • Identify the location of the bug in the chunkTextByBreakResolver function, specifically the inconsistency between trimming chunks inside the loop and not trimming the final chunk.
  • Apply the suggested fix by modifying the code to trim the final chunk using trimEnd() before pushing it to the chunks array.
  • Verify the fix by running the provided test cases and checking that the resulting chunks no longer have trailing whitespace.
  • Consider adding additional test cases to ensure the fix does not introduce any new bugs.

Example

if (remaining.length) {
  const finalChunk = remaining.trimEnd();
  if (finalChunk.length > 0) {
    chunks.push(finalChunk);
  }
}

Notes

The provided fix assumes that the intention is to trim trailing whitespace from all chunks, including the final chunk. If this is not the intended behavior, further clarification may be needed.

Recommendation

Apply the suggested fix to ensure consistent trimming of whitespace from all chunks. This fix is a simple and targeted change that addresses the identified bug without introducing any new complexity.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

All chunks should have no trailing whitespace, consistent with the behavior for chunks produced inside the loop.

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

openclaw - ✅(Solved) Fix [Bug]: chunkTextByBreakResolver final chunk has trailing whitespace [2 pull requests, 3 comments, 2 participants]