openclaw - 💡(How to fix) Fix ProcessStatements: Coordinate step merges shard resulting in oversized shard timeout [1 participants]

What's happening: The ProcessStatements step function failed with a Max poll count exceeded error. Investigation shows the process-statement Lambda silently timed out/died on the 3rd shard of a 199,988 row statement.

What should happen: Shards should not exceed the maximum row limit (50,000). The coordinator should not merge a near-max shard into an existing max-sized shard.

Evidence:

• Upload ID: 3fefc6838babf39db9c95b7f895cb663eb54de816d3587a04ebbbdc1bd0de1fd
• Statement ID: 1d90ec998320c1affc01fe4883e5bdbc297c862b6b9fe80f9346dedbda4ee371
• In coordinate-statement-processing, Shard 4 (49,989 rows) was deemed "too small": "Last shard too small, merging with previous" | lastShardRows: 49989 | secondLastShardRows: 50000
• This created a merged Shard 3 with nearly 100,000 rows.
• process-statement processing Shard 3 logged output up to 90,000 rows at 09:14:46 UTC, then silently stopped (likely OOM or timeout).
• poll-shard-status polled for 45 minutes until throwing Max poll count exceeded.

Affected: Statements exactly around the size boundary (e.g. 199,988 rows). This affects Avex rightsholder processing in Prod.

Suggested fix: Adjust the shard merging logic in coordinate-statement-processing. A shard of 49k rows is not "too small" and should not be merged into an already-full 50k shard. The threshold for merging should be much lower (e.g., < 1,000 rows).

Priority: High

• Functional bug causing statement processing pipeline failures for large files.