hermes - 💡(How to fix) Fix Telegram message splitting can slice UTF-16 surrogate pairs, causing delivery failures [1 comments, 1 participants]

On Telegram, long messages that contain emojis outside the Basic Multilingual Plane (e.g. 🎩 ) can fail to deliver with the error: The underlying exception is:

TL;DR

Adjust the message splitting logic to ensure that the split point respects UTF-16 boundaries, especially for emojis outside the Basic Multilingual Plane.

Guidance

• Review the code that splits long messages to identify where the backward scan for natural break points is performed and adjust it to re-validate UTF-16 boundaries.
• Consider using a library or function that handles Unicode characters and surrogate pairs correctly to find safe split points.
• Test the adjusted logic with messages containing emojis like 🎩 to verify that the issue is resolved.
• Ensure that the fix does not introduce any performance issues or affect the delivery of messages without emojis.

Example

No specific code example can be provided without the exact codebase, but the adjustment should involve using a function similar to String.prototype.codePointAt() or equivalent logic to clamp the split point to a valid UTF-16 boundary.

Notes

The fix might require additional handling for edge cases, such as messages that are exactly at the 4096 UTF-16 code unit limit or contain sequences of emojis that could still cause issues after the initial fix.

Recommendation

Apply the workaround by adjusting the message splitting logic to respect UTF-16 boundaries, as this directly addresses the root cause of the issue and should prevent message delivery failures on Telegram due to emojis.