claude-code - 💡(How to fix) Fix Multi-byte UTF-8 characters corrupted in Write tool and Bash tool output [2 comments, 2 participants]

Description

When using Claude Code with Japanese text (3-byte UTF-8 characters), characters are frequently corrupted to U+FFFD replacement characters. This has become noticeably more frequent in recent sessions.

Environment

• Claude Code: v2.1.117
• Model: Claude Opus 4.6 (1M context)
• OS: WSL2 (Linux 6.6.87.2-microsoft-standard-WSL2)
• Shell: fish

Reproduction

Two distinct corruption patterns observed:

Pattern 1: Write tool — mid-text corruption

When the Write tool outputs a long Japanese markdown file (~160 lines), random multi-byte characters in the middle of the text are corrupted.

Example:

• Expected: として正常ハンドリングする
• Actual: と���て正常ハンドリングする

The corrupted character ��� is U+FFFD, indicating an incomplete UTF-8 byte sequence. In a single Write call, 16 characters across 164 lines were corrupted.

Pattern 2: Bash tool — trailing character truncation

When passing Japanese text as a CLI argument (e.g., gh issue create --title "..."), the final multi-byte character is truncated.

Example:

• Expected: Phase 1 残りツール実装（11 ツール）
• Actual: Phase 1 残りツール実装（11 ツール���

The closing full-width parenthesis ） (3 bytes: EF BC 89) loses its final byte.

Likely Cause

The output streaming buffer appears to split at arbitrary byte boundaries rather than UTF-8 character boundaries. Since Japanese characters are 3 bytes each, a chunk boundary falling mid-character produces incomplete byte sequences that are replaced with U+FFFD.

This is consistent with:

• Longer texts having more corruptions (more chunk boundaries)
• Corruption occurring at seemingly random positions
• Only multi-byte characters being affected (ASCII is never corrupted)

Workaround

• Re-read the file after Write and manually fix corrupted characters
• For Bash --title arguments, append a trailing ASCII character (space) to avoid end-of-buffer truncation

Expected Behavior

All UTF-8 characters should be correctly preserved when passed to tool parameters, regardless of text length or character byte width.