claude-code - 💡(How to fix) Fix Frequent CJK character substitution errors during Japanese text generation in MCP tool calls (Notion content_updates etc.) [1 participants]

Summary

When using Claude (Opus 4.7 / Sonnet 4.6) to generate or edit Japanese text via MCP tools — particularly mcp__notion__notion-update-page with single-character old_str/new_str replacements — the model frequently substitutes incorrect Unicode code points. The wrong characters are typically Simplified Chinese or rare/archaic Japanese characters that look semantically related but are wrong, indicating an issue with how the model generates CJK code points one character at a time.

Frequency observed

In a single ~3-hour session editing one Notion page in Japanese, I observed 8+ such errors:

Reproduction pattern

Error rate is highest in three scenarios:

1. Single-character replacements in mcp__notion__notion-update-page content_updates arrays where old_str and new_str differ by one kanji
2. Generation of new Japanese paragraphs with mixed kanji content (especially when generating long markdown via notion-create-pages content arg)
3. Self-correction attempts — correcting one typo frequently introduces another (compounding errors)

Hypothesized cause

The model appears to mishandle Unicode escape sequences \uXXXX when generating CJK characters one at a time. The errors are NOT transcription errors from input — input Japanese is preserved correctly when read back via notion-fetch. The errors happen during generation of new content / replacements.

This suggests a tokenizer or JSON-encoding-pipeline issue specific to single-CJK-character spans, rather than a general Japanese understanding limitation.

Impact

• Documents intended for external sharing (clients, business partners) require manual character-by-character review before publication
• Self-correction workflows are unreliable — fixing one error introduces new ones
• Trust in MCP-based document workflows degrades quickly with this error rate
• For Japanese-language users, the workaround load is significant

Current workaround

After every MCP write, immediately notion-fetch the document, grep for known-bad characters (Simplified Chinese, rare CJK), and correct manually. This is brittle and high-toil.

Suggested investigation areas

1. Tokenizer behavior for CJK single-character spans (possibly subword splits introducing wrong neighbors)
2. JSON encoding pipeline between Claude → MCP client → MCP server (any Unicode normalization happening?)
3. Whether the same error rate occurs in non-MCP Edit tool calls on local files (initial impression: somewhat, but less frequent than MCP tool calls)
4. Whether including context in old_str/new_str (e.g., 5+ chars instead of 1) reduces error rate (anecdotally yes)

Environment

• Claude Code on macOS Darwin 25.2.0
• Model: Claude Opus 4.7 (1M context)
• MCP server: Anthropic-managed Notion connector (mcp__notion__*)
• Use case: Japanese business documents (TERIYAKI Inc.)

Related

This appears to be a recurring issue. Documenting it here so the pattern is centrally visible for the team's CJK quality work.