vllm - ✅(Solved) Fix [Bug]: Gemma 4 tool usage produces invalid json due to bug in streaming [1 pull requests, 2 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#38946Fetched 2026-04-08 02:44:47
View on GitHub
Comments
2
Participants
2
Timeline
7
Reactions
3
Author
Participants
Timeline (top)
commented ×2subscribed ×2closed ×1cross-referenced ×1

Currently, the parsing logic while streaming does not consider the string delimiter. This leads to incorrect strings and somtimes even invalid json during tool usage in applications such as OpenCode.

Error Message

invalid [tool=todowrite, error=Invalid input for tool todowrite: JSON parsing failed: Text: {"todos": [{"content": "<|\Demonstrate todowrite tool usage<|"|", "priority": "<|\medium<|"|", "status": "<|"in_progress<|"}]}. Error message: JSON Parse error: Invalid escape character D]

Root Cause

Currently, the parsing logic while streaming does not consider the string delimiter. This leads to incorrect strings and somtimes even invalid json during tool usage in applications such as OpenCode.

PR fix notes

PR #38992: [Bugfix] Fix invalid JSON in Gemma 4 streaming tool calls by stripping partial delimiters

Description (problem / solution / changelog)

Issue #38946

Fix Gemma 4 streaming tool calls producing invalid JSON due to partial delimiter chars not being stripped

cc @sallyom

Changed files

  • tests/tool_parsers/test_gemma4_tool_parser.py (modified, +29/-0)
  • vllm/tool_parsers/gemma4_tool_parser.py (modified, +4/-3)
RAW_BUFFERClick to expand / collapse

Summary

Currently, the parsing logic while streaming does not consider the string delimiter. This leads to incorrect strings and somtimes even invalid json during tool usage in applications such as OpenCode.

🐛 Describe the bug

When using tools like OpenCode connected to the nvidia/Gemma-4-31B-IT-NVFP4 model hosted with vllm on an H100, the model is mostly unable to use tools due to a bug in the parsing / streaming logic.

Test Result

Before fix:

try to write a simple todo using the tool

invalid [tool=todowrite, error=Invalid input for tool todowrite: JSON parsing failed: Text: {"todos": [{"content": "<|\Demonstrate todowrite tool usage<|"|", "priority": "<|\medium<|"|", "status": "<|"in_progress<|"}]}. Error message: JSON Parse error: Invalid escape character D]

After fix:

try to write a simple todo using the tool

Todos
[•] Demonstrate todowrite tool usage

Pull request

[Bugfix] Gemma 4: Fix bug around invalid JSON diffs during tool usage

I believe this is also related to #38910.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The parsing logic needs to be updated to correctly handle string delimiters while streaming to prevent invalid JSON and incorrect strings.

Guidance

  • Review the parsing logic to ensure it properly accounts for string delimiters, especially when dealing with special characters or escape sequences.
  • Verify the fix by testing the tool usage with various input scenarios, including those that previously resulted in JSON parsing errors.
  • Check the pull request for the specific code changes made to address this issue, as it may provide insight into the necessary updates.
  • Consider testing the tool with different models and inputs to ensure the fix is robust and doesn't introduce new issues.

Example

No specific code snippet is provided due to the lack of detailed code information in the issue.

Notes

The provided information suggests that the issue is related to the parsing logic and string delimiter handling. However, without more specific code details, it's difficult to provide a precise fix. The mentioned pull request may offer more insight into the necessary changes.

Recommendation

Apply the workaround or fix provided in the pull request, as it directly addresses the parsing logic issue and has been tested to resolve the JSON parsing errors.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING