litellm - ✅(Solved) Fix [Bug]: Ollama chat transformation fails with JSONDecodeError 'Unterminated string' when parsing tool call arguments [1 pull requests, 1 participants]

litellm2026-04-17 23:33:38

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

BerriAI/litellm#25985•Fetched 2026-04-18 05:52:38

View on GitHub

Comments

Participants

Timeline

Reactions

Author

vcliment89

Participants

vcliment89

Timeline (top)

cross-referenced ×1labeled ×1referenced ×1

Error Message

json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 10 (char 9)

Root Cause

The code assumes typed_tool["function"]["arguments"] is always valid JSON. However, the upstream model may return:

Incomplete/truncated JSON (e.g., due to streaming issues or token limits)
Malformed JSON with unescaped characters
Partial tool call arguments

The current implementation does not handle these edge cases.

Fix Action

Fixed

Fixed by PR: fix(ollama): raise BadRequestError on malformed tool call JSON arguments (https://github.com/BerriAI/litellm/pull/25992)

PR fix notes

PR #25992: fix(ollama): raise BadRequestError on malformed tool call JSON arguments

Repository: BerriAI/litellm
Author: octo-patch
State: open | merged: False
Link: https://github.com/BerriAI/litellm/pull/25992

Description (problem / solution / changelog)

Fixes #25985

Problem

When an Ollama-compatible model returns truncated or otherwise invalid JSON in tool call arguments, transform_request propagated the raw json.JSONDecodeError upward. This surfaced to callers as an opaque APIConnectionError with no context about which arguments failed.

Solution

Wrap json.loads() in a try/except json.JSONDecodeError block:

Log the raw arguments string via verbose_logger.error for debugging.
Re-raise as litellm.BadRequestError with a message that includes the original parse error and the raw arguments string, giving callers an actionable error with full context.

Testing

Added two new unit tests in TestOllamaToolCalling:

test_transform_request_malformed_tool_call_arguments_raises_bad_request – verifies that truncated JSON arguments raise litellm.BadRequestError with a descriptive message.
test_transform_request_valid_tool_call_arguments_passes – verifies that valid JSON arguments continue to be parsed without error.

All 23 existing ollama chat transformation tests still pass.

Changed files

litellm/llms/ollama/chat/transformation.py (modified, +13/-1)
tests/test_litellm/llms/ollama/test_ollama_chat_transformation.py (modified, +78/-1)

Code Example

json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 10 (char 9)

---

arguments = json.loads(typed_tool["function"]["arguments"])

---

2026-04-17 18:28:21.669 | 23:28:21 - LiteLLM Proxy:ERROR: common_request_processing.py:1522 - litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.APIConnectionError: Unterminated string starting at: line 1 column 10 (char 9)
2026-04-17 18:28:21.669 | Traceback (most recent call last):
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/site-packages/litellm/main.py", line 4015, in completion
2026-04-17 18:28:21.669 |     response = base_llm_http_handler.completion(
2026-04-17 18:28:21.669 |         model=model,
2026-04-17 18:28:21.669 |     ...<14 lines>...
2026-04-17 18:28:21.669 |         client=client,
2026-04-17 18:28:21.669 |     )
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 410, in completion
2026-04-17 18:28:21.669 |     data = provider_config.transform_request(
2026-04-17 18:28:21.669 |         model=model,
2026-04-17 18:28:21.669 |     ...<3 lines>...
2026-04-17 18:28:21.669 |         headers=headers,
2026-04-17 18:28:21.669 |     )
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/site-packages/litellm/llms/ollama/chat/transformation.py", line 274, in transform_request
2026-04-17 18:28:21.669 |     arguments = json.loads(typed_tool["function"]["arguments"])
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/json/__init__.py", line 352, in loads
2026-04-17 18:28:21.669 |     return _default_decoder.decode(s)
2026-04-17 18:28:21.669 |            ~~~~~~~~~~~~~~~~~~~~~^^^
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/json/decoder.py", line 345, in decode
2026-04-17 18:28:21.669 |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2026-04-17 18:28:21.669 |                ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/json/decoder.py", line 361, in raw_decode
2026-04-17 18:28:21.669 |     obj, end = self.scan_once(s, idx)
2026-04-17 18:28:21.669 |                ~~~~~~~~~~~~~~^^^^^^^^
2026-04-17 18:28:21.669 | json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 10 (char 9)
2026-04-17 18:28:21.669 | . Received Model Group=glm-5
2026-04-17 18:28:21.669 | Available Model Group Fallbacks=None LiteLLM Retried: 3 times, LiteLLM Max Retries: 3

---

try:
    arguments = json.loads(typed_tool["function"]["arguments"])
except json.JSONDecodeError as e:
    # Log the malformed arguments for debugging
    verbose_logger.error(f"Failed to parse tool call arguments: {typed_tool['function']['arguments']!r}")
    raise litellm.InternalServerError(
        f"Model returned malformed tool call arguments: {e.msg}. Raw arguments: {typed_tool['function']['arguments']!r}"
    )

RAW_BUFFERClick to expand / collapse

Bug Description

When using LiteLLM proxy with an Ollama-compatible provider (custom model glm-5), the request transformation fails with a JSONDecodeError: Unterminated string error when parsing tool call arguments. The error occurs in litellm/llms/ollama/chat/transformation.py at line 274.

Steps to Reproduce

Configure LiteLLM proxy with an Ollama-compatible provider (e.g., custom local model server)
Send a chat completion request that includes tool/function definitions
The upstream model returns a tool call with arguments
LiteLLM attempts to transform the request and fails to parse the tool arguments JSON

Expected Behavior

LiteLLM should gracefully handle tool call arguments, either:

Successfully parse valid JSON arguments
Handle malformed/incomplete JSON with proper error handling (catch exception, log the actual arguments string for debugging, and fail with a more informative error message)

Actual Behavior

LiteLLM crashes with:

json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 10 (char 9)

The error is raised at litellm/llms/ollama/chat/transformation.py:274:

arguments = json.loads(typed_tool["function"]["arguments"])

Root Cause Analysis

The code assumes typed_tool["function"]["arguments"] is always valid JSON. However, the upstream model may return:

Incomplete/truncated JSON (e.g., due to streaming issues or token limits)
Malformed JSON with unescaped characters
Partial tool call arguments

The current implementation does not handle these edge cases.

Relevant Log Output

2026-04-17 18:28:21.669 | 23:28:21 - LiteLLM Proxy:ERROR: common_request_processing.py:1522 - litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.APIConnectionError: Unterminated string starting at: line 1 column 10 (char 9)
2026-04-17 18:28:21.669 | Traceback (most recent call last):
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/site-packages/litellm/main.py", line 4015, in completion
2026-04-17 18:28:21.669 |     response = base_llm_http_handler.completion(
2026-04-17 18:28:21.669 |         model=model,
2026-04-17 18:28:21.669 |     ...<14 lines>...
2026-04-17 18:28:21.669 |         client=client,
2026-04-17 18:28:21.669 |     )
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 410, in completion
2026-04-17 18:28:21.669 |     data = provider_config.transform_request(
2026-04-17 18:28:21.669 |         model=model,
2026-04-17 18:28:21.669 |     ...<3 lines>...
2026-04-17 18:28:21.669 |         headers=headers,
2026-04-17 18:28:21.669 |     )
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/site-packages/litellm/llms/ollama/chat/transformation.py", line 274, in transform_request
2026-04-17 18:28:21.669 |     arguments = json.loads(typed_tool["function"]["arguments"])
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/json/__init__.py", line 352, in loads
2026-04-17 18:28:21.669 |     return _default_decoder.decode(s)
2026-04-17 18:28:21.669 |            ~~~~~~~~~~~~~~~~~~~~~^^^
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/json/decoder.py", line 345, in decode
2026-04-17 18:28:21.669 |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2026-04-17 18:28:21.669 |                ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
2026-04-17 18:28:21.669 |   File "/usr/lib/python3.13/json/decoder.py", line 361, in raw_decode
2026-04-17 18:28:21.669 |     obj, end = self.scan_once(s, idx)
2026-04-17 18:28:21.669 |                ~~~~~~~~~~~~~~^^^^^^^^
2026-04-17 18:28:21.669 | json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 10 (char 9)
2026-04-17 18:28:21.669 | . Received Model Group=glm-5
2026-04-17 18:28:21.669 | Available Model Group Fallbacks=None LiteLLM Retried: 3 times, LiteLLM Max Retries: 3

Environment

LiteLLM Proxy running via Docker
Python 3.13
Model: glm-5 (custom local provider via Ollama-compatible endpoint)

Suggested Fix

Add try/except around the JSON parsing in transformation.py to:

Log the actual arguments string that failed to parse (for debugging)
Return a more informative error message including the raw arguments string
Optionally, attempt to repair common JSON issues (truncated strings, missing closing braces)

try:
    arguments = json.loads(typed_tool["function"]["arguments"])
except json.JSONDecodeError as e:
    # Log the malformed arguments for debugging
    verbose_logger.error(f"Failed to parse tool call arguments: {typed_tool['function']['arguments']!r}")
    raise litellm.InternalServerError(
        f"Model returned malformed tool call arguments: {e.msg}. Raw arguments: {typed_tool['function']['arguments']!r}"
    )

Related Issues

#13823 - Similar JSONDecodeError in Ollama completion transformation (response side)
#11267 - APIConnectionError parsing Tool call response from Ollama
#18920 - BadRequestError with Unterminated string (Anthropic)

extent analysis

TL;DR

To fix the JSONDecodeError: Unterminated string issue in LiteLLM proxy, add try/except handling around JSON parsing in transformation.py to catch and log malformed JSON arguments.

Guidance

Implement try/except block: Wrap the json.loads() call in a try/except block to catch json.JSONDecodeError exceptions.
Log malformed arguments: Log the actual arguments string that failed to parse for debugging purposes.
Return informative error message: Raise a litellm.InternalServerError with a message that includes the raw arguments string and the JSON decode error message.
Consider JSON repair attempts: Optionally, attempt to repair common JSON issues like truncated strings or missing closing braces before raising an error.

Example

The suggested fix provides a code snippet that demonstrates how to implement the try/except block and log the malformed arguments:

try:
    arguments = json.loads(typed_tool["function"]["arguments"])
except json.JSONDecodeError as e:
    # Log the malformed arguments for debugging
    verbose_logger.error(f"Failed to parse tool call arguments: {typed_tool['function']['arguments']!r}")
    raise litellm.InternalServerError(
        f"Model returned malformed tool call arguments: {e.msg}. Raw arguments: {typed_tool['function']['arguments']!r}"
    )

Notes

This fix assumes that the upstream model may return malformed or incomplete JSON arguments. By adding try/except handling, LiteLLM proxy can gracefully handle such cases and provide more informative error messages.

Recommendation

Apply the suggested workaround by adding try/except handling around JSON parsing in transformation.py. This will allow LiteLLM proxy to handle malformed JSON arguments and provide better error messages.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #API routing #API middleware #SSR setup #ISR setup

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.