PR #21090: fix(core): handle ValueError and TypeError from structured output failures

• Repository: run-llama/llama_index
• Author: gautamvarmadatla
• State: closed | merged: True
• Link: https://github.com/run-llama/llama_index/pull/21090

Description (problem / solution / changelog)

Description

Expanded Refine’s structured-response fallback handlers to catch ValueError and TypeError in addition to ValidationError, so tool-calling structured-output failures are handled gracefully instead of bubbling up and failing the query.

Fixes #21089

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

• I added new unit tests to cover this change
• I believe this change is already covered by existing unit tests

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran uv run make format; uv run make lint to appease the lint gods

Changed files

• llama-index-core/llama_index/core/response_synthesizers/refine.py (modified, +8/-16)
• llama-index-core/tests/response_synthesizers/test_refine.py (modified, +47/-0)

PR #21138: fix(core): catch ValueError/TypeError in Refine structured output handling

• Repository: run-llama/llama_index
• Author: joaquinhuigomez
• State: closed | merged: False
• Link: https://github.com/run-llama/llama_index/pull/21138

Description (problem / solution / changelog)

Description

The Refine response synthesizer only catches ValidationError when structured output programs fail. After #21036 introduced explicit ValueError and TypeError raises in the function-calling structured output path, these uncaught exceptions cause queries to crash instead of falling through to the existing warning + fallback behavior.

This expands the exception handling in all four response methods (_give_response_single, _refine_response_single, _agive_response_single, _arefine_response_single) to catch ValueError and TypeError alongside ValidationError, matching the graceful degradation pattern already in place.

Related issues: #19926, #19044

Fixes #21089

Testing

Verified with the exact reproduction script from the issue:

from unittest.mock import patch
from pydantic import BaseModel, Field
from llama_index.core import SummaryIndex, Document
from llama_index.core.response_synthesizers import Refine
from llama_index.core.program.function_program import FunctionCallingProgram
from llama_index.llms.openai import OpenAI

class Answer(BaseModel):
    response: str = Field(description="Answer to the question")
    confidence: float = Field(description="Confidence score between 0 and 1")

docs = [Document(text="Paris is the capital of France.")]
index = SummaryIndex.from_documents(docs)
query_engine = index.as_query_engine(
    response_synthesizer=Refine(llm=OpenAI(api_key="sk-fake"), output_cls=Answer)
)

with patch.object(FunctionCallingProgram, "__call__", side_effect=ValueError("LLM did not return any tool calls")):
    response = query_engine.query("What is the capital of France?")

print(response)  # "Empty Response" instead of crash

Before the fix: ValueError: LLM did not return any tool calls propagates and crashes. After the fix: warning is logged and the query returns "Empty Response" gracefully.

Changed files

• llama-index-core/llama_index/core/response_synthesizers/refine.py (modified, +4/-4)

PR #21237: fix(core): catch ValueError and TypeError in Refine structured output handlers

• Repository: run-llama/llama_index
• Author: GopalGB
• State: closed | merged: False
• Link: https://github.com/run-llama/llama_index/pull/21237

Description (problem / solution / changelog)

Summary

• Broadens exception handling in Refine response synthesizer to catch ValueError and TypeError alongside ValidationError
• Prevents query failures when the function-calling structured-output path raises these exceptions (e.g., missing tool calls, non-BaseModel outputs)
• Applied to all four try/except blocks: _give_response_single, _refine_response_single, and their async counterparts

Root Cause

After changes in #21036, the function-calling structured-output path can raise:

• ValueError for missing/failed tool calls
• TypeError for non-BaseModel tool outputs

These were not caught by the except ValidationError clause, causing the query to fail instead of falling through to the existing warning/fallback path.

Test Plan

• Verified all four except blocks updated consistently
• No behavior change for ValidationError — still caught and logged as before
• ValueError and TypeError now trigger the same warning path instead of propagating

Fixes #21089

Changed files

• llama-index-core/llama_index/core/indices/document_summary/base.py (modified, +4/-2)
• llama-index-core/llama_index/core/response_synthesizers/refine.py (modified, +4/-4)
• llama-index-core/tests/indices/document_summary/test_index.py (modified, +22/-0)

PR #21285: Fix Refine to catch ValueError/TypeError from structured outputs

• Repository: run-llama/llama_index
• Author: akhilc08
• State: closed | merged: False
• Link: https://github.com/run-llama/llama_index/pull/21285

Description (problem / solution / changelog)

Fixes #21089

Description

When using output_cls or structured_answer_filtering with a function-calling LLM, the Refine response synthesizer only catches ValidationError from structured output programs. However, the function-calling path can also raise:

• ValueError when the LLM returns no tool calls
• TypeError when the tool output is not a BaseModel subclass

These are semantically equivalent to ValidationError (the structured output failed) and should follow the same warning-and-fallback path rather than crashing the query.

Changes

In refine.py, extend all four except ValidationError blocks to also catch ValueError and TypeError:

1. _give_response_single (sync) -- line ~240
2. _refine_response_single (sync) -- line ~300
3. _arefine_response_single (async) -- line ~408
4. _agive_response_single (async) -- line ~448

Each change is identical: except ValidationError as e: becomes except (ValidationError, ValueError, TypeError) as e:.

The log message is also generalized from "Validation error" to "Error" since it now covers more exception types.

Testing

from unittest.mock import patch
from pydantic import BaseModel, Field
from llama_index.core import SummaryIndex, Document
from llama_index.core.response_synthesizers import Refine
from llama_index.core.program.function_program import FunctionCallingProgram
from llama_index.llms.openai import OpenAI

class Answer(BaseModel):
    response: str = Field(description="Answer to the question")
    confidence: float = Field(description="Confidence score between 0 and 1")

docs = [Document(text="Paris is the capital of France.")]
index = SummaryIndex.from_documents(docs)
query_engine = index.as_query_engine(
    response_synthesizer=Refine(llm=OpenAI(api_key="sk-fake"), output_cls=Answer)
)

# Previously crashed with ValueError; now falls back to "Empty Response"
with patch.object(FunctionCallingProgram, "__call__", side_effect=ValueError("LLM did not return any tool calls")):
    response = query_engine.query("What is the capital of France?")
print(response)  # "Empty Response"

Changed files

• llama-index-core/llama_index/core/response_synthesizers/refine.py (modified, +8/-8)

PR #21288: fix: catch ValueError and TypeError in Refine structured output handlers

• Repository: run-llama/llama_index
• Author: akhilc08
• State: closed | merged: False
• Link: https://github.com/run-llama/llama_index/pull/21288

Description (problem / solution / changelog)

Fixes #21089

When using output_cls with a function-calling LLM, you can get a ValueError if the model returns no tool calls, or a TypeError if the output isn't a BaseModel subclass. These were crashing the query instead of falling back gracefully like ValidationError already does. Extended all five except blocks in refine.py (the sync and async versions of _give_response_single and _refine_response_single, plus the initial model_validate_json call) to catch all three exception types.

Changed files

• llama-index-core/llama_index/core/response_synthesizers/refine.py (modified, +9/-9)