crewai - ✅(Solved) Fix [BUG] when using Task(..., output_pydantic=MyModel) some JSON substring identification within the data occurs with possible false positive [2 pull requests, 3 comments, 3 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
crewAIInc/crewAI#5460Fetched 2026-04-16 06:44:46
View on GitHub
Comments
3
Participants
3
Timeline
9
Reactions
0
Author
Timeline (top)
commented ×3cross-referenced ×2renamed ×2labeled ×1

Specifically the input is a graphql schema, so it has curly braces like JSON but it is not json.

in crewai.utilities.converter: _JSON_PATTERN: Final[re.Pattern[str]] = re.compile(r"({.*})", re.DOTALL)

tries to identify json in a larger string by just looking for curly braces. this makes any look-alike JSON like GraphQL schemas to be matches as false-positives. making the conversion fail

Error Message

Traceback (most recent call last): File "/Users/kalfa/Source/graphql-test/graphql/src/graphql/main.py", line 24, in run Graphql().crew().kickoff(inputs=inputs) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/crew.py", line 743, in kickoff result = self._run_sequential_process() File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/crew.py", line 1150, in _run_sequential_process return self._execute_tasks(self.tasks) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/crew.py", line 1236, in _execute_tasks task_output = task.execute_sync( agent=exec_data.agent, context=context, tools=exec_data.tools, ) File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 499, in execute_sync return self._execute_core(agent, context, tools) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 740, in _execute_core raise e # Re-raise the exception after emitting the event ^^^^^^^ File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 678, in _execute_core pydantic_output, json_output = self._export_output(result) ~~~~~~~~~~~~~~~~~~~^^^^^^^^ File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 968, in _export_output model_output = convert_to_model( result, ...<3 lines>... self.converter_cls, ) File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/utilities/converter.py", line 194, in convert_to_model return handle_partial_json( result=result, ...<3 lines>... converter_cls=converter_cls, ) File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/utilities/converter.py", line 261, in handle_partial_json exported_result = model.model_validate_json(match.group()) File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/pydantic/main.py", line 746, in model_validate_json return cls.pydantic_validator.validate_json( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ json_data, strict=strict, context=context, by_alias=by_alias, by_name=by_name ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ pydantic_core._pydantic_core.ValidationError: 1 validation error for GraphQLSchema Invalid JSON: key must be a string at line 2 column 3 [type=json_invalid, input_value='{\n code: ID!\n countr...ing\n name: String!\n}', input_type=str] For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Root Cause

Specifically the input is a graphql schema, so it has curly braces like JSON but it is not json.

in crewai.utilities.converter: _JSON_PATTERN: Final[re.Pattern[str]] = re.compile(r"({.*})", re.DOTALL)

tries to identify json in a larger string by just looking for curly braces. this makes any look-alike JSON like GraphQL schemas to be matches as false-positives. making the conversion fail

Fix Action

Fixed

PR fix notes

PR #5461: fix(converter): fall back on invalid JSON-like partial matches

Description (problem / solution / changelog)

Summary

Fixes issue #5460.

output_pydantic could fail when the model output contained brace-delimited text that looked like JSON but was actually plain string content, such as a GraphQL schema. In that case, partial JSON extraction matched the {...} block, Pydantic validation failed, and the error was raised instead of falling back to the normal conversion path.

This PR fixes that fallback behavior and adds regression coverage for the reported case.

Changes

  • treat invalid JSON-like partial matches as a fallback case in handle_partial_json(...)
  • add a regression test for GraphQL schema text containing braces

Rationale

The existing partial JSON logic is meant to recover valid JSON embedded in a larger response. The bug was that a false-positive match was handled as fatal, even though it should have been treated the same way as any other unsuccessful partial extraction attempt.

With this change, valid partial JSON still parses as before, while non-JSON brace content now falls through to the existing instruction-based converter path.

Testing

  • ./.venv/bin/python -m pytest lib/crewai/tests/utilities/test_converter.py -k "invalid_json_like_partial or partial_json" -q

Notes

  • uv.lock is intentionally not included because it is unrelated to this fix
  • running the full test_converter.py file in this environment still hits an unrelated pre-existing LiteLLM dependency issue

Changed files

  • lib/crewai/src/crewai/utilities/converter.py (modified, +1/-1)
  • lib/crewai/tests/utilities/test_converter.py (modified, +21/-0)

PR #5487: fix: validate JSON before Pydantic conversion to avoid false-positive matches on GraphQL schemas

Description (problem / solution / changelog)

Summary

Fixes #5460

_JSON_PATTERN matches any {.*} substring, including non-JSON content like GraphQL schemas. When such content is passed to model_validate_json, Pydantic raises a ValidationError that surfaces as a conversion failure rather than a graceful fallback.

Root cause: The regex matches curly-brace content but does not verify it is valid JSON before attempting Pydantic validation.

Fix: Call json.loads() on the regex match first. If it raises JSONDecodeError, the content is not JSON and we fall through to convert_with_instructions instead of attempting Pydantic validation.

Changes

  • src/crewai/utilities/converter.py: Added json.loads() guard before model_validate_json in handle_partial_json
  • tests/utilities/test_converter.py: Added test_handle_partial_json_with_graphql_schema_does_not_raise to cover the false-positive case

Test plan

  • Existing converter tests pass
  • New test test_handle_partial_json_with_graphql_schema_does_not_raise passes
  • Task with output_pydantic and a GraphQL schema as agent output no longer raises ValidationError

Changed files

  • lib/crewai/src/crewai/utilities/converter.py (modified, +15/-0)
  • lib/crewai/tests/utilities/test_converter.py (modified, +21/-0)

Code Example

# [... import ...]

class GraphQLSchema(BaseModel):
    gql_schema: str = Field(description="The schema of the graphql server")


[...crew class modified from template]
    @task
    def determine_schema(self) -> Task:
        return Task(
            config=self.tasks_config["determine_schema"],  # type: ignore[index]
            output_pydantic=GraphQLSchema,
        )

---

determine_schema:
  description: >
    Query the graphql server to get the schema.
    Use the graphql mcp tool to get the schema.
    Return any error from the server, if the query results in error.
    No other outcome is expected and allowed.
  markdown: true
  expected_output: >
    The schema of the GraphQL server, as a string.

  agent: query_manager

---

type Query {
  countries: [Country]
  country(code: String!): Country
}

type Country {
  code: String
  name: String
  native: String
  phone: String
  continent: Continent
  currency: String
  languages: [Language]
  emoji: String
}

type Continent {
  code: String
  name: String
}

type Language {
  code: String
  name: String
}

---

Traceback (most recent call last):
  File "/Users/kalfa/Source/graphql-test/graphql/src/graphql/main.py", line 24, in run
    Graphql().crew().kickoff(inputs=inputs)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/crew.py", line 743, in kickoff
    result = self._run_sequential_process()
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/crew.py", line 1150, in _run_sequential_process
    return self._execute_tasks(self.tasks)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/crew.py", line 1236, in _execute_tasks
    task_output = task.execute_sync(
        agent=exec_data.agent,
        context=context,
        tools=exec_data.tools,
    )
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 499, in execute_sync
    return self._execute_core(agent, context, tools)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 740, in _execute_core
    raise e  # Re-raise the exception after emitting the event
    ^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 678, in _execute_core
    pydantic_output, json_output = self._export_output(result)
                                   ~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 968, in _export_output
    model_output = convert_to_model(
        result,
    ...<3 lines>...
        self.converter_cls,
    )
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/utilities/converter.py", line 194, in convert_to_model
    return handle_partial_json(
        result=result,
    ...<3 lines>...
        converter_cls=converter_cls,
    )
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/utilities/converter.py", line 261, in handle_partial_json
    exported_result = model.model_validate_json(match.group())
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/pydantic/main.py", line 746, in model_validate_json
    return cls.__pydantic_validator__.validate_json(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        json_data, strict=strict, context=context, by_alias=by_alias, by_name=by_name
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
pydantic_core._pydantic_core.ValidationError: 1 validation error for GraphQLSchema
  Invalid JSON: key must be a string at line 2 column 3 [type=json_invalid, input_value='{\n  code: ID!\n  countr...ing\n  name: String!\n}', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

---

# Logic:
# 1. Objects: Must be empty {} or start with a double-quoted key {"key":
# 2. Arrays:  Must be empty [] or start with a JSON value: ", digit, -, true, false, null, [, or {
# 3. Strings: Standard JSON double-quoted strings
# 4. Primitives: true, false, null, and numbers.
json_pattern = r'(\{\s*\}|\{\s*"[^"\\]*"\s*:.*?[^"\\]\}|\[\s*\]|\[\s*(?:"|[\d\-]|true|false|null|[\[{]).*?[^"\\]\]|"(?:\\.|[^"\\])*"|\b(?:true|false|null)\b|-?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?)'
RAW_BUFFERClick to expand / collapse

Description

Specifically the input is a graphql schema, so it has curly braces like JSON but it is not json.

in crewai.utilities.converter: _JSON_PATTERN: Final[re.Pattern[str]] = re.compile(r"({.*})", re.DOTALL)

tries to identify json in a larger string by just looking for curly braces. this makes any look-alike JSON like GraphQL schemas to be matches as false-positives. making the conversion fail

Steps to Reproduce

  1. have an agent that returns any lookalike-JSON string like `pre {boo} post'
  2. have a task that uses output_pydantic=MyModel so that
  3. look the task fail

Expected behavior

the aim is to narrow down as much as possible the false-positives, but without doing a slow json.load. the most effective way to understand if the identifies substring is json, is to try to load the substring, to determine it.

if json.load is acceptable, it will deterministically and accurately tell if it's JSON.

the ultimate aim is to allow a generic look-alike-JSON string to be treated as string, and not serialised, while JSON to be serialised.

Screenshots/Code snippets

# [... import ...]

class GraphQLSchema(BaseModel):
    gql_schema: str = Field(description="The schema of the graphql server")


[...crew class modified from template]
    @task
    def determine_schema(self) -> Task:
        return Task(
            config=self.tasks_config["determine_schema"],  # type: ignore[index]
            output_pydantic=GraphQLSchema,
        )

a simple task config as

determine_schema:
  description: >
    Query the graphql server to get the schema.
    Use the graphql mcp tool to get the schema.
    Return any error from the server, if the query results in error.
    No other outcome is expected and allowed.
  markdown: true
  expected_output: >
    The schema of the GraphQL server, as a string.

  agent: query_manager

has been used, with a graphql mcp server. but it can be faked by asking to return exactly the schema, if required.

Operating System

macOS Catalina

Python Version

3.12

crewAI Version

1.9.3

crewAI Tools Version

1.9.3

Virtual Environment

Venv

Evidence

The schema in input is a generic example of graphql schema, any will work:

type Query {
  countries: [Country]
  country(code: String!): Country
}

type Country {
  code: String
  name: String
  native: String
  phone: String
  continent: Continent
  currency: String
  languages: [Language]
  emoji: String
}

type Continent {
  code: String
  name: String
}

type Language {
  code: String
  name: String
}

the stack trace is:

Traceback (most recent call last):
  File "/Users/kalfa/Source/graphql-test/graphql/src/graphql/main.py", line 24, in run
    Graphql().crew().kickoff(inputs=inputs)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/crew.py", line 743, in kickoff
    result = self._run_sequential_process()
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/crew.py", line 1150, in _run_sequential_process
    return self._execute_tasks(self.tasks)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/crew.py", line 1236, in _execute_tasks
    task_output = task.execute_sync(
        agent=exec_data.agent,
        context=context,
        tools=exec_data.tools,
    )
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 499, in execute_sync
    return self._execute_core(agent, context, tools)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 740, in _execute_core
    raise e  # Re-raise the exception after emitting the event
    ^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 678, in _execute_core
    pydantic_output, json_output = self._export_output(result)
                                   ~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/task.py", line 968, in _export_output
    model_output = convert_to_model(
        result,
    ...<3 lines>...
        self.converter_cls,
    )
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/utilities/converter.py", line 194, in convert_to_model
    return handle_partial_json(
        result=result,
    ...<3 lines>...
        converter_cls=converter_cls,
    )
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/crewai/utilities/converter.py", line 261, in handle_partial_json
    exported_result = model.model_validate_json(match.group())
  File "/Users/kalfa/Source/graphql-test/graphql/.venv/lib/python3.13/site-packages/pydantic/main.py", line 746, in model_validate_json
    return cls.__pydantic_validator__.validate_json(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        json_data, strict=strict, context=context, by_alias=by_alias, by_name=by_name
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
pydantic_core._pydantic_core.ValidationError: 1 validation error for GraphQLSchema
  Invalid JSON: key must be a string at line 2 column 3 [type=json_invalid, input_value='{\n  code: ID!\n  countr...ing\n  name: String!\n}', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Possible Solution

There are various ways to fix the problem, the quickest is to make the _JSON_PATTERN stronger and filter out more false-positives.

# Logic:
# 1. Objects: Must be empty {} or start with a double-quoted key {"key":
# 2. Arrays:  Must be empty [] or start with a JSON value: ", digit, -, true, false, null, [, or {
# 3. Strings: Standard JSON double-quoted strings
# 4. Primitives: true, false, null, and numbers.
json_pattern = r'(\{\s*\}|\{\s*"[^"\\]*"\s*:.*?[^"\\]\}|\[\s*\]|\[\s*(?:"|[\d\-]|true|false|null|[\[{]).*?[^"\\]\]|"(?:\\.|[^"\\])*"|\b(?:true|false|null)\b|-?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?)'

might be a stronger option. feel free to make it stronger

Additional context

extent analysis

TL;DR

Update the _JSON_PATTERN regular expression in crewai.utilities.converter to a stronger pattern that accurately identifies JSON strings and reduces false positives.

Guidance

  1. Review the proposed solution: Examine the provided json_pattern regular expression and consider its effectiveness in filtering out false positives.
  2. Test the updated pattern: Apply the new pattern to the existing code and verify its performance with various input strings, including the provided GraphQL schema example.
  3. Refine the pattern if necessary: If the updated pattern still yields false positives, further refine it to improve its accuracy in distinguishing between JSON and non-JSON strings.
  4. Consider alternative approaches: If updating the regular expression proves insufficient, explore alternative methods for determining whether a string is valid JSON, such as using a JSON parsing library.

Example

_JSON_PATTERN: Final[re.Pattern[str]] = re.compile(r'(\{\s*\}|\{\s*"[^"\\]*"\s*:.*?[^"\\]\}|\[\s*\]|\[\s*(?:"|[\d\-]|true|false|null|[\[{]).*?[^"\\]\]|"(?:\\.|[^"\\])*"|\b(?:true|false|null)\b|-?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?)')

Notes

The provided json_pattern may still require refinement to handle all possible JSON formats and edge cases. Be cautious when using regular expressions to parse JSON, as they may not cover all valid JSON syntax.

Recommendation

Apply the proposed json_pattern update and test its effectiveness in reducing false positives. If issues persist, consider exploring alternative approaches, such as using a dedicated JSON parsing library.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

the aim is to narrow down as much as possible the false-positives, but without doing a slow json.load. the most effective way to understand if the identifies substring is json, is to try to load the substring, to determine it.

if json.load is acceptable, it will deterministically and accurately tell if it's JSON.

the ultimate aim is to allow a generic look-alike-JSON string to be treated as string, and not serialised, while JSON to be serialised.

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING