PR #4510: Fix #4509: Strip markdown code-block fences from LLM JSON responses in Converter

• Repository: crewAIInc/crewAI
• Author: devin-ai-integration[bot]
• State: closed | merged: False
• Link: https://github.com/crewAIInc/crewAI/pull/4510

Description (problem / solution / changelog)

Fix #4509: Strip markdown code-block fences from LLM JSON responses

Summary

Some LLMs wrap their JSON responses in markdown code-block fences (```json ... ```), which causes model_validate_json() to fail with a Pydantic validation error. This manifests as "Failed to add to long term memory" errors when memory is enabled.

The fix adds a _strip_markdown_fences() helper that removes these delimiters before JSON parsing, applied in three locations within converter.py:

• Converter.to_pydantic() — function-calling path
• Converter.to_pydantic() — non-function-calling path
• convert_to_model() — before json.loads()

Additionally, the function-calling path now has a handle_partial_json() fallback (previously only present in the non-function-calling path), and the except clauses now catch ValueError alongside ValidationError since model_validate_json can raise either.

11 new unit tests cover the helper function, both converter paths, the TaskEvaluation model, and convert_to_model.

Review & Testing Checklist for Human

• Regex edge cases: The fence pattern requires a \n before the closing ```. Verify this matches all real-world LLM outputs (e.g., some models may omit the trailing newline before closing fences).
• Fallback chain in function-calling path: When handle_partial_json is called with agent=None and regex extraction fails, it will fall through to convert_with_instructions() which raises TypeError. This is caught by the outer retry loop, but the error message could be confusing. Consider whether this needs a cleaner error path.
• Widened except clause: (ValidationError, ValueError) — confirm no legitimate errors are being silently swallowed that should propagate.
• End-to-end verification: Run the reproduction script from the issue with memory=True against a real LLM that wraps JSON in markdown fences, and confirm long-term memory saves succeed without errors.
• Unused import: Entity is imported but unused in test_converter_to_pydantic_strips_fences_multiline_no_function_calling.

Notes

• Requested by: João
• Link to Devin run

Changed files

• lib/crewai/src/crewai/utilities/converter.py (modified, +47/-5)
• lib/crewai/tests/utilities/test_converter.py (modified, +160/-0)

PR #4528: Fix Pydantic validation error for LLM responses wrapped in markdown code blocks

• Repository: crewAIInc/crewAI
• Author: paipeline
• State: open | merged: False
• Link: https://github.com/crewAIInc/crewAI/pull/4528

Description (problem / solution / changelog)

Description

Fixes #4509 - Pydantic Validation error while saving in Memory

When LLMs return JSON responses wrapped in markdown code blocks (e.g., json...json...), the current Pydantic validation fails because it attempts to parse the entire response including markdown formatting as raw JSON.

This is a common issue where LLMs naturally format their JSON responses in markdown code blocks for readability, but the parsing logic expects raw JSON.

Root Cause

The Converter.to_pydantic() method was attempting to validate LLM responses directly with Pydantic without stripping markdown formatting. When LLMs returned responses like:

{
  "suggestions": ["Include more metrics", "Add visual charts"],
  "quality": 8.5,
  "entities": [...]
}

The parser would try to parse the entire string including ```json` markers, causing validation errors.

Solution

• Added strip_markdown_code_blocks() function to extract raw JSON from markdown blocks
• Updated Converter.to_pydantic() to strip markdown before validation attempts
• Updated handle_partial_json() to clean markdown before pattern matching
• Updated convert_to_model() to handle markdown-wrapped responses
• Added comprehensive tests for markdown JSON parsing scenarios

Changes Made

1. New function: strip_markdown_code_blocks(text: str) -> str

   • Extracts JSON from json...json... and ...json... blocks
   • Returns original text if no markdown blocks found
   • Handles both language-specified and plain code blocks

2. Enhanced error handling in Converter.to_pydantic():

   • First attempts direct JSON validation
   • On ValidationError, strips markdown and retries
   • Falls back to existing partial JSON handling if still fails

3. Updated helper functions:

   • handle_partial_json() now cleans markdown before pattern matching
   • convert_to_model() handles markdown-wrapped responses

4. Comprehensive tests covering:

   • JSON with ```json` language specifier
   • JSON with plain ```` blocks
   • Regular JSON (unchanged behavior)
   • Edge cases and error scenarios

Testing

• ✅ All new functionality tested with comprehensive test suite
• ✅ Backward compatibility maintained (regular JSON unchanged)
• ✅ Manual testing with reproduction case from issue #4509
• ✅ Existing tests continue to pass

Impact

• Fixes memory storage failures when LLMs wrap JSON in markdown
• Improves reliability of TaskEvaluation parsing
• Maintains backward compatibility for existing functionality
• No breaking changes - only enhanced error handling

This fix resolves the core issue where CrewAI's memory system would fail when LLMs naturally format their JSON responses in markdown code blocks.

Changed files

• lib/crewai/src/crewai/utilities/converter.py (modified, +55/-25)
• lib/crewai/tests/utilities/test_converter.py (modified, +93/-0)

PR #4534: fix: strip markdown code blocks from LLM JSON responses in memory

• Repository: crewAIInc/crewAI
• Author: hztBUAA
• State: open | merged: False
• Link: https://github.com/crewAIInc/crewAI/pull/4534

Description (problem / solution / changelog)

Summary

Fixes #4509

When LLMs return JSON wrapped in markdown code blocks (```json ... ```), Pydantic validation fails with Invalid JSON error during memory storage. This is a common behavior across many LLM providers.

• Added _strip_markdown_code_blocks() helper that removes triple-backtick wrappers before JSON parsing
• Applied the helper at all JSON parsing entry points in converter.py: Converter.to_pydantic() (both function-calling and non-function-calling paths), convert_to_model(), and handle_partial_json()
• Handles ```json, ```, and case-insensitive language tags

Test Plan

• Added unit tests for _strip_markdown_code_blocks() (json blocks, plain blocks, no blocks, multiline, empty string, plain text)
• Added integration tests for convert_to_model() with markdown-wrapped JSON
• Added integration tests for Converter.to_pydantic() with both function-calling and non-function-calling LLMs
• End-to-end test reproducing the exact TaskEvaluation scenario from #4509
• Integration test for handle_partial_json() with markdown code blocks
• All 11 new tests pass; all existing tests continue to pass

🤖 Generated with Claude Code

Changed files

• lib/crewai/src/crewai/utilities/converter.py (modified, +32/-2)
• lib/crewai/tests/utilities/test_converter.py (modified, +133/-0)

PR #4623: Fix Pydantic validation error when saving to memory

• Repository: crewAIInc/crewAI
• Author: Anandesh-Sharma
• State: open | merged: False
• Link: https://github.com/crewAIInc/crewAI/pull/4623

Description (problem / solution / changelog)

Summary

Fixes #4509

• Strip markdown code fences (\``json ... ```) from LLM responses before JSON parsing in Converter.to_pydantic()andconvert_to_model()`
• Some LLMs wrap JSON responses in markdown code blocks, causing ValidationError: Invalid JSON: expected value at line 1 column 1 during task evaluation memory saves
• Reuses the same _CODE_BLOCK_PATTERN regex already established in base_converter_adapter.py for consistency

Test plan

• Added 7 unit tests for _strip_markdown_code_fences() covering: json code fences, plain code fences, no code fences, multiline JSON, surrounding text, plain text passthrough, whitespace stripping
• Added 4 integration tests for Converter.to_pydantic() and convert_to_model() verifying markdown-wrapped JSON is correctly parsed
• Verified all existing converter and task evaluator tests still pass

🤖 Generated with Claude Code

Changed files

• lib/crewai/src/crewai/utilities/converter.py (modified, +33/-0)
• lib/crewai/tests/utilities/test_converter.py (modified, +111/-0)

PR #4627: fix: strip markdown code fences from LLM responses before JSON parsing

• Repository: crewAIInc/crewAI
• Author: Anandesh-Sharma
• State: open | merged: False
• Link: https://github.com/crewAIInc/crewAI/pull/4627

Description (problem / solution / changelog)

Summary

• Fixes Pydantic ValidationError when LLM returns JSON wrapped in markdown code fences (e.g., ```json ... ```)
• Adds _strip_code_fences() helper that extracts content from markdown code fence blocks before model_validate_json()
• Applied to both function-calling and non-function-calling converter paths in Converter.to_pydantic()
• This commonly occurs during memory saving when the LLM wraps TaskEvaluation JSON in code fences

Fixes #4509

Test plan

• Verify JSON wrapped in ```json ... ``` is correctly parsed
• Verify JSON wrapped in ``` ... ``` (no language tag) is correctly parsed
• Verify plain JSON without fences still works correctly
• Verify memory saving works when LLM returns code-fenced JSON

🤖 Generated with Claude Code

Changed files

• lib/crewai/src/crewai/utilities/converter.py (modified, +23/-3)